Package: pdf-rules
pdf_rulesThis library is used to extract information from PDFs and output a CSV.The library was designed to automate the extraction of data from invoices and other such documents; which hold data in hierarchical structures. The user must first define this structure withadd_levelandadd_fieldfunctions, the library is designed to allow maximum user control.InstallationFrom the PyPIpython -m pip install pdf_rulesFrom Sourcegit clone https://gitlab.com/sean-c/pdf_rules
cd pdf_rules
python setup.py installDependenciesThe modulepdftotextis currently used to read the pdf files into a txt format before applying the rules, but this is currently not installed by default on Windows.  On Windows, you will have to find another way to convert the pdfs into txt and then pass the txt into thePDFclass.Please seewsl-wrapperfor a way to convert pdfs to good quality txts on Windows.TutorialFor the purpose of this example, please refer totests/test_pdf.pdf(ortests/test_txt.txtif you couldn't installpdftotext), the example assumes the working directory to be the locaton of this document.The Basics (no levels)Before we extract data, we need to create an instance of thepdf_rules.PDFclass, passingtests/test_pdf.pdfas an argument.import pdf_rules

pdf = pdf_rules.PDF('tests/test_pdf.pdf')

print(pdf)This creates the objectpdfand reads the filetests/test_pdf.pdfinto it as a list of strings, you can usetests/test_txt.txtinstead by passing it in place of the pdf version.  Theprintfunction will print the file as held by thepdfobject (with line numbers). Next, data can be found with theadd_fieldmethod:pdf.add_field(
	'Account',
	lambda rd, i, l: 'Account' in l,
	lambda rd, i, l: l[-7:])Where 'Account' is the field heading and the twolambdafunctions are the 'trigger' and the 'rule'. The trigger and rule have to follow the formatlambda rd, i, l: <expression>, whererdis the whole document,iis the line number, andlis the line. Failiure to passrd,i, andlin that order will result in an exception.The library reads the document and stops when the 'trigger' returnsTrue, the data is then extracted by the 'rule' function. The data is kept inpdf.hierarchy. To create a csv:csv = pdf_rules.CSV(pdf)
print(csv)
csv.write()This should output:['Account']
['ABC1234']and the same output should be written totests/test_pdf_pdfrules.csv.Creating LevelsIn order to extract the invoice data, we must create 'levels', these levels allow the extraction of recurring similar data, like the addresses in the example pdf.pdf.add_level(
            lambda rd, i, l: 'Charges' in l,
            lambda rd, i, l: l == '.')
pdf.add_field(
            'Address',
            lambda rd, i, l: 'Charges' in l,
            lambda rd, i, l: l.split(' - ')[1],

            # IMPORTANT: always remember to pass the appropriate level!
            level=1)Here, an extra level is added, which starts on every line containing 'Charges', and ends on every line containing just the '.' character. The tables each contain some unique data, which we can collect by creating another field 'Addrssses'.Note that allpdfobjects have a level 0, covering the whole document, so theadd_fieldfunction needs the level stated if it is not intended for level 0The output should be:['Account', 'Address']
['ABC1234', '1234 Fake Street, London W15 6GH']
['ABC1234', '5678 Fake Ave., Glasgow G3 6HJ'Note that 'Address' items found in level 1 inherited the 'Account' from level 0PDF.showYou can usepdf.show()to display a view of the pdf with the rows highlighted according to the levels picked up by all theadd_levelscalls.  This can help with troubleshooting and fine-tuning theadd_levelstrigger arguments.As of v1.3,pdf.show()now hasTandDin the left margin to indicate when anadd_fieldtrigger or rule callback is triggered.PDF.showcurrently relies on curses, so it will only work from a terminal.Levels Within LevelsTheadd_levelfunction can act within levels, where all data found in sub-levels will inherit data from higher ones.Adding another level, we can extract more data:import re

pdf.add_level(
            lambda rd, i, l: re.search(r'[A-Z]{2}/\d{4}/[A-Z]', l),
            lambda rd, i, l: False)
pdf.add_field(
            'ID',
            lambda rd, i, l: re.search(r'[A-Z]{2}/\d{4}/[A-Z]', l),
            lambda rd, i, l: l.strip(),
            level=2)Notice that the 'trigger' for level 2 is set toFalse, this will cut off the table just before the start of the next one.Our output so far:['Account', 'Address', 'ID']
['ABC1234', '1234 Fake Street, London W15 6GH', 'IG/1234/H']
['ABC1234', '1234 Fake Street, London W15 6GH', 'ID/5678/I']
['ABC1234', '1234 Fake Street, London W15 6GH', 'RD/9012/P']
['ABC1234', '1234 Fake Street, London W15 6GH', 'IN/5724/O']
['ABC1234', '5678 Fake Ave., Glasgow G3 6HJ', 'DH/0471/U']
['ABC1234', '5678 Fake Ave., Glasgow G3 6HJ', 'JF/8364/N']
['ABC1234', '5678 Fake Ave., Glasgow G3 6HJ', 'HD/1684/Q']Adding yet another level to find each line with charges, we will use trigger on every line with a date in, the line 'Invoice Date 01/10/2020' will not csuse a trigger as it is not in level 2.pdf.add_level(
            lambda rd, i, l: re.search(r'\d{2}/\d{2}/\d{4}', l),
            lambda rd, i, l: False)
pdf.add_field(
            'Cost',
            lambda rd, i, l: True,
            lambda rd, i, l: pdf_rules.listify(l)[-1],
            level=3)Note the use ofpdf_rules.listify, this is a helper function to crudely convery the line into a list, delimited by two+ spaces.
Also note the 'trigger' for 'Cost' is set toTrue, this means it will trigger on every line, use with caution.We now have:['Account', 'Address', 'ID', 'Cost']
['ABC1234', '1234 Fake Street, London W15 6GH', 'IG/1234/H', '£100.00']
['ABC1234', '1234 Fake Street, London W15 6GH', 'IG/1234/H', '-$23.00']
['ABC1234', '1234 Fake Street, London W15 6GH', 'IG/1234/H', '£50.00']
['ABC1234', '1234 Fake Street, London W15 6GH', 'ID/5678/I', '£52.00']
['ABC1234', '1234 Fake Street, London W15 6GH', 'RD/9012/P', '£48.00']
['ABC1234', '1234 Fake Street, London W15 6GH', 'IN/5724/O', '-£324.00']
['ABC1234', '5678 Fake Ave., Glasgow G3 6HJ', 'DH/0471/U', '£64.00']
['ABC1234', '5678 Fake Ave., Glasgow G3 6HJ', 'JF/8364/N', '£83.00']
['ABC1234', '5678 Fake Ave., Glasgow G3 6HJ', 'HD/1684/Q', '£45.00']From here we could keep adding rules in order to catch all the data, just as we did with the cost field.Helpful ThingsThere are some very useful features that I won't go into here, but you can seetutorial.pyfor examples of how to use them.FallbacksAn optional argument of thePDF.add_fieldfunction is thefallback.  This isNoneby default but if you passfallback='1234', then pdf_rules will use '1234' for that field whenever the trigger doesn't trigger, or the rule returnsNoneor throws an exception.Get last EntryYou can get the last entry found by pdf_rules for a given field withpdf.last_entry('field')To DoHighlight matches inPDF.showOffer alternatives to curses forPDF.show, maybe usepillowto create an image and show in popup.More tests