Package: pdfinsight
PDFInsightText Mining & Classification ToolkitExtract and categorise text-based PDFs into the following categoriestable of contentsheaderheadingtablescontentfootnotefooterpage numberunsure (text that cannot be categorised)Prepare the extracted content into a document store ready for ingestion by a sentence transformers modelInstallationpip install pdfinsightExamplefrom pdfinsight import pdf_extractor, remove_toc, pivot_df_by_heading, df2docstore
df = pdf_extractor("sample.pdf")
dffilepageblockrefined_blockblock_ymin_diffblock_is_list...font_characteristicsfontfont_colortextimagecat0tests/sample.pdf121NaNFalse...0Calibri0THIS IS Atoc1tests/sample.pdf13295.0False...0Calibri0SAMPLE PDFtoc2tests/sample.pdf213-373.0False...0Calibri0Sample PDFtoc3tests/sample.pdf265-707.0False...16Calibri-Bold0TITLEtoc4tests/sample.pdf27630.0False...0Calibri0Lorem ipsum dolor sit amet, consectetur adipis...toc..........................................76tests/sample.pdf31520-15.0False...0Calibri03956table77tests/sample.pdf3132059.0False...0Calibri0euismod sit amet tortor.table78tests/sample.pdf31420-15.0False...0Calibri0rhoncus semper.table79tests/sample.pdf3315730.0False...0Calibri4485572THIS IS A FOOTERfooter80tests/sample.pdf341514.0False...0Calibri4485572Page 3 of 3footer# remove rows where cat column is marked as 'toc' 
df = remove_toc(df)

# pivot the dataframe such that for each row's text, it will
# 1) merge with previous row if they are of the same category
# 2) it will iteratively search up the rows to search for the 
#    relevant headings for the row's text
pivot_df = pivot_df_by_heading(df)
pivot_dffileheading1heading2content0tests/sample.pdfTITLENoneLorem ipsum dolor sit amet, consectetur adipis...1tests/sample.pdfTITLEMaecenas eu dapibus diam.a) Suspendisse id sem sed lacus luctus digniss...2tests/sample.pdfTITLEProin at lorem euProin at lorem eu urna volutpat dignissim vel ...# set the links to be the same as the filename
link_dict = dict(zip(transformed_df.file.unique(), transformed_df.file.unique()))

# convert the pivot_df into a dictionary format suitable for ingestion by a sentence transformers model
docStore = df2docstore(pivot_df, chunk_size = 100, link_dict = link_dict)
docStoreoutput:[{'content':'TITLE\nLoremipsumdolorsitamet,consecteturadipiscingelit.Namaccumsansollicitudinullamcorper.Integeridvestibulumrisus.Suspendisseodioerat,venenatisdictummiat,porttitorcursusmauris.Maurisnonleoeunisirhoncuslaciniabibendumvitaeorci.Maecenasdiamlectus,ultriciesvitaeenima,maximusportamagna.Fusceposueredolorblandit,egestasleoposuere,ornareenim.Utdignissimiaculisleoeueuismod.Etiampulvinaracexnonpretium.Sedtempusinestvitaetristique.Sedacjustouterosgravidalaoreetinvelerat.Sedfinibusnonanteelementumpretium.Proinleonunc,feugiatvelnullaat,porttitorpulvinarnibh.\n1.Sedrhoncusposueremattis.\n2.Fuscedictumnisiatfaucibusvulputate.\n3.Crasmassavelit,suscipitaeroset,aliquetconsequatmauris.Proinmaximuspharetraligulabibendumcommodo.Fuscequisnequedui.Inmolestiepurusetturpismaximusultricies.Morbiultriciestincidunttellus,acpharetraurnaconsequata.Nullamconsequatleolobortisaliquettincidunt.Etiamlaciniafermentumipsum.Phasellusdapibusleomagna,necsollicitudinsapienaliquetvel.Suspendisseaantesedtortorsollicitudinluctusnonnecligula.Utauctorelitsedduisodalesblandit.Fuscevulputatemagnanonarcumollis,eueleifenderattincidunt.Aeneanconsectetur,eratvelsollicitudinvenenatis,felisantecursusvelit,quispellentesquetellusloremidneque.Praesentviverratinciduntodio,inluctustellussodalesviverra.','source':'tests/sample.pdf','update':''},{'content':'TITLE\nMaecenaseudapibusdiam.\na)Suspendisseidsemsedlacusluctusdignissimaceumi.Praesenteunislenim.Etiamacliberosapien.Maurisaterosneque.Vestibulumlectusligula,temporaccumsannuncut,sodalesrhoncuspurus.Duisveltristiqueipsum.Utnecnullaetturpisfinibuspulvinar.Aeneaneleifendmalesuadasapienvelmalesuada.Proinviverranisinontelluscongueauctor.Nameuismodgravidaduiataliquet.Praesentvitaefacilisislibero.Donecvestibulumsodalesaugue,etcommodomauris.Utsodalesinterdumex,quisfeugiatnisifringillasitamet.Fusceegetnequeacestullamcorpertristiqueatnecleo.\nb)Donecsemenim,fermentumsitamettinciduntsed,sempersitametodio.Quisquevitaeodioturpis.Aliquameratvolutpat.Suspendissepotenti.Praesentsitametviverraenim,portapellentesqueturpis.Utposuerelacuspharetrasapienmaximus,sedfeugiatmieleifend.Duiscongueerosinblanditvarius.Maecenasefficitur,urnasedcommodopulvinar,minisiconsecteturaugue,velsagittistortorrisusnectellus.Namsollicitudinlacuseuenimfringilla,veldapibusquammattis.Nullamacleoet1maurispharetradictumacquistortor.Aeneannullami,semperacultriciessed,placerateterat.Maecenasullamcorper,orciegetpulvinarfermentum,magnatortorlaoreetmagna,egetlaciniaquamnullavitaenulla.Quisqueauguelacus,ullamcorperinnuncsed,sodalesaccumsandui.','source':'tests/sample.pdf','update':''},{'content':'TITLE\nProinatloremeu\nProinatloremeuurnavolutpatdignissimvelnecerat.Maurisacduivelfelisrutrummalesuadaegetquisante.Phaselluselementumportalorem,eusagittistortorconguesed.Vivamusnecdiamsagittis,sagittiseratnec,laciniaerat.Maecenasatleometus.Vestibulumsitametdiamutleoaccumsanpharetra.Prointinciduntvestibulumtincidunt.Pellentesquepurusnibh,fermentumsitametduiat,maximusporttitorsapien.\nColumn1Column2Column3Praesentvariusconsequatidultriciesdiamaliquam456justo,volutpat\nVestibulumanteipsum\netposuereelitelitsedorc567\nprimisinfaucibusorciluctusetultricesposuerecubiliacurae;\nconguenecmolestieet,Nullamposuerenibhutnisi3956euismodsitamettortor.rhoncussemper.','source':'tests/sample.pdf','update':''}]Referenceshttps://github.com/pymupdf/PyMuPDF