Package: pdf2dataset
pdf2datasetConverts a whole subdirectory with any volume (small or huge) of PDF documents to a dataset (pandas DataFrame).
No need to setup any external service (no database, brokers, etc). Just install and run it!Main featuresConversion of a whole subdirectory with PDFs documents into a pandas DataFrameSupport for parallel and distributed processing throughrayExtractions are performed by page, making tasks distribution more uniform for handling documents with big differences in number of pagesIncremental writing of resulting DataFrame, making possible to process data bigger than memoryError tracking of faulty documentsResume interrupted processingExtract text throughpdftotextUse OCR for extracting text throughpytesseractExtract images throughpdf2imageSupport to implement custom features extractionHighly customizable behavior through paramsInstallationInstall DependenciesFedora# "-por" for portuguese, use the documents language$sudodnfinstall-ygcc-c++poppler-utilspkgconfigpoppler-cpp-develpython3-develtesseract-langpack-porUbuntu (or debians)$sudoaptupdate# "-por" for portuguese, use the documents language$sudoaptinstall-ybuild-essentialpoppler-utilslibpoppler-cpp-devpkg-configpython3-devtesseract-ocr-porInstall pdf2datasetFor usage$pip3installpdf2dataset--user# Please, isolate the environmentFor development# First, install poetry, clone repository and cd into it$poetryinstallUsageSimple - CLI# Note: path, page and error will always be present in resulting DataFrame# Reads all PDFs from my_pdfs_dir and saves the resultant dataframe to my_df.parquet.gzip$pdf2datasetmy_pdfs_dirmy_df.parquet.gzip# Most basic, extract all possible features$pdf2datasetmy_pdfs_dirmy_df.parquet.gzip--features=text# Extract just text$pdf2datasetmy_pdfs_dirmy_df.parquet.gzip--features=image# Extract just image$pdf2datasetmy_pdfs_dirmy_df.parquet.gzip--num-cpus1# Maximum reducing of parallelism$pdf2datasetmy_pdfs_dirmy_df.parquet.gzip--ocrtrue# For scanned PDFs$pdf2datasetmy_pdfs_dirmy_df.parquet.gzip--ocrtrue--langeng# For scanned documents with english textResume processingIn case of any interruption, to resume the processing, just use the same path as output and the
processing will be resumed automatically. The flag--saving-interval(or the paramsaving_interval)
controls the frequency the output path will be updated, and so, the processing "checkpoints".Using as a libraryMain functionsThere're some helper functions to facilitate pdf2dataset usage:extract:function can be used analogously to the CLIextract_text:extractwrapper withfeatures=textextract_image:extractwrapper withfeatures=imageimage_from_bytes:(pdf2image.utils) get a PillowImageobject given the image bytesimage_to_bytes:(pdf2image.utils) get the image bytes given the a PillowImageobjectBasic examplefrompdf2datasetimportextractextract('my_pdfs_dir','all_features.parquet.gzip')Small dataOne feature, not available to the CLI, is the custom behavior for handling small volumes of data (small can
be understood as that: the extraction won't run for hours or days and won't be distributed).The complete list of differences are:Faster initialization (use multiprocessing instead of ray)Don't save processing progressDistributed processing not supportedDon't write dataframe to diskReturns the dataframeExample:frompdf2datasetimportextract_textdf=extract_text('my_pdfs_dir',small=True)# ...Pass list of files pathsInstead of specifying a directory, one can specify a list of files to be processed.Example:frompdf2datasetimportextractmy_files=['./tests/samples/single_page1.pdf','./tests/samples/invalid1.pdf',]df=extract(my_files,small=True)# ...Pass files from memoryIf you don't want to specify a directory for the documents, you can specify the tasks that
will be processed.The tasks can be of the form(document_name, document_bytes, page_number)or just(document_name, document_bytes),document_namemust ends with.pdfbut
don't need to be a real file,document_bytesare the bytes of the pdf document andpage_numberis the number of the page to process (all pages, if not specified).Example:frompdf2datasetimportextract_texttasks=[('a.pdf',a_bytes),# Processing all pages of this document('b.pdf',b_bytes,1),('b.pdf',b_bytes,2),]# 'df' will contain results from all pages from 'a.pdf' and page 1 and 2 from 'b.pdf'df=extract_text(tasks,'my_df.parquet.gzip',small=True)# ...Returning a listIf you don't want to handle the DataFrame, is possible to return a nested list with the features values.
The structure for the resulting list is:result = List[documents]
documents = List[pages]
pages = List[features]
features = List[feature]
feature = anyanyis any type supported by pyarrow.features are ordered by the feature name (text,image, etc)Example:>>>frompdf2datasetimportextract_text>>>extract_text('tests/samples',return_list=True)[[[None]],[['First page'],['Second page'],['Third page']],[['My beautiful sample!']],[['First page'],['Second page'],['Third page']],[['My beautiful sample!']]]Features with error will haveNonevalue as resultHere,extract_textwas used, so the only feature istextCustom FeaturesWith version >= 0.4.0, is also possible to easily implement extraction of custom features:Example:This is the structure:frompdf2datasetimportextract,feature,PdfExtractTaskclassMyCustomTask(PdfExtractTask):@feature('bool_')defget_is_page_even(self):returnself.page%2==0@feature('binary')defget_doc_first_bytes(self):returnself.file_bin[:10]@feature('string',exceptions=[ValueError])defget_wrong(self):raiseValueError("There was a problem!")if__name__=='__main__':df=extract('tests/samples',small=True,task_class=MyCustomTask)print(df)df.dropna(subset=['text'],inplace=True)# Discard invalid documentsprint(df.iloc[0].error)First print:path  page doc_first_bytes  ...                  text  wrong                                              error
0                invalid1.pdf    -1   b"I'm invali"  ...                  None   None  image_original:\nTraceback (most recent call l...
1             multi_page1.pdf     2  b'%PDF-1.5\n%'  ...           Second page   None  wrong:\nTraceback (most recent call last):\n  ...
2             multi_page1.pdf     3  b'%PDF-1.5\n%'  ...            Third page   None  wrong:\nTraceback (most recent call last):\n  ...
3   sub1/copy_multi_page1.pdf     1  b'%PDF-1.5\n%'  ...            First page   None  wrong:\nTraceback (most recent call last):\n  ...
4   sub1/copy_multi_page1.pdf     3  b'%PDF-1.5\n%'  ...            Third page   None  wrong:\nTraceback (most recent call last):\n  ...
5             multi_page1.pdf     1  b'%PDF-1.5\n%'  ...            First page   None  wrong:\nTraceback (most recent call last):\n  ...
6  sub2/copy_single_page1.pdf     1  b'%PDF-1.5\n%'  ...  My beautiful sample!   None  wrong:\nTraceback (most recent call last):\n  ...
7   sub1/copy_multi_page1.pdf     2  b'%PDF-1.5\n%'  ...           Second page   None  wrong:\nTraceback (most recent call last):\n  ...
8            single_page1.pdf     1  b'%PDF-1.5\n%'  ...  My beautiful sample!   None  wrong:\nTraceback (most recent call last):\n  ...

[9 rows x 8 columns]Second print:wrong:
Traceback (most recent call last):
  File "/home/icaro/Desktop/pdf2dataset/pdf2dataset/extract_task.py", line 32, in inner
    result = feature_method(*args, **kwargs)
  File "example.py", line 16, in get_wrong
    raise ValueError("There was a problem!")
ValueError: There was a problem!Notes:@featureis the decorator used to define new features.The extraction method name must start with the prefixget_(avoids collisions with attribute names and increases readability)First argument to@featuremust be a valid PyArrow type, complete listhereexceptionsparam specify a list of exceptions to be recorded on DataFrame, otherwise they are raisedFor this example, all available features plus the custom ones are extractedResults FileThe resulting "file" is a directory with structure specified by dask with pyarrow engine,
it can be easily read with pandas or dask:Example with pandas>>>importpandasaspd>>>df=pd.read_parquet('my_df.parquet.gzip',engine='pyarrow')>>>dfpathpagetexterrorindex0single_page1.pdf1Mybeautifulsample!1sub1/copy_multi_page1.pdf2Secondpage2sub2/copy_single_page1.pdf1Mybeautifulsample!3sub1/copy_multi_page1.pdf3Thirdpage4multi_page1.pdf1Firstpage5multi_page1.pdf3Thirdpage6sub1/copy_multi_page1.pdf1Firstpage7multi_page1.pdf2Secondpage0invalid1.pdf-1Traceback(mostrecentcalllast):\nFile"/h...There is no guarantee about the uniqueness or order ofindex, you might need to create a new index with
the whole data in memory.The-1page number means that was not possible of even parsing the document.Run on a ClusterSetup the ClusterFollow ray documentation formanualorautomaticsetup.Run itTo go distributed you can run it just like local, but using the--addressand--redis-passwordflags to point to your cluster (More information).With version >= 0.2.0, only the head node needs to have access to the documents in disk.CLI Helpusage: pdf2dataset [-h] [--features FEATURES]
                   [--saving-interval SAVING_INTERVAL] [--ocr-lang OCR_LANG]
                   [--ocr OCR] [--chunksize CHUNKSIZE]
                   [--image-size IMAGE_SIZE] [--ocr-image-size OCR_IMAGE_SIZE]
                   [--image-format IMAGE_FORMAT] [--num-cpus NUM_CPUS]
                   [--address ADDRESS] [--dashboard-host DASHBOARD_HOST]
                   [--redis-password REDIS_PASSWORD]
                   input_dir out_file

Extract text from all PDF files in a directory

positional arguments:
  input_dir             The folder to lookup for PDF files recursively
  out_file              File to save the resultant dataframe

optional arguments:
  -h, --help            show this help message and exit
  --features FEATURES   Specify a comma separated list with the features you
                        want to extract. 'path' and 'page' will always be
                        added. Available features to add: image, page, path,
                        text Examples: '--features=text,image' or '--
                        features=all'
  --saving-interval SAVING_INTERVAL
                        Results will be persisted to results folder every
                        saving interval of pages
  --ocr-lang OCR_LANG   Tesseract language
  --ocr OCR             'pytesseract' if true, else 'pdftotext'. default:
                        false
  --chunksize CHUNKSIZE
                        Chunksize to use while processing pages, otherwise is
                        calculated
  --image-size IMAGE_SIZE
                        If adding image feature, image will be resized to this
                        size. Provide two integers separated by 'x'. Example:
                        --image-size 1000x1414
  --ocr-image-size OCR_IMAGE_SIZE
                        The height of the image OCR will be applied. Width
                        will be adjusted to keep the ratio.
  --image-format IMAGE_FORMAT
                        Format of the image generated from the PDF pages
  --num-cpus NUM_CPUS   Number of cpus to use
  --address ADDRESS     Ray address to connect
  --dashboard-host DASHBOARD_HOST
                        Which IP ray webui will try to listen on
  --redis-password REDIS_PASSWORD
                        Redis password to use to connect with rayTroubleshootingTroubles with high memory usageDecrease the number of CPUs in use, reducing the level of parallelism, test it with--num-cpus 1flag and then increase according to your hardware.Use smaller chunksize, so less documents will be put in memory at once. Use--chunksize 1for having1 * num_cpusdocuments in memory at once.How to ContributeJust open yourissuesand/orpull requests, all are welcome :smiley:!