Extracting headers and paragraphs from pdf using PyMuPDF | by Louis de Bruijn | Towards Data Science

Here’s for something completely different: parsing pdf documents and extracting the headers and paragraphs! There are various packages that extract text from pdf documents and convert them to HTML, but I’ve found these to be either too elaborate for the task at hand and/or too complex. In my experience, generic pdf parsers generalize okay-ish over all documents, but for a specific use-case of somewhat similarly structured documents, we can enhance performance with some code of our own!
Since pdf files consist of unstructured text, we need to find some similarities over the different documents on how headers and paragraphs are separated. Using a small sample of large (50–150 pages each) pdf files concerning Dutch policy terms for insurers, what I’ve found somewhat consistently is that headers and paragraphs are often separated by the font size and font weight of the text and that the most used font can be considered the paragraph. Now, this is a good starting point for us to create a methodology.
Use PyMuPDF to identify the paragraphs as text with the most used font in the document, headers as anything larger, and subscripts as anything smaller than the paragraph style.Create a dictionary with HTML style element tags such as <h1>, <p> and <s0> for the headers, paragraphs, and subscripts.Annotate pieces of text with these element <tags>.
Use PyMuPDF to identify the paragraphs as text with the most used font in the document, headers as anything larger, and subscripts as anything smaller than the paragraph style.
Create a dictionary with HTML style element tags such as <h1>, <p> and <s0> for the headers, paragraphs, and subscripts.
Annotate pieces of text with these element <tags>.
We’re using the PyMuPDF package for reading the pdf files. This package opens pdf documents page per page and saves all its content in a block and identifies the text size, font, colour and flags. What I’ve found is that some pdf documents discriminate headers and paragraphs only by the font and size, but others use all four attributes. To account for this, we’re going to add a granularity flag so that we can decide which attributes to incorporate in the distinction of the different textual parts in the document.
At this stage, we’re going to create a dictionary with all the different styles and attributes and a list of [(font_size, count), ..] for all these styles.
We iterate over the pages and blocks of the document, which is parsed by the PyMuPDF package (imported as fitz) and identify all the styles and attributes according to our granularity flag.
Output for one of our documents looks like this:
font_counts, styles = fonts(doc, granularity=False)[('9.5', 1079), ('10.0', 190), ('8.5', 28), ('10.5', 24), ...]{'12.0': {'size': 12.0, 'font': 'ArialMT'}, '9.0': {'size': 9.0, 'font': 'XKZKVH+VAGRoundedStd-Light'}, ...}
We can see that the most used font-size is 9.5, with a count of 1079 text spans of this size. It is very likely that this font-size represents the paragraphs in our document.
Next up we’re going to create a dictionary with the element tags for each of the font sizes. Note, we’re only considering the font-sizes here, but with a few extra lines of code, you can find a way to incorporate the other attributes if you’re using the granularity=True flag in the fonts() function!
line 12-13 First we’re identifying the paragraph’s size to discriminate between the type of tag <header>, <paragraph> or <subscript>. line 16-19 we’re sorting the sizes high to low so that we can add the correct integer to each element tag. Note that we’re using 1 for the largest tag and this number decreases as the font-sizes decrease for both the headers and subscripts! We’re doing this because it is in the same fashion as the ordering of HTLM tags. line 22-32 populates the dictionary with the tags, which is shown below.
{60.0: '<h1>', 59.69924545288086: '<h2>', 36.0: '<h3>', 30.0: '<h4>', 24.0: '<h5>', 20.0: '<h6>', 16.0: '<h7>', 14.0: '<h8>', 13.0: '<h9>', 10.5: '<h10>', 10.0: '<h11>', 9.5: '<p>', 9.452380180358887: '<s1>', 9.404520988464355: '<s2>', 8.5: '<s3>', 8.0: '<s4>', 7.5: '<s5>', 7.0: '<s6>'}
We again iterate over the pages of the document and the blocks. For the first block, we initialize the block_string with the element tag and the actual text from the span s['text']. For each following span, we check whether the font size matches the previous span’s font size or whether there is a new text size. Accordingly, we concatenate strings if they’re the same size. blocks are parts of text that are separated and identified by the PyMuPDF package, but I’ve found that they sometimes contain parts of a sentence. Hence, why I concatenate them with a '|' delimiting the fact that a new block has started. In post-processing steps, we can then decide what to do with these pipe-delimited parts (concatenate them or separate them).
We return a list of strings with pipes in them and are then able to identify which textual parts are headers, paragraphs, or subscripts, as shown below.
['<h4>Als onderdeel van het | ZekerheidsPakket Particulieren |', '', '<h1>Informatie  over  uw|', '<h2>Inboedelverzekering|', '<h1>Basis|', '', '<h6>Inhoud|', '', '<p>pagina| Leeswijzer, Uw verzekering in het kort | 3 | Polisvoorwaarden Inboedelverzekering Basis | 7 |', '', '<s3>3|', '', '<h7>Uw verzekering in het kort|',
As you can see we still need to perform several post-processing steps to clean the data and maybe order it in a different way, but this is at least a starting point. I hope you’ve learned something here and happy coding! The full script and example pdf document can be found here.