Package: pypdfplot
OverviewThepypdfplotpackage provides a backend toMatplotlibthat generates a PDF file of the plot with the generating Python script embedded.Normally, once aMatplotlibplot is saved as PNG or PDF file, the link between the plot and its generating Python script is lost. The philosophy behindpypdfplotis that there should be no distinction between the Python script that generates a plot and its output PDF file, much like there is no such distinction in an Origin or Excel file. As far aspypdfplotis concerned,the generating scriptisthe plot.When thepypdfplotbackend is loaded and a figure is saved withplt.savefig(), the generating Python script is embedded into the output PDF file in such a way that when the PDF file is renamed from .pdf to .py, the file can be read by a Python interpreter directly without alteration. The script can be modified to implement changes in the plot, after which the script is ran again to produce the updated PDF file of the plot – including the updated embedded generating script.The resulting file is both a valid Python file and a valid PDF file, and is conveniently call a PyPDF file. The compatibility with both Python and PDF is achieved by arranging the data blocks in the PyPDF file in a very specific order, such that the PDF-part is read as comment block in Python, and the Python-part is seen as an embedded file by a PDF reader.To learn more about how to usepypdfplot, continue with readingQuickstart, or check out the commented examples in theexamples folder.InstallationPyPI repositoryInstall the package from the PyPI repository opening a command prompt and enter:pipinstallpypdfplotGithub or downloadAlternatively, the source files can be downloaded directly from the GitHubrepository. After downloading the source files, navigate to the project directory and install the package by opening a command prompt and enter:pythonsetup.pyinstallQuickstartIn this example, a plot is produced withMatplotliband saved as PyPDF-file using thepypdfplotbackend.First, create a new python file and call it e.g.example.py.To produce a PyPDF-file, all you have to do is import thepypdfplotbackend by adding the lineimport pypdfplot.backendbefore importingMatplotlib:importpypdfplot.backendimportmatplotlib.pyplotaspltimportnumpyasnpx=np.arange(-10,20,0.1)y=x**2plt.plot(x,y)plt.savefig('example.pdf')After running this script, the fileexample.pywill have been removed and replaced by a new fileexample.pdf:As can be seen in the “Attachments” column on the left, the orginalexample.pygenerating script is embedded in the PDF file.The script can be accessed by renamingexample.pdfback toexample.pyand opening it in a text editor:#%PDF-1.4 24 0 obj << /Type /EmbeddedFile /Length        690 >> streamimportpypdfplot.backendimportmatplotlib.pyplotaspltimportnumpyasnpx=np.arange(-10,20,0.1)y=x**2plt.plot(x,y)plt.savefig('example.pdf')"""
--- Do not edit below ---
endstream
endobj
1 0 obj

<< ... >>

startxref
9567
%%EOF
0000010174 LF
PyPDF-1.0
"""It can be seen that after saving the plot with thepypdfplotbackend, a commented line was added at the first line and a large comment block was appended at the end of the file. These comments contain all the necessary data for displaying the PDF and should not be altered directly by the user.To update the plot, the user should instead modify the generating Python script and the PDF will be updated after running the script again!For example, let’s add another plot, e.g. a sine function:#%PDF-1.4 24 0 obj << /Type /EmbeddedFile /Length        690 >> streamimportpypdfplot.backendimportmatplotlib.pyplotaspltimportnumpyasnpx=np.arange(-10,20,0.1)y1=x**2y2=100*np.sin(x)plt.plot(x,y1)plt.plot(x,y2)plt.savefig('example.pdf')"""
--- Do not edit below ---
endstream
endobj
1 0 obj

<< ... >>

startxref
9567
%%EOF
0000010174 LF
PyPDF-1.0
"""After runningexample.py, the file is again replaced by our updatedexample.pdf:Functionssavefig()Saves the current plot as PyPDF file.savefig(fname,pack_list=[],cleanup=True,multiple='pickle',force_pickle=False,verbose=Trueprompt_overwrite=False,**kwargs)fname:strFilename of the output file.pack_list:list, default =[]List with filenames that will be embedded in the PyPDF-file. The generating script is added separately and should not be included here. SeePacking and unpackingfor more details.multiple:str, default ='pickle'How to handle multiple plots in a single generating script. Can be any of'pickle','add_page', or'finalize'. SeeMultiple plotsfor more detailscleanup:bool, default =TrueWhether or not to cleanup files that have been embedded in the PyPDF file. Set toFalseand run script to extract embedded files.force_pickle:bool, default =FalsePickles the figure and embeds a Python script that unpickles and reads the figure again. This can be useful when dealing with very large source files, seePicklingfor more details.verbose:bool, default =TrueWether or not to show verbose comments during saving.prompt_overwrite:bool, default =FalseWether or not to prompt when the output file already exists and is about to be overwritten. IfFalseand the output file does already exist, file will be overwritten if possible.**kwargs:Any keyword arguments accepted bymatplotlib.pyplot.savefig()unpack()Extracts the files embedded in the PyPDF-file. Must be called before embedded files are read by the generating script. This can be guaranteed by importing the backend usingpypdfplot.backend.unpack, which automatically callsunpack()with its default parameters. SeePacking and unpackingfor more details.unpack(fname=None,verbose=True)fname:str, default =NoneFilename of the PyPDF file to unpack. IfNone, the filename of the currently executing script is taken.verbose:bool, default =TrueWether or not to show verbose comments during extraction.fix_pypdf()Fixes PyPDF files that have been severed, e.g. because they were saved as ‘regular’ PDF-files outside ofpypdfplot. SeePyPDF compliance typesfor more details.fix_pypdf(input_fname,output_fname=None,verbose=True)input_fname:strFilename of the severed PyPDF fileoutput_fname:str, default =NoneFilename of the fixed output PyPDF file. IfNone, the input PDF file is overwritten.verbose:bool, default =TrueWether or not to show verbose comments during fixing.Packing and unpackingIn many cases you may want to plot data that is stored in a separate external file.
In order for this to work, the external data file must be included, which can be achieved by packing and unpacking the data into the PyPDF file.Packing filesIn this section we show how to write a script that opens data from an external Excel file and reads the title and axis label from an extrnal text file, where both files are embedded in the PyPDF file.Create an excel filedata.xlsxand fill the Excel file with data, e.g. the first 10 numbers of the Fibonacci sequence:Next, create a text filetitle.txtand add names for the plot title and axes:Finally, create a new python filepacking.pythat looks as follows:importpypdfplot.backend.unpackimportmatplotlib.pyplotaspltimportpandasaspddf=pd.read_excel('data.xlsx')plt.plot(df.x,df.y,'o')withopen('title.txt','r')asf:title=f.readline()xlabel=f.readline()ylabel=f.readline()plt.title(title)plt.xlabel(xlabel)plt.ylabel(ylabel)plt.savefig('packing.pdf',pack_list=['data.xlsx','title.txt'],cleanup=True,)By appending the filenamesdata.xlsxandtitle.txtto thepack_listlist, the files will be embedded in the output PyPDF file.Note that in order for this script to work, the next time it is ran these files must be extractedbeforePython attempts to open them. To make sure they are extracted before they are referenced, the PyPDF file can be unpacked immediately when thepypdfplotbackend is loaded, by importingpypdfplot.backend.unpackinstead of importing justpypdfplot.backend.Finally, the keywordcleanup=Truewas passed tosavefig()to toggle whether the external files will be removed (cleanup=True,default) or not (cleanup=False).Warning:By passingcleanup=True(which is the default!), the local files are removed. If the original Python scriptwithoutthe files embedded is now ran again, an exception will be raised because the local files no longer exist. Therefore whencleanup=Trueis passed, the only way the script can be ran again is by opening the output PyPDF file, which has the external files embedded.After running the script, thepacking.pyfile is replaced by the output PyPDF filepacking.pdf:As can be seen in the “Attachments” list, the external filesdata.xlsxandtitle.txthave been embedded in the PyPDF file, in addition to the generating Python script.Unpacking filesA PyPDF file that has files embedded must always be unpacked at the beginning of the script (seePacking files), so unpacking files is just a matter ofnot removing the filesafter they have been extracted.This is done by simply passing the keyword argumentcleanup=False.Multiple plotsIn some cases a single Python file may be generating multiple plots, e.g. in an iterative loop. This poses a challenge, because according to thepypdfplotphilosophy,“the generating scriptisthe plot”– but now there are multiple plots and only one generating script.There are two options available to resolve this issue; in the first (default), the generating script (including the iterative loop), is embedded in the first PyPDF file. After that, following plots arepickled: thefigureobject is saved as external file, and a small script that reads the pickledfigureis included as generating Python script in the PyPDF file. With this option, each plot is stored in a separate PyPDF file, and the iterative loop can be accessed by opening the first PyPDF file.The second way is to add the multiple plots asmultiple pagesin asinglePyPDF file. Because there is now only a single PyPDF file, there is no ambiguity in what file the generating Python script should be embedded.Below each of the two options are detailed with an example.PicklingThe way of handling multiple files can be selected with the keywordmultiple. Since pickling is the default (multiple='pickle'), it does not have to be specified explicitly.The code below generates plots of a linear, quadratic, and cubic function respectively:importpypdfplot.backendimportmatplotlib.pyplotaspltimportnumpyasnpx=np.arange(-10,20,0.1)forninrange(3):y=x**(n+1)plt.plot(x,y)plt.savefig('plot{:d}.pdf'.format(n+1))plt.clf()Which produces the following three plots:The first plot,plot1.pdf, has the original generating script embedded.Plotsplot2.pdfandplot3.pdfhave theirfigureobjects pickled, which are stored inplot2.pklandplot3.pkl. By pickling theirfigureobjects, the plots can still be modified by opening the PyPDF files in a text editor and running them as Python scripts.Theplot2.pdffile looks as follows when opened in a text editor:#%PDF-1.4 26 0 obj << /Type /EmbeddedFile /Length        269 >> streamimportpypdfplot.backend.unpackimportmatplotlib.pyplotaspltfrompickleimportloadwithopen('plot2.pkl','rb')asf:fig=load(f)plt.figure(fig.number)## Plot customizations go here...plt.savefig('plot2.pdf',pack_list=['plot2.pkl'])"""
--- Do not edit below ---
endstream
endobj
1 0 obj

<< ... >>

startxref
79815
%%EOF
0000080463 LF
PyPDF-1.0
"""The individual plots can now be customized by modifying these generating scripts. By pickling the figure some flexibility is lost, but it should still be relatively straightforward to change e.g. line colors, font sizes, etc.Pickling the figure can be advantageous even if it is not generated inside an iterative loop, for example if the generating script is very slow or relies on large external data files. By pickling the figure, the external files do not have to be re-processed and do not have to be embedded in the PyPDF file.In such cases it is possible to forcepypdfplotto pickle the plot by passing theforce_pickle=Truekeyword tosavefig().Multiple pagesThe other solution is to add the multiple plots to asinglePyPDF file as multiple pages. This is done by callingsavefig()with themultiple='add_page'keyword for every new plot, and finally callingsavefig()one more time with the keywordmultiple='finalize'to finalize the PyPDF file.When themultiple='add_page'keyword is specified, every other argument tosavefig()is ignored. The filename of the output PyPDF file is thus determined by the filename that is passed to the finalsavefig()call (withmultiple=finalize), so an empty string can be passed for the calls withmultiple='add_page'.Any other arguments, such as thepack_listlist, should also be passed to the finalizingsavefig()call.Revisiting the previous example, but this time adding the multiple plots as separate pages, the updated script looks as follows:importpypdfplot.backendimportmatplotlib.pyplotaspltimportnumpyasnpx=np.arange(-10,20,0.1)forninrange(3):y=x**(n+1)plt.plot(x,y)plt.savefig('',multiple='add_page')plt.clf()plt.savefig('multi_page.pdf',multiple='finalize')The output of this results in a single PyPDF file with three pages:PyPDF file specificationThis page describes the PyPDF file specification version 1.0MotivationA PyPDF file complies with the PDF specification and is additionally a valid Python file, in addition to meeting some other restrictions. Adhering to both file formats, the PyPDF file can be thought of as either a Python file with some extra restrictions to make it PDF compliant, or a PDF file with some extra restrictions to make it readable by a Python interpreter. Of the two file types, the PDF file has the most complex structure, so the simplest way to think of the PyPDF file is as a PDF file with extra restrictions.In essence, a PyPDF file is a PDF file with the script that produced the PDF file embedded.The PDF file structure consists of so called “blocks” which can be placed in arbitrary order in the PDF document. In order to pass as a PyPDF file, the block containing the filestream of the embedded generating script must be in ASCII (i.e. no encoding filter may be used) and must be the first object described in the document (the objectnumberis not specified and can be chosen freely).With these restrictions, the PDF file would so far look something like this:%PDF-1.4
%Ã¶Ã¤Ã¼ÃŸ
24 0 obj
<< /Type /EmbeddedFile /Length 187 >>
stream
import pypdfplot.backend
import matplotlib.pyplot as plt

<< ... >>The characters on the second line,%Ã¶Ã¤Ã¼ÃŸ, are usually included in a PDF file to indicate that we are dealing with a binary file as opposed to a ASCII file. However, in order to be a Python file itshouldbe read as ASCII, so this line is simply omitted.In the remaining first 4 lines (from%PDFtostream), all linebreaks are replaced by spaces so that there is only a single line preceding the generating Python script. This single line is then preceded by a pound sign (#) so that it is read as comment by the Python interpreter. The PDF specification states that the PDF header (%PDF) must appear within the first 1024 bytes of the document, so prepending the#is allowed within the PDF specification.When the generating script is later modified, the length of its filestream will likely change. If the string of the new length is larger than the old string, all bytes in the document must be shifted to accomodate the increased size of the length string. In order to obviate this byte shift, the PyPDF file must leave some additional space between the/Lengthkeyword and the number string. A total space of 10 digits is recommended.The file now looks as follows:#%PDF-1.4 24 0 obj << /Type /EmbeddedFile /Length        187 >> streamimportpypdfplot.backendimportmatplotlib.pyplotasplt<<...>>The filestream of the generating script consists of the generating Python script itself, in addition to a line with triple quotes ("""), followed by a linebreak, so that all PDF blocks that follow are read as a comment block by the Python interpreter, optionally followed by a warning string that warns the user against edits in the comment block:#%PDF-1.4 24 0 obj << /Type /EmbeddedFile /Length        187 >> streamimportpypdfplot.backendimportmatplotlib.pyplotaspltimportnumpyasnpx=np.arange(-10,20,0.1)y=x**2plt.plot(x,y)plt.savefig('example.pdf')"""
--- Do not edit below ---
endstream
endobj
1 0 obj

<< ... >>Because the use of non-ASCII binary characters is precluded in the PyPDF file, all stream objects in the PDF file must be encoded with one of the ASCII filters (either/ASCIIHexDecodefilter or/ASCII85Decodefilter).Moreover, to remain PEP-compliant, lines may not exceed the length of 79 characters.Finally, at the end of the PyPDF file some lines are added in addition to those specified by the PDF specification. A normal PDF file will end with the end-of-file marker%%EOF. The EOF marker must appear within the last 1024 bytes of the document, so it is allowed within the PDF specification to add a couple of lines after that.Specifically, three lines are added after the PDF EOF marker:The first line is a 10-digit integer with leading zeros giving the filesize in bytes, followed by a whitespace andLForCRLFdepending on which type of linebreaks were used to produce the document.The next line contains the PyPDF version number in the formatPyPDF-#.#, where the#’s are replaced by the major and minor version numbers.Finally, a line with triple quotes ("""), followed by a linebreak, is added to close the comment block that was opened immediately following the generating script.For example:<<...>>startxref9542%%EOF0000010149LFPyPDF-1.0"""By reporting the filesize inside the PyPDF file, the PDF xref table can be corrected after modification of the generating script in the PyPDF file (SeePyPDF compliance types). Similarly, by explicitly stating which linebreak character was used (LForCRLF), the PyPDF file can be repaired if the linebreaks were inadvertedly replaced.As a final safety measure, the PyPDF file must identify the filename of the embedded generating script and specify what PyPDF version was used in the/Rootobject of the PDF file. This is done by specifying the/PyFileand/PyPDFVersionkeys respectively in the dictionary of the/Rootobject:<<...>>50obj<</Type/Catalog/Pages10R/PageMode/UseAttachments/PyFile(example.py)/PyPDFVersion(1.0)>><<...>>trailer<</Size25/Root50R/Info20R>><<...>>PyPDF file structureThe PyPDF file structure consists both of PDF and Python elements. While the PDF file structure is well documented in the PDF specification, in the PyPDF file certain functional elements blend into each other depending on if it is interpreted as PDF or as Python file. It is therefore useful to redefine the structural elements in the framework of the combined PyPDF file.The PyPDF file can thus be considered as comprised of the following consecutive elements:PyPDF header:A single line starting with#, followed by the%PDFheader, followed by the filestream header of the embedded generating script, without any linebreaks. A 10-character wide space should be reserved for the value of the/Lengthkeyword.Generating script:The Python script that produces the PyPDF output. This element is itself a regular Python file without any PDF components. The generating script must end with a linebreak.PDF remainder:The remainder of the PDF document, starting from the triple quotes as part of the filestream of the generating script, and ending at the%%EOFPDF end-of-file marker. By appending the PyPDF header, generating script, and the PDF remainder, not including the initial#, a regular PDF file is obtained. The PDF/Rootobject should contain/PyFileand/PyPDFVersionentries specifying the generating script and PyPDF version.PyPDF trailer:A line with the 10-digit filesize in bytes including leading zeros, followed by a string specifying what linebreak character  was used (LF/CRLF), followed by a new line with the PyPDF version number in the formatPyPDF-#.#, followed by a new line with triple quotes, ending with a linebreak.The following table identifies each of these elements in the example PyPDF file:Element:Example:PyPDFheader#%PDF-1.4 24 0 obj << /Type /EmbeddedFile /Length        187 >> streamGeneratingscriptimportpypdfplot.backendimportmatplotlib.pyplotaspltimportnumpyasnpx=np.arange(-10,20,0.1)y=x**2plt.plot(x,y)plt.savefig('example.pdf')PDFremainder"""
--- Do not edit below ---
endstream
endobj

<< ... >>

startxref
9033%%EOFPyPDFtrailer0000009636 LF
PyPDF-1.0
"""PyPDF compliance typesThe first time a generating script is run, it does not have any PDF component yet and so it is technically not a PyPDF file. Still it is closer to a PyPDF file than most other odd Python file. Specifically, it can be easilyturned intoa PyPDF file by running the script.In the same vain, during its life the PyPDF file may lose some compliance to the PyPDF specification in other ways that can be restored by running the generating script or external programs. This section discusses the different ways in which this can happen by identifying the different PyPDFcompliance types. We distinguish 4 of these types:Generating script:Pure Python file that lacks any PDF structure.Compliant PyPDF file:Fully PyPDF compliant, both valid Python file and PDF compliant.Stale PyPDF file:Mixed Python/PDF file that lost some PDF compliance.Severed PyPDF file:Pure PDF file that lost its Python structure, but still has the generating script embedded.Thegenerating scriptand thecompliantPyPDF file have both been discussed at length already. In the following thestaleandseveredPyPDF files will be discussed in more detail.Stale PyPDF fileA Stale PyPDF file is a PyPDF file that lost its PDF compliance because the Python script was modified. This will happen whenever the Python script is updated and is therefore almost always intented. When the generating script – placed at the top of the document – is modified, the byte address of the following PDF blocks will shift. Because of this shift, thexref-table at the end of the PDF document is compromised, resulting in the file losing its PDF compliance.The adresses in thexref-table are only off by a constant shift, so if the size of this shift could be obtained, thexref-table could be corrected and the file made PyPDF compliant once again. To facilitate retrieval of the address shift, the filesize in bytes is printed in the PyPDF trailer. By comparing the recorded filesize with the current filesize, the shift can be determined and thexref-table corrected. This correction is applied every time thesavefig()function is called with thepypdfplotbackend loaded. This means that converting a stale PyPDF file to a compliant PyPDF file is done by simply running the script.For correcting thexref-table it is assumed that the corruption was the result of the Python script changing in size. Another process by which thexref-table may be corrupted, is that UNIX linebreaks are inadvertedly replaced by Windows linebreaks or vice versa. This can happen e.g. when the PyPDF file is committed to a Github repository, without PDF-files being explicitly set to binary files.Safeguards are in place to correct for this as well: The PyPDF file must specify what linebreak it used in the PyPDF trailer and must use this linebreak character consistently throughout the document. A correction procedure can compare the reported linebreak character with the ones found in the file and correct if necessary. This correction should be performedbeforecorrecting for the change in size of the generating script.Severed PyPDF fileA severed PyPDF file is a PyPDF file that lost its Python structure, usually because it is saved by a PDF reader that is unaware of the PyPDF file format. There is normally no reason the PyPDF file should lose its Python structure, and when it does it is usually unintended. Just like the stale PyPDF however, there are some builtin safeguards to convert the severed PyPDF back into a compliant PyPDF file.Since the severed PyPDF is still PDF compliant, it only needs to be restructured with the following two basic rules: the generating Python script (which is must still embedded in the PDF file) has to be placed on top, and only ASCII characters may be used in writing the corrected document. To figure out which embedded file is the generating script, a PyPDF file must always specify the filename of the generating script with the/PyFilekey in the PDF/Rootdict.A severed PyPDF file can be fixed by passing its filename tofix_pypdf()in a Python script:frompypdfplotimportfix_pypdffix_pypdf('severed_pypdf.pdf','compliant_pypdf.pdf')Thefix_pypdfcommand can also be ran from the command line.The different compliance types and how they can be converted into fully compliant PyPDF files are summarized in the figure below:Changelogv0.6.5
- Previous patch introduced a new problem with the “Do not edit below” string. This is now solved.v0.6.4
- Prevent deletion of output when input is .pdf
- Fixes additional PyPDF4 compatibility issuesv0.6.3Fix compatibility with PyPDF4 v1.27.0v0.6.2Fix missing installation of backendFix some links in docsv0.6.1Documentation completely updatedRemoved legacypublish()function, only works asMatplotlibbackend now.Changedauto_extract()tounpack()Changedfile_listtopack_listAdded__PYPDFVERSION__as canonical version no.Addedpw.setPyPDFVersion()tofix_pypdf()v0.6.0First official release