Package: pdf2doi
pdf2doipdf2doiis a Python library/command-line tool to automatically extract the DOI or other identifiers (e.g. arXiv ID) starting from the .pdf file of a publication
(or from a folder containing several .pdf files), and to retrieve bibliographic information.
It exploits several methods (see below for detailed description) to find a valid identifier of a pdf file, and it validates any result
via web queries to public archives (e.g.http://dx.doi.org).
The validation process also returns raw bibtex infos, which can be used for further processing, such as generating BibTeX entries (pdf2bib) or
automatically renaming pdf files (pdf-renamer).pdf2doi can be used either fromcommand line, or inside yourpython scriptor, only for Windows, directly from theright-click context menuof a pdf file or a folder.Latest stable versionThe latest stable version ofpdf2doiis the1.5. Seeherefor the full change log.[v1.5] - 2022-12-31Main changesThe librarytextracthas been removed from the required dependencies because it often creates problems during installation (due to conflicts between library versions),
and because it generally requires installing many other dependencies which are not needed bypdf2doi. The user can still decide to installtextract==1.6.4if desired.pdf2doiwill usetextractonly if it is installed.pdf2doinow stores any found identifier into a tag called/pdf2doi_identifier(previously was/identifier).AddedThe librarypdfmineris now directly used bypdf2doito extract the text from a pdf file (instead of doing it indirectly viatextract)An additional method to find the title of a pdf file, based on the librarypymupdf, has been added .[Issue https://github.com/MicheleCotrufo/pdf2doi/issues/21]: When an arXiv ID is found, a corresponding DOI is also returned when available. This could be either the standard arXiv DOI (see alsohere),
or the DOI of the corresponding journal publication. This behavior can be disabled by adding the optional command-no_arxiv2doito thepdf2doiinvocation.[Issue https://github.com/MicheleCotrufo/pdf2doi/issues/22]: The functionget_pdf_text(finders.py) has been modified to allow the libraryPyPDF2to extract also the text of any annotation/comment present in the pdf file.FixedPotential titles of the papers were often not correctly found, because the functionfind_possible_titles()(finders.py) would mistakenly disregard all the results if one of the three methods (pdftitle, PyPDF2, filename) generated an error.Fixed bug in the functionadd_metadata()(finders.py). In previous versions, some of the pre-existing metadata were not preserved when a new one was added  (Commit).InstallationUse the package manager pip to install pdf2doi.pipinstallpdf2doi==1.5The librarytextractprovides additional ways to analyze pdf files, and it is sometimes more powerful thanPyPDF2, but it comes with a large overhead of additional required dependencies, and sometimes it generates version conflicts.
The user can decide whether to install it or not.pdf2doiwill only try to use this library if it detects that it is installed.
To install it,pipinstalltextract==1.6.4
pipinstallpdfminer.six==20191110Under Windows, after installation ofpdf2doiit is also possible to addshortcuts to the right-click context menu.Used byHere is a list of applications/repositories that make use ofpdf2doi. If you usepdf2doiin your application and you wish to add it to this list, send me a message.mendeley-migrationpub2sumDataIngestpdf2bibpdf-renamerTable of ContentsInstallationDescriptionUsageCommand line usageManually associate the correct identifier to a file from command lineUsage inside a python scriptManually associate the correct identifier to a fileInstalling the shortcuts in the right-click context menu of Windows-ContributingLicenseAcknowledgmentDonatingDescriptionAutomatically associating a DOI or other identifiers (e.g. arXiv ID) to a pdf file can be either a very easy or a very difficult
(sometimes nearly impossible) task, depending on how much care was placed in crafting the file. In the simplest case (which typically works with most recent publications)
it is enough to look into the file metadata. For older publications, the identifier is often found within the pdf text and it can be
extracted with the help of regular expressions. In the unluckiest cases, the only method left is to google some details of the publication
(e.g. the title or parts of the text) and hope that a valid identifier is contained in one of the first results.pdf2doiapplies sequentially all these methods (starting from the simplest ones) until a valid identifier is found and validated.
Specifically, for a given .pdf file it will, in order,Look into the metadata of the .pdf file (extracted via the libraryPyPDF2) and check if any of them contains a string that matches the pattern of
a DOI or an arXiv ID. Priority is given to metadata which contain the word 'doi' in their label.Check if the name of the pdf file contains any sub-string that matches the pattern of
a DOI or an arXiv ID.Scan the text inside the .pdf file, and check for any string that matches the pattern of
a DOI or an arXiv ID. The text is extracted with the librariesPyPDF2andpdfminer. If the librarytextractis installed,pdf2doiwill try to use that too.Try to find possible titles of the publication. In the current version, possible titles are identified via
the librariespdftitleandPyMuPDF, and by the file name. For each possible title a google search
is performed and the plain text of the first results is scanned for valid identifiers.As a last desperate attempt, the first N=1000 characters of the pdf text are used as a query for
a google search. The plain text of the first results is scanned for valid identifiers.Any time that a potential identifier is found, it is also validated by performing a query to a relevant website (e.g.,http://dx.doi.orgfor DOIs andhttp://export.arxiv.orgfor arxiv IDs).
This validation process also returns rawBibTeXinfo when the identifier is valid.When a valid identifier is found with any method different than the first one, the identifier is also stored inside the metadata of
the pdf file. In this way, future lookups of this same file will be able to extract the identifier with the
first method, speeding up the search (This feature can be disabled by the user, in case edits to the pdf file are not desired).The library is far from being perfect. Often, especially for old publications, none of the currently implemented methods will work. Other times the wrong DOI might be extracted: this can happen, for example,
if the DOI of another paper is present in the pdf text and it appears before the correct DOI. A quick and dirty solution to this problem is to look up the identifier manually and then add it to the metadata
of the file, with the methods shownhere(from python console) orhere(from command line).
In this way,pdf2doiwill always retrieve the correct DOI when analyzing this same file in the future, which can be useful whenpdf2doiis used  to automatize
bibliographic procedures for a large number of files (e.g. viapdf2biborpdf-renamer).Currently, only the format of arXiv identifiers in use after1 April 2007is supported.Usagepdf2doi can be used either as astand-alone applicationinvoked from the command line, or byimporting it in your python projector, only for Windows,
directly from theright-click context menuof a pdf file or a folder.Command line usagepdf2doican be invoked directly from the command line, without having to open a python console.
The simplest command-line invokation is$ pdf2doi 'path/to/target'wheretargetis either a valid pdf file or a directory containing pdf files. Adding the optional command '-v' increases the output verbosity,
documenting all steps.
For example, when targeting the folderexampleswe get the following output$ pdf2doi ".\examples" -v
[pdf2doi]: Looking for pdf files in the folder ....
[pdf2doi]: Found 4 pdf files.
[pdf2doi]: ................
[pdf2doi]: Trying to retrieve a DOI/identifier for the file: .\examples\chaumet_JAP_07.pdf
[pdf2doi]: Method #1: Looking for a valid identifier in the document infos...
[pdf2doi]: Could not find a valid identifier in the document info.
[pdf2doi]: Method #2: Looking for a valid identifier in the file name...
[pdf2doi]: Could not find a valid identifier in the file name.
[pdf2doi]: Method #3: Looking for a valid identifier in the document text...
[pdf2doi]: Extracting text with the library PyPdf...
[pdf2doi]: Text extracted succesfully. Looking for an identifier in the text...
[pdf2doi]: Validating the possible DOI 10.1063/1.2409490 via a query to dx.doi.org...
[pdf2doi]: The DOI 10.1063/1.2409490 is validated by dx.doi.org.
[pdf2doi]: A valid DOI was found in the document text.
[pdf2doi]: Trying to add the tag '/pdf2doi_identifier'-> '10.1063/1.2409490' into the metadata of the file '.\chaumet_JAP_07.pdf'...
[pdf2doi]: The tag '/pdf2doi_identifier'-> '10.1063/1.2409490' was added succesfully to the metadata of the file '.\chaumet_JAP_07.pdf'...
[pdf2doi]: 10.1063/1.2409490
[pdf2doi]: ................
[pdf2doi]: Trying to retrieve a DOI/identifier for the file: .\examples\paper12.2009_unknown_040916_440842.pdf
[pdf2doi]: Method #1: Looking for a valid identifier in the document infos...
[pdf2doi]: Could not find a valid identifier in the document info.
[pdf2doi]: Method #2: Looking for a valid identifier in the file name...
[pdf2doi]: Could not find a valid identifier in the file name.
[pdf2doi]: Method #3: Looking for a valid identifier in the document text...
[pdf2doi]: Extracting text with the library PyPdf...
[pdf2doi]: Text extracted succesfully. Looking for an identifier in the text...
[pdf2doi]: Could not find a valid identifier in the document text extracted by PyPdf.
[pdf2doi]: Extracting text with the library pdfminer...
[pdf2doi]: Text extracted succesfully. Looking for an identifier in the text...
[pdf2doi]: Validating the possible DOI 10.1037/a0015278 via a query to dx.doi.org...
[pdf2doi]: The DOI 10.1037/a0015278 is validated by dx.doi.org.
[pdf2doi]: A valid DOI was found in the document text.
[pdf2doi]: Trying to add the tag '/pdf2doi_identifier'-> '10.1037/a0015278' into the metadata of the file '.\paper12.2009_unknown_040916_440842.pdf'...
[pdf2doi]: The tag '/pdf2doi_identifier'-> '10.1037/a0015278' was added succesfully to the metadata of the file '.\paper12.2009_unknown_040916_440842.pdf'...
[pdf2doi]: 10.1037/a0015278
[pdf2doi]: ................
[pdf2doi]: Trying to retrieve a DOI/identifier for the file: .\examples\PhysRevLett.116.061102.pdf
[pdf2doi]: Method #1: Looking for a valid identifier in the document infos...
[pdf2doi]: Could not find a valid identifier in the document info.
[pdf2doi]: Method #2: Looking for a valid identifier in the file name...
[pdf2doi]: Could not find a valid identifier in the file name.
[pdf2doi]: Method #3: Looking for a valid identifier in the document text...
[pdf2doi]: Extracting text with the library PyPdf...
[pdf2doi]: Text extracted succesfully. Looking for an identifier in the text...
[pdf2doi]: Standardised DOI: 10.1103/PhysRevLett.116.061102 -> 10.1103/physrevlett.116.061102
[pdf2doi]: Validating the possible DOI 10.1103/physrevlett.116.061102 via a query to dx.doi.org...
[pdf2doi]: The DOI 10.1103/physrevlett.116.061102 is validated by dx.doi.org.
[pdf2doi]: Standardised DOI: 10.1103/PhysRevLett.116.061102 -> 10.1103/physrevlett.116.061102
[pdf2doi]: A valid DOI was found in the document text.
[pdf2doi]: Trying to add the tag '/pdf2doi_identifier'-> '10.1103/physrevlett.116.061102' into the metadata of the file '.\PhysRevLett.116.061102.pdf'...
[pdf2doi]: The tag '/pdf2doi_identifier'-> '10.1103/physrevlett.116.061102' was added succesfully to the metadata of the file '.\PhysRevLett.116.061102.pdf'...
[pdf2doi]: 10.1103/physrevlett.116.061102
[pdf2doi]: ................
[pdf2doi]: Trying to retrieve a DOI/identifier for the file: .\examples\s41586-019-1666-5.pdf
[pdf2doi]: Method #1: Looking for a valid identifier in the document infos...
[pdf2doi]: Could not find a valid identifier in the document info.
[pdf2doi]: Method #2: Looking for a valid identifier in the file name...
[pdf2doi]: Could not find a valid identifier in the file name.
[pdf2doi]: Method #3: Looking for a valid identifier in the document text...
[pdf2doi]: Extracting text with the library PyPdf...
[pdf2doi]: Text extracted succesfully. Looking for an identifier in the text...
[pdf2doi]: Validating the possible DOI 10.1038/s41586-019-1666-5 via a query to dx.doi.org...
[pdf2doi]: The DOI 10.1038/s41586-019-1666-5 is validated by dx.doi.org.
[pdf2doi]: A valid DOI was found in the document text.
[pdf2doi]: Trying to add the tag '/pdf2doi_identifier'-> '10.1038/s41586-019-1666-5' into the metadata of the file '.\s41586-019-1666-5.pdf'...
[pdf2doi]: The tag '/pdf2doi_identifier'-> '10.1038/s41586-019-1666-5' was added succesfully to the metadata of the file '.\s41586-019-1666-5.pdf'...
[pdf2doi]: 10.1038/s41586-019-1666-5
[pdf2doi]: ................
DOI             10.1063/1.2409490                        .\chaumet_JAP_07.pdf

DOI             10.1037/a0015278                         .\paper12.2009_unknown_040916_440842.pdf

DOI             10.1103/physrevlett.116.061102           .\PhysRevLett.116.061102.pdf

DOI             10.1038/s41586-019-1666-5                .\s41586-019-1666-5.pdfEvery line which begins with[pdf2doi]is omitted when the optional command '-v' is absent.
In the final output, the first column specifies the kind of identifier (currently either 'DOI' or 'arxiv'), the second column contains the found DOI/identifier, and the third column contains the file path.A list of all optional arguments can be generated bypdf2doi --h$ pdf2doi --h
usage: pdf2doi [-h] [-v] [-nws] [-nwv] [-nostore] [-no_arxiv2doi] [-id IDENTIFIER] [-google GOOGLE_RESULTS] [-s FILENAME_IDENTIFIERS] [-clip] [-install--right--click] [-uninstall--right--click] [path ...]

Retrieves the DOI or other identifiers (e.g. arXiv) from pdf files of a publications.

positional arguments:
  path                  Relative path of the target pdf file or of the targe folder.

options:
  -h, --help            show this help message and exit
  -v, --verbose         Increase verbosity. By default (i.e. when not using -v), only a table with the found identifiers will be printed as output.
  -nws, --no_web_search
                        Disable any method to find identifiers which requires internet searches (e.g. queries to google).
  -nwv, --no_web_validation
                        Disable the online validation of identifiers (e.g., via queries to http://dx.doi.org/).
  -nostore, --no_store_identifier_metadata
                        By default, anytime an identifier is found it is added to the metadata of the pdf file (if not present yet). By using this additional option, the identifier is not stored in the file
                        metadata.
  -no_arxiv2doi         If a valid arXiv ID is found for a given .pdf file, by default pdf2doi will try to also look for a DOI (either because the paper has been published in a journal or because arXiv has
                        assigned to it a DOI of the form "10.48550/arXivID"). By adding this command, the arXiv ID is instead always returned.
  -id IDENTIFIER        Stores the string IDENTIFIER in the metadata of the target pdf file, with key '/pdf2doi_identifier'. Note: when this argument is passed, all other arguments (except for the path to the
                        pdf file) are ignored.
  -google GOOGLE_RESULTS
                        Set how many results should be considered when doing a google search for the DOI (default=6).
  -s FILENAME_IDENTIFIERS, --save_identifiers_file FILENAME_IDENTIFIERS
                        Save all the identifiers found in the target folder in a text file inside the same folder with name specified by FILENAME_IDENTIFIERS. This option is only available when a folder is
                        targeted.
  -clip, --save_doi_clipboard
                        Store all found DOI/identifiers into the clipboard.
  -install--right--click
                        Add a shortcut to pdf2doi in the right-click context menu of Windows. You can copy the identifier and/or bibtex entry of a pdf file (or all pdf files in a folder) into the clipboard by
                        just right clicking on it! NOTE: this feature is only available on Windows.
  -uninstall--right--click
                        Uninstall the right-click context menu functionalities. NOTE: this feature is only available on Windows.```

#### Manually associate the correct identifier to a file from command line
Sometimes it is not possible to retrieve a DOI/identifier automatically, or maybe the one that is retrieved is not the correct one. In these (hopefully rare) occasions
it is possible to manually add the correct DOI/identifier to the pdf metadata, by using the ```-id``` argument,$ pdf2doi "path	o\pdf" -id "doi1234"This creates a new metadata in the pdf file with label '/pdf2doi_identifier' and containing the string ```doi1234```.  Future lookups of this same file via ```pdf2doi``` (in particular when used by other tools such as [pdf2bib](https://github.com/MicheleCotrufo/pdf2bib) or
[pdf-renamer](https://github.com/MicheleCotrufo/pdf-renamer)) will then return the correct identifier and BibTeX infos.

### Usage inside a python script
```pdf2doi``` can also be used as a library within a python script. The function ```pdf2doi.pdf2doi``` is the main point of entry. The function looks for the identifier of a pdf file by applying all the available methods. 
The first input argument must be a valid path (either absolute or relative) to a pdf file or to a folder containing pdf files. 
The same optional arguments available in the command line operation are now available via the methods ```set``` and ```get``` of the object ```pdf2doi.config```
For example, we can scan the folder [examples](/examples) while soppressing output verbosity by, 

```python
>>> import pdf2doi
>>> pdf2doi.config.set('verbose',False)
>>> results = pdf2doi.pdf2doi('.\examples')A full list of the library settings can be printed by the methodpdf2doi.config.print()>>>importpdf2doi>>>pdf2doi.config.print()verbose:True(bool)separator:\(str)method_dxdoiorg:application/citeproc+json(str)webvalidation:True(bool)websearch:True(bool)numb_results_google_search:6(int)N_characters_in_pdf:1000(int)save_identifier_metadata:True(bool)replace_arxivID_by_DOI_when_available:True(bool)The output of the functionpdf2doiis a list of dictionaries (or just a single dictionary if a single file was targeted). Each dictionary has the following keysresult['identifier']=DOIorotheridentifier(orNoneifnothingisfound)result['identifier_type']=stringspecifyingthetypeofidentifier(e.g.'doi'or'arxiv')result['validation_info']=Additionalinfoonthepaper.Ifconfig.get('webvalidation')=True,thenresult['validation_info']willtypicallycontainrawbibtexdataforthispaper.OtherwiseitwilljustcontainTrueresult['path']=pathofthepdffileresult['method']=methodusedtofindtheidentifierFor example, the DOIs/identifiers of each file can be printed by>>>forresultinresults:>>>print(result['identifier'])10.1016/0021-9991(86)90093-810.1063/1.240949010.1103/PhysRevLett.116.06110210.1038/s41586-019-1666-5By default, everytime that a valid DOI/identifier is found, it is stored in the metadata of the pdf file. In this way, subsequent lookups of the same folder/file will be much faster.
This behaviour can be removed (e.g. if the user does not want or cannot edit the files) by setting save_identifier_metadata to False, via>>>pdf2doi.config.set('save_identifier_metadata',False)Manually associate the correct identifier to a fileSimilarly to what describedabove, it is possible to associate a (manually found)
identifier to a pdf file also from within python, by using the functionpdf2doi.add_found_identifier_to_metadata:>>>importpdf2doi>>>pdf2doi.add_found_identifier_to_metadata(path_to_pdf_file,identifier)this creates a new metadata in the pdf file with label '/pdf2doi_identifier' and containing the stringidentifier.Installing the shortcuts in the right-click context menu of WindowsThis functionality is only available on Windows (and so far it has been tested only on Windows 10). It adds additional commands to the context menu of Windows
which appears when right-clicking on a pdf file or on a folder.The different menu commands allow to copy the paper(s) identifier(s) into the system clipboard, or also to manually
set the identifier of a pdf file (see alsohere).To install this functionality, first installpdf2doivia pip (as described above), then open a command promptwith administrator rightsand execute$ pdf2doi  -install--right--clickTo remove it, simply run (again from a terminal with administrator rights)$ pdf2doi  -uninstall--right--clickIf it is not possible to run this command from a terminal with administrator rights, the batch filesherecan be alternatively used (see readme.MD file in the same folder for instructions), although it is still required to have
admnistrator rights.NOTE: when multiple pdf files are selected, and the right-click context menu commands are used,pdf2doiwill be called separately for each file, and thus
only the info of the last file will be stored in the clipboard. In order to copy the info of multiple files it is necessary to save them in a folder and right-click on the folder.ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.AcknowledgmentI am thankful to my friend and colleague Yarden Mazor for leading the beta-testing efforts for this project.DonatingIf you find this library useful (or amazing!), please consider making donations on my behalf to organizations that advocate for and promote free dissemination of science, such asarXivSci-HubWikipediaLicenseMIT