Package: grobid-client-python
Simple python client for GROBID REST servicesThis Python client can be used to process in an efficient concurrent manner a set of PDF in a given directory by theGROBIDservice. It includes a command line for processing PDF on a file system and write results in a given output directory and a library for import in other python scripts. The client can also process similarly a list of files with reference strings (one per line) and patents in XML ST36 formats.Before you startPlease be aware that, at the moment,grobid does not support Windows.
If you are a Windows user, don't worry. You can stillrun grobid
via Docker.Build and runYou need first a runninggrobidservice, latest stable version, see thedocumentationfor installation.
By default, it is assumed that the server will run on the addresshttp://localhost:8070.
You can change the server address by editing the fileconfig.json, see below.RequirementsThis client has been developed and was tested with Python3.5-3.9and should work with any higher3.*versions. It usesrequestsas dependency beyond the Standard Python Library.InstallGet the github repo:git clone https://github.com/kermitt2/grobid_client_pythoncd grobid_client_pythonpython3 setup.py installThere is nothing more needed to start using the python command lines, see the next section.Usage and optionsThe call to the script can normally be realized interchangeably withpython3 -m grobid_client.grobid_clientor simplygrobid_client.usage: grobid_client [-h] [--input INPUT] [--output OUTPUT] [--config CONFIG]
                     [--n N] [--generateIDs] [--consolidate_header]
                     [--consolidate_citations] [--include_raw_citations]
                     [--include_raw_affiliations] [--force] [--teiCoordinates]
                     [--verbose]
                     service

Client for GROBID services

positional arguments:
  service               one of ['processFulltextDocument',
                        'processHeaderDocument', 'processReferences',
                        'processCitationList','processCitationPatentST36',
                        'processCitationPatentPDF']

optional arguments:
  -h, --help            show this help message and exit
  --input INPUT         path to the directory containing PDF files or .txt
                        (for processCitationList only, one reference per line)
                        to process
  --output OUTPUT       path to the directory where to put the results
                        (optional)
  --config CONFIG       path to the config file, default is ./config.json
  --n N                 concurrency for service usage
  --generateIDs         generate random xml:id to textual XML elements of the
                        result files
  --consolidate_header  call GROBID with consolidation of the metadata
                        extracted from the header
  --consolidate_citations
                        call GROBID with consolidation of the extracted
                        bibliographical references
  --include_raw_citations
                        call GROBID requesting the extraction of raw citations
  --include_raw_affiliations
                        call GROBID requestiong the extraciton of raw
                        affiliations
  --force               force re-processing pdf input files when tei output
                        files already exist
  --teiCoordinates      add the original PDF coordinates (bounding boxes) to
                        the extracted elements
  --segmentSentences    segment sentences in the text content of the document
                        with additional 
 elements
  --verbose             print information about processed files in the consoleExamples:> grobid_client --input ~/tmp/in2 --output ~/tmp/out processFulltextDocumentThis command will process all the PDF files present under the input directory recursively (files with extension.pdfonly) with theprocessFulltextDocumentservice of GROBID, and write the resulting XML TEI files under the output directory, reusing the file name with a different file extension (.grobid.tei.xml), using the default10concurrent workers.If--outputis omitted, the resulting XML TEI documents will be produced alongside the PDF in the--inputdirectory.> grobid_client --input ~/tmp/in2 --output ~/tmp/out --n 20 processHeaderDocumentThis command will process all the PDF files present in the input directory (files with extension.pdfonly) with theprocessHeaderDocumentservice of GROBID, and write the resulting XML TEI files under the output directory, reusing the file name with a different file extension (.grobid.tei.xml), using20concurrent workers.By default if an existing.grobid.tei.xmlfile is present in the output directory corresponding to a PDF in the input directory, this PDF will be skipped to avoid reprocessing several times the same PDF. To force the processing of PDF and over-write of existing TEI files, use the parameter--force.processCitationListdoes not take a repertory of PDF as input, but a repertory of.txtfiles, with one reference raw string per line, for example:> grobid_client --input resources/test_txt/ --output resources/test_out/ --n 20 processCitationListThe following command example will process all the PDF files present in the input directory and add bounding box coordinates (--teiCoordinates) relative to the original PDFs for the elements listed in the config file. It will also segment the sentences (--segmentSentences, this is a "layout aware" sentence segmentation) in the identified paragraphs with bounding box coordinates for the sentences.> grobid_client --input ~/tmp/in2 --output ~/tmp/out --teiCoordinates --segmentSentences processFulltextDocumentThe fileexample.pygives an example of usage as a library, from a another python script.Using the client in your pythonImport and call the client as follow:fromgrobid_client.grobid_clientimportGrobidClientclient=GrobidClient(config_path="./config.json")client.process("processFulltextDocument","/mnt/data/covid/pdfs",n=20)See alsoexample.py.Configuration of the clientThere are a few parameters that can be set with theconfig.jsonfile.grobid_serverindicates the URL of the GROBID server to be used by the client.batch_sizeis the the size of the pool of threads used by ThreadPoolExecutor, you normally don't want to change this. This should be a high number (default 1000) - but not too high to protect the memory on the machine running the client. This should not be confused with the concurrency parameternwhich indicates how many parallel requests can be send to GROBID.sleep_timeindicates in seconds the time to wait for sending a new request to GROBID when the server indicates that all its threads are currently used. The client need to re-send the query after a wait time that will allow the server to free some threads. This wait time usually depends on the service and the capacities of the server, we suggest 5-10 seconds for theprocessFulltextDocumentservice and 2 seconds forprocessHeaderDocumentservice.timeoutis a client side timeout - the process on server side will still be running until the server finished the task or the server timeout is reached.coordinatesindicates the structure XML elements that should contains PDF coordinates when the parameters--teiCoordinatesis used seeherefor more details.Here is the defaultconfig.jsonfile for the client:{
    "grobid_server": "http://localhost:8070",
    "batch_size": 1000,
    "sleep_time": 5,
    "timeout": 60,
    "coordinates": [ "persName", "figure", "ref", "biblStruct", "formula", "s" ]
}BenchmarkingFull text processing of136 PDF(total 3443 pages, in average 25 pages per PDF) on Intel Core i7-4790K CPU 4.00GHz, 4 cores (8 threads), 16GB memory,nbeing the concurrency parameter:nruntime (s)s/PDFPDF/s1209.01.540.652112.00.821.21380.40.591.69562.90.462.16855.70.412.441055.30.402.45As complementary info, GROBID processing of header of the 136 PDF and withn=10takes 3.74 s (15 times faster than the complete full text processing because only the two first pages of the PDF are considered), 36 PDF/s.In similar conditions, extraction and structuring of bibliographical references takes 26.9 s (5.1 PDF/s).Processing of 3500 raw bibliographical take 4.3 s withn=10(814 references parsed per second).License and contactDistributed underApache 2.0 license.Main author and contact: Patrice Lopez (patrice.lopez@science-miner.com)