Package: nougat-ocr
Nougat: Neural Optical Understanding for Academic DocumentsThis is the official repository for Nougat, the academic document PDF parser that understands LaTeX math and tables.Project page:https://facebookresearch.github.io/nougat/InstallFrom pip:pip install nougat-ocrFrom repository:pip install git+https://github.com/facebookresearch/nougatNote, on Windows: If you want to utilize a GPU, make sure you first install the correct PyTorch version. Follow instructionshereThere are extra dependencies if you want to call the model from an API or generate a dataset.
Install viapip install "nougat-ocr[api]"orpip install "nougat-ocr[dataset]"Get prediction for a PDFCLITo get predictions for a PDF run$ nougat path/to/file.pdf -o output_directoryA path to a directory or to a file where each line is a path to a PDF can also be passed as a positional argument$ nougat path/to/directory -o output_directoryusage: nougat [-h] [--batchsize BATCHSIZE] [--checkpoint CHECKPOINT] [--model MODEL] [--out OUT]
              [--recompute] [--markdown] [--no-skipping] pdf [pdf ...]

positional arguments:
  pdf                   PDF(s) to process.

options:
  -h, --help            show this help message and exit
  --batchsize BATCHSIZE, -b BATCHSIZE
                        Batch size to use.
  --checkpoint CHECKPOINT, -c CHECKPOINT
                        Path to checkpoint directory.
  --model MODEL_TAG, -m MODEL_TAG
                        Model tag to use.
  --out OUT, -o OUT     Output directory.
  --recompute           Recompute already computed PDF, discarding previous predictions.
  --full-precision      Use float32 instead of bfloat16. Can speed up CPU conversion for some setups.
  --no-markdown         Do not add postprocessing step for markdown compatibility.
  --markdown            Add postprocessing step for markdown compatibility (default).
  --no-skipping         Don't apply failure detection heuristic.
  --pages PAGES, -p PAGES
                        Provide page numbers like '1-4,7' for pages 1 through 4 and page 7. Only works for single PDFs.The default model tag is0.1.0-small. If you want to use the base model, use0.1.0-base.$ nougat path/to/file.pdf -o output_directory -m 0.1.0-baseIn the output directory every PDF will be saved as a.mmdfile, the lightweight markup language, mostly compatible withMathpix Markdown(we make use of the LaTeX tables).Note: On some devices the failure detection heuristic is not working properly. If you experience a lot of[MISSING_PAGE]responses, try to run with the--no-skippingflag. Related:#11,#67APIWith the extra dependencies you useapp.pyto start an API. Call$nougat_apiTo get a prediction of a PDF file by making a POST request tohttp://127.0.0.1:8503/predict/. It also accepts parametersstartandstopto limit the computation to select page numbers (boundaries are included).The response is a string with the markdown text of the document.curl-X'POST'\'http://127.0.0.1:8503/predict/'\-H'accept: application/json'\-H'Content-Type: multipart/form-data'\-F'file=@<PDFFILE.pdf>;type=application/pdf'To use the limit the conversion to pages 1 to 5, use the start/stop parameters in the request URL:http://127.0.0.1:8503/predict/?start=1&stop=5DatasetGenerate datasetTo generate a dataset you needA directory containing the PDFsA directory containing the.htmlfiles (processed.texfiles byLaTeXML) with the same folder structureA binary file ofpdffigures2and a corresponding environment variableexport PDFFIGURES_PATH="/path/to/binary.jar"Next runpython -m nougat.dataset.split_htmls_to_pages --html path/html/root --pdfs path/pdf/root --out path/paired/output --figure path/pdffigures/outputsAdditional arguments includeArgumentDescription--recomputerecompute all splits--markdown MARKDOWNMarkdown output dir--workers WORKERSHow many processes to use--dpi DPIWhat resolution the pages will be saved at--timeout TIMEOUTmax time per paper in seconds--tesseractTesseract OCR prediction for each pageFinally create ajsonlfile that contains all the image paths, markdown text and meta information.python -m nougat.dataset.create_index --dir path/paired/output --out index.jsonlFor eachjsonlfile you also need to generate a seek map for faster data loading:python -m nougat.dataset.gen_seek file.jsonlThe resulting directory structure can look as follows:root/
├── images
├── train.jsonl
├── train.seek.map
├── test.jsonl
├── test.seek.map
├── validation.jsonl
└── validation.seek.mapNote that the.mmdand.jsonfiles in thepath/paired/output(hereimages) are no longer required.
This can be useful for pushing to a S3 bucket by halving the amount of files.TrainingTo train or fine tune a Nougat model, runpython train.py --config config/train_nougat.yamlEvaluationRunpython test.py --checkpoint path/to/checkpoint --dataset path/to/test.jsonl --save_path path/to/results.jsonTo get the results for the different text modalities, runpython -m nougat.metrics path/to/results.jsonFAQWhy am I only getting[MISSING_PAGE]?Nougat was trained on scientific papers found on arXiv and PMC. Is the document you're processing similar to that?
What language is the document in? Nougat works best with English papers, other Latin-based languages might work.Chinese, Russian, Japanese etc. will not work.
If these requirements are fulfilled it might be because of false positives in the failure detection, when computing on CPU or older GPUs (#11). Try passing the--no-skippingflag for now.Where can I download the model checkpoint from.They are uploaded here on GitHub in the release section. You can also download them during the first execution of the program. Choose the preferred preferred model by passing--model 0.1.0-{base,small}Citation@misc{blecher2023nougat,
      title={Nougat: Neural Optical Understanding for Academic Documents}, 
      author={Lukas Blecher and Guillem Cucurull and Thomas Scialom and Robert Stojnic},
      year={2023},
      eprint={2308.13418},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}AcknowledgmentsThis repository builds on top of theDonutrepository.LicenseNougat codebase is licensed under MIT.Nougat model weights are licensed under CC-BY-NC.