Nougat: Neural Optical Understanding for Academic Documents
Authors: Lukas Blecher
Guillem Cucurull
Thomas Scialom
Robert Stojnic
Venue: ICLR 2024 poster (2024)
Keywords: Visual Document Understanding
Optical Character Recognition
Mathematical Expression Recognition
Information Extraction
Primary area: general machine learning (i.e., none of the above)
TL;DR: A transformer-based model trained to convert document images to formatted markup text, focusing on extracting mathematical expressions and tables from scientific papers.

Abstract: Scientific knowledge is predominantly stored in books and scientific journals, often in the form of PDFs. However, the PDF format leads to a loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical Understanding for Academic Documents), a Visual Transformer model that performs an Optical Character Recognition (OCR) task for processing scientific documents into a markup language, and demonstrate the effectiveness of our model on a new dataset of scientific documents. The proposed approach offers a promising solution to enhance the accessibility of scientific knowledge in the digital age, by bridging the gap between human- readable documents and machine-readable text. We release the models and code to accelerate future work on scientific text recognition.

Published as a conference paper at ICLR 2024
NOUGAT: NEURAL OPTICAL UNDERSTANDING FOR ACADEMIC DOCUMENTS
Lukas Blecher'
Guillem Cucurull
Thomas Scialom
Robert Stojnic
Meta AI
ABSTRACT
Scientific knowledge is predominantly stored in books and scientific journals, often in the form of PDFs. However, the PDF format leads to loss of semantic information, particularly for mathematical expressions. We propose Nougat (Neural Optical Understanding for Academic Documents), Visual Transformer model that performs an Optical Character Recognition (OCR) task for processing scientific documents into markup language, and demonstrate the effectiveness of our model on a new dataset of scientific documents. The proposed approach offers a promising solution enhance the accessibility of scientific knowledge in the digital age, by bridging the gap between human-readabe readable documents and machine-readable text. We release the models and code to accelerate future work on scientific text recognition.
INTRODUCTION
The majority of scientific knowledge is stored in books or published in scientific journals, most commonly in the Portable Document Format (PDF). Next to HTML, PDFs are the second most prominent data format on the internet, making up 2.4% of common crawl (Sebastian Spiegler 2013). However, the information stored in these files very difficult to extract into any other formats. This is especially true for highly specialized documents, such as scientific research papers, where the semantic information of mathematical expressions is lost.
Existing Optical Character Recognition (OCR) engines, such as Tesseract OCR (Smith, 2007). excel detecting and classifying individual characters and words in an image, but fail to understand the relationship between them due to their line-by-line approach. This means that they treat superscripts and subscripts in the same way the surrounding text, which is significant drawback for mathemat- ical expressions. In mathematical notations like fractions, exponents, and matrices, relative positions of characters are crucial.
Converting academic research papers into ability science as a whole. The information of millions academic papers can not be fully accessed because they are locked behind an unreadable format. Existing corpora, such the S2ORC dataset (Lo et al. 2020). capture the text of 1 12M papers using GROBID (opez 2023 but are missing meaningful representations of the mathematical equations.
text also enables accessibility and search-
To this end, we introduce Nougat, a transformer based model that can convert images of document pages to formatted markup text.
The primary contributions in this paper are
Release of pre-trained model capable of converting PDF to lightweight markup language. We release the code and the model on GitHut"
We introduce pipeline create dataset for pairing PDFs source code
Our method is only dependent on the image of page, allowing access to scanned papers and books
"Correspondence to: 1b1 eelecher@@ee remeta. com
'The paper reports IM papers but the authors recently updated the numbers on GitHub page https
//gi thub. wwt.coo/aaaa..// a Tenai /
https://giiii com http://ww......oooooooeeereer///

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
Optical Character Recognition (OCR) is an extensively researched field in computer vision for variety applications, such document digitalization (Moysset al. 2017 Smith 2007), handwriting
recognition and scene text recognition (Bautista & Atienza 2022, Li et al., 2022: Diaz et al. 2021)
More concretely, recognizing mathematical expressions a heavily researched subtopic. Grammar based methods (MacLean & LLbahn 2013 Awal et al. 2014 Alvaro et al. 2014) for handwritten mathematical expressions were improved upon by different encoder-decoder models. The fully convolutional model (Yan et al. 2020) was succeeded by various RNN decoder models (Deng et al. 2016 [e & Nakagawa 2017; Singh 2018), Zhang et al. [2018] Wang & Liu, 2019), both for handwritten and printed formulas. Recently, the decoder (Zhao et al. 2021 Mahdavi et al. 2019) as well as the encoder (Blecher 2023) were replaced with the Transformer (Vaswani et al. 2017 architecture.
Visual Document Understanding (VDU) is another related topic of deep learning research and focuses on extracting relevant information variety of document types. Previous works depend on pre- trained models that learn extract information by jointly modeling text and layout information using
the Transformer architecture. The LayoutLM model family (Xu et al. 2020 2022 Huang et al. 2022) uses masked layout prediction task to capture the spatial relationships between different document elements.
Open source solutions with related goal ours include GROBID (Lopez 2023), ), which parses
digital-born scientific documents to XML with a focus on the bibliographic data and pdd 22 EX (Lu Wang & Wanmin Liu 2013), that converts digital-born PDFs to HTML while preserving the layout and appearance of the document. However, both solutions can not recover the semantic information of mathematical expressions.
Previous VDU methods either rely on OCR text from third party tool (Xu et al. 2020 2022
Appalaraju et al. (202)) or focus on document types such as receipts, invoices or
documents (Majumder et al. (020). Recent studies (Kim et al.) 2022 Davis et al. (022) show that an external OCR engine not necessarily needed to achieve competitive results VDU.
MODEL
The architecture is a encoder-decoder transformer (Vaswani et al., , 2017) architecture, that allows for end-to-end training procedure. We build the Donut (Kim et al. 2022) architecture. The model does not require any OCR related inputs or modules. The text is recognized implicitly by the network. See Fig. for overview of the approach.
Encoder The visual encoder receives document image x E R3x crops the margins and resizes the image fit fixed rectangle of size (H, W). If the image is smaller than the rectangle, additional padding is added to ensure each image has the same dimensionality. We use a Swin Transformer (Liu et al. 202T). a hierarchical vision transformer (Dosovitskiy et al. 2021 that splits the image into overllallg windows of fixed size and applies series of self-attention layers to aggregate information across these windows. The model output sequence of the embedded patches [2x.NN where d is the latent dimension and N is the number of patches.
Decoder The encoded image is decoded into a sequence tokens using a transformer decoder architecture with cross-attention. The tokens are generated in an auto-regressive manner, using self-attention and cross-attention to attend to different parts of the input sequence and encoder output respectively. Finally, the output is projected to the size of the vocabulary v, yiclding the logits & IR". Following Kim et al. (2022). we use the implementation of the mBART (Lewis et al., 2019) decoder. We use the same tokenizer Taylor al. (2022) because their model is also specialized in the scientific text domain.
3.1 SETUP
We render the document images at a resolution of 96 DPI. Due to the restrictive possible input
dimensions of the Swin Transformer choose the input size (H, W) (896, 672). The aspect
ratio is in between the US letter and Din A4 format 22 < v2.
The document images resized
2

---

Published as a conference paper at ICLR 2024
Example
Swin Transformer
Tranaformer Docoder
pparn dolor
adipuicing
eausned temper incidiiunt dolore magna allqus.
Our simple end-to-end architecture following Donut <Kim et al. 2022). The Swin Trans- former encoder takes a document image and converts it into latent embeddings, which are subsequently converted to a sequence of tokens in auto-regressive manner
Figure 1:
and then padded achieve the desired input size. This input size allows us use the Swin base model architecture (Liu et al. 2021). We initialize the model with the pre-trained weights. The Transformer decoder has a maximal sequence length of S 4096. This relatively large sizing is due to the fact that the text of academic research papers can be dense and the syntax for tables in particular is token intensive. The BART decoder is a decoder-only transformer with 10 layers. The entire architecture has total of 350M parameters.
We also test experiment with a smaller model (250M parameters) with a slightly smaller sequence length of S 3584 and only 4 decoder layers, where we start from the pre-trained base model. During inference the text is generated using greedy decoding.
Training We use AdamW optimizer (Loshchilov Hutter 2019) train for epochs with an effective batch size of 192. Due to training instabilities, we choose a learning rate of Irmit = 5- 10 which is reduced by factor of 0.9996 every 15 updates until it reaches ITend 7.5 10-
3.2 DATA AUGMENTATION
In image recognition tasks, it is often beneficial use data augmentation improve generalization. Since we are only using digital-born academic research papers, we need to employ number of transformations to simulate the imperfections and variability of scanned documents. These transformations include erosion, dilation, gaussian noise, gaussian blur, bitmap conversion, image compression, grid distortion and elastic transform (Simard et al., (003). Each has a fixed probability of being applied (Buslaev et al. 2020) library. For an overview of the effect of each transformation, see Fig. A. During training time, we also add perturbations to the ground truth text by randomly replacing tokens. We found this to reduce the collapse into a repeating loop significantly. For more details, see Section 5.4
given image. The transformations are implemented in the Albumentations
DATASETS
To the best of our knowledge there no paired dataset of PDF pages and corresponding source code there, created our own from the open access articles on arXiv For layout diversity we also include subset of the PubMed Central (PMC) open access non-commercial dataset. During the pretraining, a portion of the Industry Documents Library (IDL) is included. See Table A.1 for the dataset composition.
arXiv We collected the source code and compiled PDFs from 1.748.201 articles released on arXiv. To ensure consistent formatting, we first process the source files using 7 and convert them into HTMLS files. This step was important standardized and removed ambiguity from the LaTeX source code, especially in mathematical expressions. The conversion process included replacing user-defined macros, standardizing whitespace, adding optional brackets, normalizing tables, and replacing references and citations with their correct numbers.
https://arziv.org nlm nih gww/pnc////gggggogcoo/.........e
wwt.:/.innustttdcco.ooo http://dlmf.n
ucsf edu/
3

---

Published as a conference paper at ICLR 2024
TeX compiler
\section(ritle)
ch2a
1 Title
1 Title
</spam-Title<<h222 qp- study the formula </p> <naths-anrow-
we study the fornula ibegintequation)
we study the formula
We study the for- muis
MI
Parser
E=ncA{2) (ebeeteinn (equattinon in Vcitefein)
<ni>E</mi> .......... </natth <p> as in </p>
[F]] (1)
me?
(1)
as [t]
<cite> [1]x/cite>
tcx
b) html
markdown
d) pdf
Figure 2: Data processing. The source file is converted into HTML which is then converted to Markdown. a) The LaTeX source provided by the authors. b) The HTML file computed form the LaTeX source using LaTeXMI.. c) The Markdown file parsed from the HTML file. d) The PDF file provided by the authors
We then parse the HTML files and convert them into a lightweight markup language that supports various elements such as headings. bold and italic text, algorithms, LaTeX inline and display math and LaTeX tables. This way, we ensure that the source code is properly formatted and ready for further processing. The process is visualized in Fig.
PMC We also processed articles from PMC, where XML files with semantic information are available in addition to the PDF file. We parse these files into the same markup language format as the arXiv articles. We chose to use fewer articles from PMC because the XML files are not always as rich in semantic information. Often times equations and tables are stored as images and these cases are not trivial to detect, which leads to our decision to limit the use of PMC articles to the pre-training phase.
The XML files are parsed into the same markup language as described above.
IDL The IDL is collection of documents produced by industries that have an impact on public health and is maintained by the University California, San Francisco Library. Biten et al. (2022) provide high quality OCR text for PDFs from the IDL dataset. This does not include text formatting and is only used for pre-training to teach the model basic OCR of scanned documents.
4.1 SPLITTING THE PAGES
We split the markdown files according to the page breaks in the PDF file and rasterize each page as an image to create the final paired dataset. During the compilation, the LaTeX compiler determines the page breaks of the PDF file automatically. Since we are not recompiling the LaTeX sources for each paper, we must heuristically split the source file into parts, which correspond to different pages. To achieve that are using the embedded text the PDF page and match it to source text. However. figures and tables in the PDF may not correspond to their position in the source code. To address this issuc, remove these elements in pre-processing step using pdff fiffiig2es22 (Clark) & Divvala 2016). The recognized captions are are then compared to the captions in the XML file and matched based their Levenshtein distance (Levenshtein) 1965). Once the source document has been split into individual pages, the removed figures and tables are reinserted at the end of each page. For better matching we also replaced unicode characters in the PDF text with corresponding LaTeX commands using the pylatexene-libaayy"
Bag of Words matching First extract the text lines from the PDF using MuPDI and preprocess them to remove page numbers and potential headers/footers. We then use a Bag of Words model (Harris 1954) with TF-IDF vectorizer and linear Support Vector Machine classifier. The model is fitted to the PDF lines with the page number as label. Next we split the LaTeX source into paragraphs and predict the page number for each of them.
hetpps://qithub.. com. http://.o//oaaattttgglaaaaeaeeee /ttps//////ttttooooo///////////s
https://mupdf. com/
4

---

Published as a conference paper at ICLR 2024
Staircuse fil Predictions
Page index
6-
2-
0-
5
10
15
20
25
30
35
40
Paragraph index
Figure 3: Example for splitting the paragraphs in the source code into different pages. The points in blue denote the page index predicted by the SVM.
Ideally, the predictions will form stair case function but in practice the signal will be noisy. To find the best boundary points we employ similar logic as decision trees and minimize a measure based on the Gini impurity
(i) = (b (b-a) (1- Pj uj (1))
where bia.b] (i) is the probability of choosing an element with the predicted page number in the interval of paragraph indices [a, b] that describes which paragraphs (elements) were considered for the split. The best splitting position t in the interval (a, b] is then
f arg min (d.) + (L.)
(2)
The search process starts with all paragraphs and for each subsequent page break, the lower bound of the search interval is set to the previous split position. See Fig. for a visualization of an example page.
Fuzzy matching After this first coarse document splitting we try to find the exact position within the paragraph. This is done by comparing the source text within the neighborhood of the predicted splitting position to the last sentences of the previous page of the embedded PDF text, and the first sentences of the next page using the fuzzysea rch library" If the two dividing points same location in the source text, the page break is considered "accurate" and receives a score of 1. On the other hand, if the splitting positions differ, the one with the smallest normalized Levenshtein distance is selected and given score of minus the distance. To be included in the dataset, a PDF page must have an average score of least 0.9 for both page breaks. This results in an acceptance rate of about 47% of all pages.
the
4.2 GROUND TRUTH ARTIFACTS
Because the dataset was pre-processed by LaTeXML, the markup version of the source code can contain artifacts and commands from unsupported packages. The HTML file may contain subsection titles with numbering even though they are not numbered in the PDF There may also be instances where figures or tables are missing from the ground truth due to processing errors.
In addition, the splitting algorithm of the source code will in some cases include text from the previous page or cut off words from the end. This is especially true for "invisible" characters used for formatting, like italic, bold text or section header.
For PMC papers the inline math written Unicode or italic text, while display math equations or tables are often included in image format and will therefore be ignored.
Each of these issues reduces the overall data quality. However, the large number of training samples compensates for these small errors.
5

---

Published as a conference paper at ICLR 2024
5 RESULTS & EVALUATION
More wleate
Teydon
Figure 4: Example of a page with many mathematical equations taken from (Sorscher et al. 2022).. Left: Image of page in the document, Right: Model output converted to LaTeX and rendered to back into a PDF. Examples of scanned documents can be found in the appendix B
In this section we discuss the results and performance of the model. For example see Fig. 4 or go to Sec. B The model focuses only on the important content relevant features of the page. The box around the equations is skipped.
5.1 METRICS
We report the following metrics on our test set.
Character Error Rate The character rate (CER), normalized Levenshtein distance (Lev-
enshtein 1965), measures the number of character manipulations (insertions, deletions, substitutions) takes get from one string another.
BLEU The BLEU (Papineni et al. 2002) metric was originally introduced for measuring the quality of text that has been machine-translated from one language another. The metric computes score based on the number of matching n g-grams between the candidate and reference sentence.
METEOR Another machine- transsaainnn metric with focus on recall instead of precision,
introduced in (Banerjee & Lavie 2005).
F-measure
We also compute the Fl-soo Fl-score and report the precision and recall.
5.2
TEXT MODALITIES
In scientific research article, there are three distinct types of text: 1) plain text, which comprises the majority of the document, 2) mathematical expressions, and 3) tables. is important to separately examine each of these components during the evaluation process. This is necessary because in LaTeX, there are multiple ways to express the same mathematical expression. While some variability has been eliminated during the LaTeXML pre-processing step, there still is a significant amount of ambiguity present, like ordering of subscript and superscript, equivalent commands with different notation (stackrel, atop, substack frac, over), situationally interchangeable commands (om, mathbf, boldsymbol, bf or \left (, \big (, etc.). whitespace commands, additional layers of brackets, and more. As consequence, there can be discrepancy between prediction and ground

---

Published as a conference paper at ICLR 2024
Method
Modality
CER
BLEU
METEOR
Prec.
Rec.
F1
PDF
All
25.5
65.8
82.1
77.1
81.4
79.2
GROBID
All
31,2
55.6
71.9
74.0
72.1
73.0
Tables
62.6
25.1
64.5
61.4
80.7
69.7
LaTeX OCR Blecher
Plain text
36.3
57.4
69.2
82.1
70.5
75.9
Math
72.7
0.3
5.0
11.0
8.6
9.7
All
7.3
88.9
92.8
93.6
92.2
92.9
Nougat small (250M*)
Tables
22.0
68.5
78.6
75.0
79.8
77.3
Plain text
5.8
91.0
94.3
96.1
95.3
95.7
Math
11.7
56.0
74.7
77.1
76.8
76.9
7.1
89.1
93.0
93.5
92.8
93.1
Nougat base (350M*)
Tables
21.1
69.7
79.1
75.4
80.7
78.0
Plain text
5.8
91.2
94.6
96.2
95.3
95.7
Math
12.8
56.9
75.4
76.5
76.6
76.5
Table Results on arXiv test set. PDF the text embedded PDF file. The modality "All" refers to the output text without any splitting. All metrics are reported in percent. "Number of parameters.
truth, even if the rendered formulas appear identical.
In addition, is not always possible to determine, where an inline math environment ends and text be- gins, when writing numbers and punctuation (Example: $ Amathrm (0) vs. [ - Ho I, vs. H,),). This ambiguity reduces both math and plain text scores. The expected score for mathematical expressions lower than for plain text.
5.3 COMPARISON
We present our results in T As expected, the mathematical expressions have the worst agreement with the ground truth. For the plain text, most discrepancies come from formatting ambiguities and missing text due to inline math, as described above. The output format of GROBID is an XML file, which convert into compatible markup language, similar the PMC or arXiv files. To some extent, GROBID provides support for formulas in its output, but identifies and stores them as the Unicode representations embedded in the PDF. We replace each Unicode symbol with its corresponding LaTeX command to increase the similarity. Additionally, GROBID mislabels small inline expressions as text. For identified formulas, GROBID stores the bounding box coordinates. We modify the program by sending the snippet to the external formula recognition software LaTex-OOCR (Blecher, 2023). This way we can also get signal for math modality. The reported results in this section quite poor, primarily due to the amount of missed formulas by GROBID and the equation prediction accuracy is affected by the quality of the bounding boxes. The performance of the embedded PDF text alone is better than GROBID, which is due formatting differences for the title page or reference section.
Both Nougat small and base are able to outperform the other approach and achieve high scores in all metrics. We note that the performance of the smaller model is on par with the larger base model.
5.4 REPETITIONS DURING INFERENCE.
We notice that the model degenerates into repeating the same sentence continuously, state from which it cannot autonomously recover. its simplest form, the last sentence or paragraph is repeated ad infinitum. We observed this behavior in 1.5% of pages in the test but the frequency increases for out-of-domain documents. Our evaluation extends to a diverse selection of academic documents, encompassing range of sources such scanned books. See Appendix B for examples.
Getting stuck in a repetitive loop is a known problem with Transformer baseedder--beeddddd models, when sampled with greedy decoding (Holtzman al. 2020). We conducted experiments with nucleus sampling,
7

---

Published as a conference paper at ICLR 2024
but encountered susceptibility to repetitive outputs. Consequently, opted against utilizing this technique due to the introduced randomness failing to effectively mitigate the issue of unwanted sentence repetitions.
can also happen that the model alternates between two sentences but sometimes changes some words, so a strict repetition detection will not suffice. Even harder to detect predictions where the model counts its own repetitions, which sometimes happens in the references section. In general we notice this kind behavior after mistake by the model. The model is not able to recover from the collapse.
Anti-repetition augmentation Because of that we introduce random perturbation during training. This helps the model to learn how to handle a wrongly predicted token. For each training example, there fixed probability that a random token will be replaced by any other randomly chosen token, This process continues until the newly sampled number is greater than specified threshold (in this case, 10%) We did not observe decrease in performance with this approach, but we did notice significant reduction in repetitions. Particularly for out-of-domain documents, where we saw 32% decline in failed page conversions.
Repetition detection Since are generating a maximum of 4096 tokens, the model will stop at some point. However, it is very inefficient and resource-intensie to wait for an "end of sentence" token that will not be generated. To detect repetition during inference, value 4 = max & for the ith token. We observed that logits after a collapse can be separated using the following heuristic consisting of two steps. First, calculate the variance of the logits for sliding window of size B = 15
examine the largest logit
(arWin((((2)
1
B+B B
Σ
4
2
Σ
B
(3)
B
B
j=
Here, é the signal of logits, and r is the index. Using this new signal, compute variances again, but this time from point to the end of the sequence:
s
VarEndn|(((s) S x
VarWinn(4100 5
VarWinn(e(())
(4)
this signal drops below a certain threshold (we choose 6.75) and stays below for the remainder of the sequence, classify the sequence having repetitions.
During inference time, is not possible to compute the entire sequence if our goal is to stop generation earlier. Here, work with subset of the last 200 tokens and half the threshold. After the generation finished, the procedure as described above is repeated for the full sequence.
5.5
LIMITATIONS & FUTURE WORK
The utility of the model is limited by a number of factors. First, the problem with repetitions outlined in section 5.4 The model is trained on research papers, which means it works particularly well on documents with a similar structure. However, can still accurately convert other types of documents.
Utility
Nearly every dataset sample is in English. Initial tests on a small sample suggest that the model's performance with other Latin-based languages satisfactory, although any special characters from these languages will be replaced with the closest equivalent from the Latin alphabet. Non-Latin script languages result in instant repetitions.
On a machine with a NVIDIA A10G graphics card with 24GB VRAM we can process 6 pages in parallel. The generation speed depends heavily on the amount of text on any given page. With an average number of tokens of s 1400 we get mean generation time of 19.5s per batch for the base model without any inference optimization. Compared to classical approaches (GROBID 10.6 PDF/s (Lopez) 2023)) this is very slow, but it is not limited to digital-bom PDFs and can correctly parse mathematical expressions.
Generation Speed
Future work The model is trained on page time without knowledge about other pages in the document. This results in inconsistencies across the document. Most notably in the bibliography where the model was trained different styles section titles where sometimes numbers are skipped
8

---

Published as a conference paper at ICLR 2024
Logis
VarWWin
20
VarEnd
10
20
2000 Sequence
4000
2000 Soquence
4000
2000 Sequence
4000
60
150
Logits
VarWin
40
VarEnd
100
500
1000
500
1000
500
1000
Sequence
Sequence
Sequence
Figure 5: Examples for repetition detection on logits. The vertical index denotes the sequence index, where repetitions start. Top: Sample generation with repetition, Bottom: Sample generation without repetition. Left: Highest logit score for each token in the sequence {(x), Center: Sliding window variance of the logits Var [&](,) Right: Variance the window variance (3) from the position to the end VarEnd (x) with threshold as horizontal dashed line.
or hallucinated. Though handling each page separately significantly improves parallelization and scalability, it may diminish the quality of the merged document The primary challenge to solve is the tendency for the model to collapse into repeating loop, which left for future work.
CONCLUSION
In this work, we present Nougat, an end-to-end trainable ricoder-decoder transformer based model for converting document pages to markup. We apply recent advances in visual document understanding to a novel OCR task. Distinct from related approaches, our method does not rely on OCR or embedded text representations, instead relying solely on the rasterized document page. Moreover, we have illustrated an automatic and unsupervised dataset generation process that we used to successfully train the model for scientific document to markup conversion. Overall, our approach has shown great potential for not only extracting text from digital-born PDFs but also for converting scanned papers and textbooks. We hope this work can be starting point for future research in related domains. All the code for model evaluation, training and dataset generation can be accessed at https htt.:///oocooccceoooeeeeeeeeceee
ACKNOWLEDGMENTS
Thanks to Ross Taylor, Marcin Kardas, Iliyan Zarov, Kevin Stone, Jian Xiang Kuan, Andrew Poulton and Hugo Touvron for their valuable discussions and feedback. Thanks to Faisal Azhar for the support throughout the project.

---

Published as a conference paper at ICLR 2024
REFERENCES
Srikar Appalaraju, Bhavan Jasani, Bhargava Urala Kota, Yusheng Xie, and R. Manmatha. DocFormer: End-to-End Transformer for Document Understanding, September 2021. URL http://arxxx org/abs/ htt5/bbs/2006.. 2106.1106.0 11539 arXiv:2106.1 1539 [cs].
Homad-Montaser Awal, Harold Mouchre, and Christian Viard-Gaudin A global learning approach for online handwritten mathematical expression recognition system. Pattern Recognition Letters, 35(C))68-77, Jamuary 2014. ISSN 0167-8655.
Satanjeev Banerjee and Alon Lavie. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72,
Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https
//aclanthology.org/ 0rg.:05-000000000000000000000000
Darwin Bautista and Rowel Atienza. Scene Text Recognition with Permuted Autoregressive Sequence Models, July 2022. URL htttttttp//xxiii org/ 2ads//2000 06966 arXiv:2207.0696 [cs]
version: 1.
Ali Furkan Biten, Rubèn Tito, Lluis Gomez, Ernest Valveny, and Dimosthenis Karatzas. OCR- IDL: OCR Annotations for Industry Document Library Dataset, February 2022. URL http
//arxiv.org//bs 22000 12985 1XXi::2202.12985 [cs].
Lukas Blecher. pix2tex LaTeX OCR, February 2023. URL https: /titps://githbb... /gggggihuub..... com/ original-date: 2020- 0020-12-11T16:35:113
lukas-blecher/ Tex-OO
Alexander Buslaev, Vladimir I. Iglovikov, Eugene Khvedchenya. Alex Parinov, Mikhail Druzhinin, and Alexandr A. Kalinin. Albumentations: Fast and Flexible Image Augmentations, Information. hRLhttps: //ww/ /www
11(2):125, Februury 2020. ISSN 2078-2489. doi: 10.3390/info1 1020125.
Christopher Clark and Santosh Divvala. PDFFigures 2.0: Mining Figures from Research Papers. In
Proceedings of the 16th ACM/EEEE-CA CS on Joint Conference on Digital Libraries, pp. 143-152, Newark New Jersey USA, June 2016. ACM. ISBN 4578--9990322229222 doi: 10. 00.1145/290896 2910904. URL UR.. htttss://d..... acm. www.m.org/dooe wwtp://////oo////oo//
291 0904
Brian Davis, Bryan Morse, Bryan Price, Chris Tensmeyer, Curtis Wigington, and Vlad Morariu. End-to-end Document Recognition and Understanding with Dessurt, June 2022. URL http: //arxiv. wtw.aa/./222ccc.o 6618 arXiv:2203. 16618 [es].
Yuntian Deng, Anssi Kanervisto, Jeffrey Ling, and Alexander M. Rush. Image-to-Markup Generatioo with Coarse-to-Fine Attention, September 2016. URL
04938 arXiv:1609.04933 [cs]
version: 1.
Daniel Hernandez Diaz, Siyang Qin, Reeve Ingle, Yasuhisa Fujii, and Alessandro Bissacco. Re- thinking Text Line Recognition Models, April 2021. URL httlltttpaaxii..oooo orggabs///a0aaa /2104
07787 arXiv:2104.07787 [cs].
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL http: // arxiv. /tt///ab///1111111
arXiv:2010.1197 [cs].
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical Neural Story Generation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 889-898, 898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v 1/P18- 1082.
wwwp:/llooyrrg.e--eo1 1082
Sanford Gordon, Sheldon Heimel, Frank Zeleznik, Richard Brokaw, Charles Baker, Roger Svehla, Frank Belles, Theodore Brabbs, Marvin Warshay, and David Bittker. Kinetics and Thermo- dynamics in High- Temperature Gases, January 1970. URL https ://nt nasa. /ov/ citationa / 1 700022795 NTRS Report/Patent Number: N70-32106-11 16 NTRS Document ID: 19700022795 NTRS Research Center: Glenn Research Center (GRC).
10

---

Published as a conference paper at ICLR 2024
Zellig S. Harris. Distributional Structure. WORD, 1WORD.,1023))1440000 100--00013311122222 1954. doi: 10. 10.1080000437956 1954.11639520. Ut/htttss///b...c///100000001113 19w.://0044..551.....1.......... 116599995 Pub- lisher: Routledge eprint: http://douur////.000000076655669
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The Curious Case of Neu- ral Text Degeneration, February 2020. URL httt:://arxiv..rg/bbbe////
arXiv:1904009551 [es].
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. LayoutL Pre-training for Document AI with Unified Text and Image Masking, July 2022. URL http::tpp//aai...rrtttt arXiv:2204.08387 [cs].
abs/2204 08387
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeongyeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. OCR-free Document Understanding Transformer, October 2022. httptttpp///rii..orraababa////// MxXiv:2111.1566 [cs].
Anh Due Le and Masaki Nakagawa. Training an End-to-End System for Handwritten Mathematical Expression Recognition by Generated Patterns. In 2017 /4th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pp. 1056-1061, November 2017. doi: 10.1 111.////...........
ISSN: 2379-2140.
Vladimir I. Levenshtein. Binary codes capable of conrecting delctions, insertions, and reversals. semant. wachtttsscclllc.o aat.oalarrrr. org/
Soviet physics. Doklady, 10:707-710, 1965.
CorpusID:: 60827152
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising seetvence-toSeqqeeee Pre-training for Natural Language Generation, Translation, and Comprehension, October 2019. http
//arxiv.org//bs:t wttpg////b///...////////// 11t.e/1111111111111111111 13 arXiv:1910. 13461 [cs, stat].
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. TrOCR: Transformer-baseed Optical Character Recognition with Pre-trained Models, September 2022. /ttt:ttt://aaxii org/ abs 2109. 10282 arXiv:2109. [cs].
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, August 2021. URL [es].
http://arxiv. org/ abs 211.22211111100
arXiv:2103.144
Kyle Lo. Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The Semantic Scholar Open Research Corpus. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969 4983, Online, July 2020. Association for Computational Linguistics. doi: 10. 10118853311/22000 acl- mad.mmaii.447. URL http::://claaattoooooooooo
2020.ac1- maii.main.... 447.
Patrice Lopez. GROBID, February 2023, URL Bt//a//ppss//ggtttss.cc///mtmttt original-date: 2012-09-13315::8:552.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. URL
http:://rxxi/ http:////aaaaiioorgga/////////// wttp:oo////bb....o// 05101 1.X.::711..051 [cs, math] version: 3.
Lu Wang and Wanmin Liu. Online publishing via pd12htm1EX, 2013. URL https:://////.. //t:::///wwww. tug
Pog//TGGboat/ttb342
Scott MacLean and George Labahn. A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. International Jourmal on Document Analysis and Recognition
(IJDAR), 16(2):139-163 163, June 2013. ISSN 1433-2825. doi: 10.10077/110033--1---11444 URL 0000--12--100 https://doi wrg/o/oggo...oo 1000/100000//// 021-000122211-------1-----------
Mahshad Mahdavi, Richard Zanibbi, Harold Mouchere, Christian Viard-Gaudin, and Utpal Garain. ICDAR 2019 CROHME + TFD: Competition Recognition of Handwritten Mathematical Expressions and Typeset Formula Detection. In 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 1533-1538, Sydney, Australia. September 2019. IEEE
ISBN 978-1-77813333 11881--2213--14499 doi: 10. 10111////A....1..0000.. URL hattps://eeexxplos/
leee /eee.orrg/dccmmmm///7700.3

---

Published as a conference paper at ICLR 2024
Bodhisattwa Prasad Majumder, Navnect Potti, Sandeep Tata, James Bradley Wendt, Qi Zhao, and Mare Najork. Representation Learning for Information Extraction from Form-like Documents. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6495-6504, Online, July 2020, Association for Computational Linguistics. doi: 10. 131853/v1/2000 acl-main.580. URL https: peattps:/accaanntalloooo
Herman W. (Herman William) March and Henry C. (Henry Charles) Wolff. Calculus. New York McGraw-Hill, 1917.
Bastien Moysset, Christopher Kermorvant, and Christian Wolf. Full-Page Text Recognition: Learning Where to Start and When to Stop, April 2017. URL http:://arr/....rrgbb////// 0868 arXiv:1704. 08628 [es].
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: Method for Auto-
matic Evaluation of Machine Translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 00.3115/107333330033333 URL
attts::/aclaattoole
1040
Sebastian
Spiegler.
Statistics of the Common Crawl Corpus 2012.
June
2013.
URL
com/ wwwmfo/////////mccccc//ee/a
96
Meet Shah, Xinlei Chen, Marcus Rohrbach, and Devi Parikh. Cycle-Consistencyy for Robust Vi- sual Question Answering, February 2019. URL htt.http//aaax...o/////////0000 05660 arXiv:1902.05660 [cs].
P.Y. Simard, D. Steinkraus, and J.C. Platt. Best practices for convolutional neural networks applied to visual document analysis. In Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings., volume 1, pp. 958 963, Edinburgh, UK, 2003. IEEE Comput. Soc. ISBN 978-0-7695-19604 doi: 2000///DAA..000.2222222 leee. org /ateeeorrr//ccmmme//22222230
URL http:::t////eeeelooor
Sumeet S. Singh. Teaching Machines to Code: Neural Markup Generation with Visual Attention,
June 2018. URL
www.ooorggaaaaaaaaoe.o. /tabss//8/22 05415, arXiv:1802.05411 [es].
R. Smith. An Overview of the Tesseract OCR Engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007) Vol 2. pp. 629-633, Curitiba, Parana, Brazil, September 2007. IEEE. ISBN 978-0-7695-2222--9, doi: 10. 109/C//A 23799937699. URL ISSN: 1520-5363.
http:://eeeexloozz ieee wtl.eee..rr//cccceee/////311
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning, November 2022. URL Utt/t 4486 arXiv:2206. 14486 [cs, stat].
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshom, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: Large Language Model for Science, November 2022. htRp:/tt://aat /ttpvv///axiivv//aa org/ abs/ 2211. 09085 1rXiv:2211.0908 [cs, stat].
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need, December 2017. URL http: // arraiivvv...... org/ abs hbtp://bs///// 03762 arXiv:1706.03762 [cs].
Zelun Wang and Jyh-Chan Liu. Translating Math Formula Images to LaTeX Sequences Using Deep Neural Networks with Sequence-level Training, September 2019. URL arXiv: 1908. 1415 [cs, stat].
abs/1 1908.
1415
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. LayoutLMv2: Multi-modal Pre- Pre-training for Visually-Red -ichh Document Understanding, January 2022. URL
org/abs/
2012 2. 14740 arXiv:2012. 14740 [cs].
12

---

Published as a conference paper at ICLR 2024
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. LayoutLM: Pre- training of Text and Layout for Document Image Understanding. In Proceedings of the 26th
ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1192-
1200, August 2020. doi: 10.1 145/3399448.34003772 URL http: htt.:t:://axxi.rrg////////o abssg////////// /912
13318 1931v.191..13318 [cs].
Zuoyu Yan, Xiaode Zhang, Liangcai Gao, Ke Yuan, and Zhi Tang. ConvMath: A Convolutional Sequence Network for Mathematical Expression Recognition, December 2020. URL http:
//arxiv.org/bab... absgg//bs///a 2212 12619 arXiv:2012.12611 [cs].
Jianshu Zhang, Jun Du, and Lirong Dai. Multi-Scale Autention with Dense Encoder for Handwritten Mathematical Expression Recognition, January 2018. BRRL:tt:///aatt wrg. 1801
03530 arXiv:1801.0355: [cs].
Wenqi Zhao, Liangcai Gao, Zuoyu Yan, Shuai Peng, Lin Du, and Ziyin Zhang. Handwritten Mathematical Expression Recognition with Bidirectionally Trained Transformer, May 2021. URL
arXiv:2105.02411 [cs].
Francisco Álvaro, Joan-Andreu Sánchez, and José-Miguel Benedí. Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Puttern Recognition Letters, 35:58-67 January 2014. ISSN 0167-8655. doi: AariccleesLL/S00166655220003
URL https:::/ www. sciencedirect. com/ science/
13

---

Published as a conference paper at ICLR 2024
A DATASET
Original
Bitmap
Erosion
INTRODUCTION
INTRODUCTION
INTRODUCTION
L
...., ang
L
at .o... indepencenn knntetie
L
A.A.-Ra..... inderndenn Meeniical with BXX) and be of and
will BIK -
led
wia BIX - anad VoT of
x
2 Ro
Dilation
Affine
Shift Scale Rotate
INTRODUCTION
INTRODUCTION
L
city Ro siing mound deprettr bi MEREER
L
Ee...al nop tndeyo lclenticilyy wis BIXI
L
Au soperiee indeeratnn RIX) Nork
St
Grid Distortion
Elastic Transform
Random Brightness Contrast
INTROUUCTION
INTRODUCTION
L
a.N. Ro besesponced acgendaan detticll with RIN in paon (e)
L
eil, No be superice inlpesdaittn skenical wis RUX) Vedk
L
eit, Ao separso txtepeden sdomtoet RLM)
15
Image Compression
Gauss Noise
Ganssian Blur
INTROOUCTION L
INTRODUCTION
INTRODUCTION
L
L
suppene
will BIKI Vet in ond
with BIX] Var wo
with EEKI Vod of in and
Figure A.1: List of the different image augmentation methods used during training on an example snippet form sample document.
Name
Number of Pages
arXiv PMC
1,511,745 536,319
IDL.
446,777
Total
8,494,841
Table A.1: Dataset composition
The most important data source is arXiv, making up 88.4% of the corpus. On arXiv most research documents are paired with the LaTeX source code provided by the authors. The LaTeX source offers more information and is left unprocessed, unlike the XML format from PMC where equations and tables are frequently substituted with images. This allows us to select exactly which information we need to build the dataset.
B EXAMPLES
In this section converted some pages from old text books using the Nougat base model. The text books from the Internet To and Project Gutenberg") and are in public domain.
The performance for these scanned pages is noticeable worse than for digital-born documents. How- ever, the model does generate sensible text for each page with few errors. For example see the first row Fig. B.I Here the model mistakes the almost illegible exponent for *. the second row of the same figure the model falls into a repetitive loop after predicting another comma instead of dot. Similar problems can be seen in Fig. B.2
In Fig. B.3 we present pages, scanned with mobile device, from a printed master thesis and the Nougat output. The model robust to artifacts that arise when band-scanning a document. Explore the examples in this section on the project page: https: /aap://:////ccbbooeeeeeeerr github.ic/nouce
hhttoss/arrchive.orr
14

---

Published as a conference paper at ICLR 2024
that
posttine
Istegin. rulo
36, The Decivetive Comstart. Let = viem Commpndding
los 28
Desisative Tho Pectican
da
-
Fie ariistine of crnaiona Interpent this wouk poesstriedly.
3s. The Desiouiie of the Bam Two Puoclics
and fuasilone Let de inei of 6. ael iepestiodly carrepanding lnrrenest
As
da
da
Dutr drvisiii
of pustive dows-
Vkggr dy
lotel
P- sodyy.
expersed terua by equation
Then
2t
I Tyvan-pa
proearr gatn edge Aifilaiie oethorof the vater theolge
Fird phppremaees vesinal aceiiie gato whone diandes water. Avtianeic Mast. deiaol by the oyenlce asalorrs du
seiee
that
Figure B.1: Example of an old caleulus text book (Mrchh & Wolfi 1917).
15

---

Published as a conference paper at ICLR 2024
rat couuiatte cemtoes Do This
DETERMSTILL ASTOCTS
EXTERIMENSAL ASPOCTS
Creet dont mere oleued CoMMeTAN fue tite surle conecase
proportued yrotey orioo acoantie serees colIeEsIi
tnd, Bull
cntire condlce suryiig unciite dee tysineen cater acouadle
sugin Racratoo wrtee crgae lel unrti guer
you
on coenet corvicaet mduit saloc dualts puitiec
quater
slaako adkudd
forer
bobael Look Comtie moeis reginn rago
suftip the
hondery locr grven
SELECTION GAS MINTURES
abeaed
SEECTION MOTTRRSS
dast
unlo leterr valee
sarding orpoeee
anoinig engloatiic
dcowe will boge
ioos, becon no senoon end alouue Host. coans scce by rsaly, was by soow saduss duasa vaccing par
Figure B.2: A selection of pages from NASA conference from 1970 (Grrdon al. 1970
16

---

Published as a conference paper at ICLR 2024
alatt
opuatt pellengs youe dundier
mosldn crsercc will wherccc dooed
Toaal
Coepting Terms
cemeees............. Sowe
groug
(195's (199"
mostle also eisaeeee)
Li Ieciendlrcill Etpention
Kade sepoees
ui
S S- aff -2.44
deperdia badk iocutiin
the
Dork battee Lae E000
aothet proriaa
shie rupeeie
Figure B.3: Scan of modern thesis with mobile device camera, with permission from the author.
17

---

Published as a conference paper at ICLR 2024
CLR
10.0
464
and
Buve
dae
dust best lud prre piay glin soiledy vide shick
181
oofist podle des To coladde
Patm
wabis The
poeee palees Boi Pork
uden Merentedyy
they modittss crsee de na
comees clems clles.
Pargorty
Parpontty
11:4
$4.00
glose Peape pouce rociudee
2156
Eanie Cara
2014
Pigee Acoming
Figure B.4: Pages with tables. Upper: Fan et al. 2018 page 6, Lower: Shah et al. (019 page 6
18


Reviews:
[{"review_id": "cHVfwjidI6", "reviewer": "Reviewer_VwBw", "summary": "The paper proposes an image-to-LaTeX model for converting PDF documents into the corresponding LaTeX source. The model comprises of Swin Transformer encoder, followed by a standard text decoder (BART). The paper also describes in detail an involved process of collecting the data to obtain the training dataset, which involves non-trivial steps to clean and split PDFs. The model and the code to generate the dataset are made available, and the model outperforms several existing benchmarks.", "questions": "One of the central topics in the results is about the repetition of the same sentence again, an artifact of the greedy sampling, and authors approach this problem by doing a data augmentation to help reduce those repetitions. Is there a particular reason why the authors chose to stick with greedy sampling instead of using Top-K / Top-p sampling, which is a standard approach to reduce the overconfidence during autoregressive decoding? (See ex. \"The curious case of neural text degeneration\")", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Significance: The paper has an excellent contribution, first and foremost by creating the dataset for pdf-2-latex and describing the methodology and individual steps behind it - these two will greatly benefit the community. The availability of the model is also a significant contribution.\nOriginality: The proposed modelling approach is not original, but the construction of the dataset is.\nQuality & Clarity: The paper is well written and clear, and reproducibility is further fostered by the model / data generation release.", "weaknesses": "No significant weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CLGSAPe75F", "reviewer": "Reviewer_SnU5", "summary": "This paper introduces a new model (Nougat) for automatic understanding of academic documents.\nThe authors highlight the main limitations of SOTA methods in the conversion of scientific articles, mainly in dealing with complex objects like tables & mathematical equations. As a result, Nougat addresses end-to-end scientific article understanding with a focus on math equations. The authors develop a large dataset of scientific articles and train Nougat to predict their Markdown formatting. Results show that Nougat outperforms SOTA approaches by a large margin. Examples of article conversion are also presented in the appendix.", "questions": "* Are the scores presented in % for the editdistance metric? This metric is referred to as CER (Character Error Rate) in the document understanding community. \n* This sentence is not clear, as Fig. 2 shows that LaTeX sources are recompiled. In Overleaf you can match paragraphs in the LaTeX source and their localization in the PDF - can this feature be used to simplify this step?\n> Since we are not recompiling the LaTeX sources for each paper, we must heuristically split the source file into parts, which correspond to different pages. To achieve that we are using the embedded text on the PDF page and match it to source text. \n* The label noise strategy is not clear. Is noise added to 10% of labels?\n> This process continues until the newly sampled number is greater than a specified threshold (in this case, 10%)\n* Text vs equations vs tables all have different semantics. Could you comment on which metrics are the most relevant for each text category? It would be interesting to use specific metrics for equation recognition (see the CHROME competition https://www.cs.rit.edu/~rlaz/files/CROHME+TFD%E2%80%932019.pdf)\n* The limitation of 4096 tokens seems low for a full page. Did you encounter pages with a larger number of tokens to predict, and how did you handle them? How would this limitation affect document-level recognition (sliding window/patch?).\n* Is any post-processing strategy used to normalize section/subsection formatting (example: `##` (section) predicted instead of `###` (subsection)) ?\n* Adaptability to other languages? While most publications are written in English, some of them are written in other languages. How much work to fine-tune Nougat? (since the tokenizer, dataset, pre-trained models are all specialized for English) \n* What if repetitions appear inside the page and not at the end? (ex: a sentence is repeated twice, but after that the model continues reading until the end of the page). In this case your method will not work as it will just stop the generation. 1) does this case happen? 2) how would you deal with this issue? \n* Do you handle references to other parts of the page? (Ref to tables, figures in the text?). It might not be crucial when dealing with pages, but it will be useful at document-level.\n* Do you handle references to the original image/document (e.g. select a word in the Markdown to highlight it in the PDF)? This feature could be used by users to check an equation if it does not make any sense, for example.\n* To go beyond recognition, how could Nougat be adapted to a multitask setting with e.g. Visual Question Answering / Neural Machine Translation / Summarization? It would certainly be useful to allow users to ask questions about the article: ask for an explanation, ask for a reference, ask to translate or summarize...\n\nErrors/typo\n* Table A.1. The total does not add up to the number of pages for each source (should be 8,494,841). As a consequence, % for each source are also wrong.\n* Typo p9 (last paragraph of section 5.4) \"to compute the to the end \"", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The experimental parameters are meticulously described in the article, with extensive information covering preprocessing, model architecture, and training parameters. Notably, the authors have made both the code and the model available on GitHub, which strengthens the reproducibility of their work. The level of detail in their descriptions provides confidence in the potential to replicate these results.\n\n- The methodology used to create the training dataset is also well described and will be available on GitHub. Scientific articles from three sources (ArXiv, PubMed, and IDL) are used. The process of automatic conversion from LaTeX to Markdown is explained step by step, with particular emphasis on the quality control measures taken. The appendix describes the dataset in detail.\n\n- Nougat outperforms GROBID by a substantial margin in the recognition of plain text, tables, and mathematical expressions. Notably, the significant improvement in handling mathematical equations is particularly impressive, given that this was a significant weakness of GROBID.", "weaknesses": "- Nougat outperforms GROBID by a substantial margin in the recognition of plain text, tables, and mathematical expressions. However, I feel that the results section is lacking a qualitative comparison between GROBID, Nougat-base and Nougat-small, especially on mathematical expressions.\n\n- The GROBID system is not sufficiently described, and authors should comment its performance on plain text (what is LaTeX OCR?)\n\n- Two Nougat models are compared in the paper (base and small). However, the article does not provide a comprehensive comparison of these architectures (performance gain, training and inference times, energy consumption…). Consequently, it leaves a significant question unanswered: whether it’s justifiable to opt for the base model when the small model performs almost as well..\n\n- I found the authors’ perspective on how to deal with repetitions very valuable, as this is a well-recognized challenge when working with transformer models. However, the gain should be measured for the repetition detection module, especially since the threshold is set manually and would be painful to adapt to other models/applications.\n\nFinally, some limitations of this work are highlighted, although I feel that this section could be developed further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bbbv7Vag8L", "reviewer": "Reviewer_jN3C", "summary": "This paper presents a way to build a system that converts an image of a scientific document to its Markup representation from both modeling and data collection perspectives. The use of Markup as its representation enables to handle texts, math equations, tables uniformly by a single model. The model is a basic end-to-end transformer-based autoregressive model. In order to mitigate the common repetition problem of such models, a simple data augmentation method is proposed. It also describes the data construction procedure to create image-Markdown pairs from arXiv where several practical techniques are required. The experimental results show that the transformer-based model is capable for the task and outperforms a baseline system. The code and pre-trained model will be released on GitHub.", "questions": "I think the important details are well described and there is no particular question.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "I see there are mainly three contributions in the paper:\n\n1. It proposes to use Markdown to extract structure information from images of scientific documents.\n\nThe use of such representations (e.g. LaTex, html, etc.) in the OCR domain is not novel (e.g. math equation recognition has been studied actively in the community for many years), but proposing to use such representations for this particular domain and tackling the problem seriously should get some credit. If scientific documents stored as images can be converted to Markdown, there should be great values to humanity.\n\n2. It describes the data construction procedure for the task.\n\nGiven the capability of latest ML models, the most challenging part is to find data to train models. Identifying arXiv as a good source for the problem and providing practical procedure to construct a dataset seems to have good values to subsequent studies.\n\n3. It shows a transformer-based end-to-end model works for the task and a simple technique further improves the accuracy.\n\nIt is not very surprising that the Donut-based model works for the in-domain data, but it should have a certain value to show that it works well in the setting. The proposed data augmentation is simple but effective, and good analyses are given.", "weaknesses": "I am not very certain about its scientific contributions to the ML community. \n\nIn a sense, this paper narrows down the large problem space of OCR into the limited domain and proposes a specialized solution to it. There is definitely a practical value if we can extract Markdown for scientific documents and it is important to have a technical solution to the problem. However, it is not really surprising that a transformer-based model can do a reasonable job for the task. In my opinion, it is challenging to claim novelty for the adoption of transformer for the task and by releasing code and pre-trained models. In my view, the novelty could be claimed for the problem definition and the data construction procedure. Both may be valuable in a certain field, but I am not very sure how much they are for a top-tier general ML conference. I see a lot of high-quality work in the paper, but I feel ICLR is not the best destination for the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VEf9P2jfnt", "reviewer": "Reviewer_YnWx", "summary": "This article presents an end-to-end model for converting PDF image files into a markup language, with a focus on accurately reconstructing mathematical formulas. The authors also discuss the preprocessing pipeline used to align PDFs with LaTeX source code and provide details on the datasets used. The model can be applied to other types of scanned document images.", "questions": "1. What does “out-of-domain documents” refer to on page 8, and what is the objective of conducting experiments on repetition data?\n\n2. In Eq. X, does “a” or “b” denote a paragraph index?\n\n3. Why were Table, Plain text, and Math treated as separate modalities for experimentation, and how does this differ from conventional methods?\n\n4. Can the model be directly applied to visually rich document understanding tasks?\n\n5. Would it be better to move the first paragraph in Section 3 Model from page 2 to the related work section?\n\n\nOther suggestions:  give eq. no and full stops", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The proposed model significantly improves the processing of scientific documents into a markup language.\n\n2. The model performs exceptionally well in parsing mathematical formulas, surpassing algorithm-based OCR engines.\n\n3. The method is simple yet effective.", "weaknesses": "1. The lack of novelty in the model architecture’s methodology is a weakness. It suggests that the approach may not bring any new or innovative ideas to the field. It would be better for an industrial track paper.\n\n2. The reported results being limited to English data is a weakness. It is unclear if the model can support other languages, such as Chinese or Japanese. However, it would be interesting to evaluate the transfer learning ability of the model for layout analysis, which could potentially address this limitation.\n\n3. Figure 5 and section 5.4 on repetitions during inference being difficult to follow. It indicates that the explanation or presentation of this aspect of the model may not be clear or well-explained, making it challenging for readers to grasp the concept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)