<figure><img src="image_1.png" /></figure>

HAL
open science

# Understanding and Extracting Table Information from BRGM Documents

Marijan Soric

<figure><img src="image_2.png" /></figure>

## To cite this version:

Marijan Soric. Understanding and Extracting Table Information from BRGM Documents. Computer Science [cs]. 2025. hal-04974179

HAL Id: hal-04974179

https://inria.hal.science/hal-04974179v1

Submitted on 3 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Understanding and Extracting Table Information
from BRGM Documents

by

Marijan Soric

Final Internship Report

In partial fulfillment of the requirements for

DIPLÔME D'INGÉNIEUR

at

CENTRALE LYON

February 2025

**Authored by:**

Marijan Soric
Centrale Lyon
Inria, DI ENS, CNRS, BRGM
marijan.soric@etu.ec-lyon.fr

**Supervisors at Inria:**

Pierre Senellart and Ioana Manolescu
pierre@senellart.com, ioana.manolescu@inria.fr

**Supervisor at BRGM:**

Cécile Gracianne
c.gracianne@brgm.fr

**Supervisor at Centrale Lyon:**

Yohann De Castro
# Contents

<table><tr><td>1</td><td>Introduction</td><td>3</td></tr><tr><td>1.1</td><td>Internship context</td><td>3</td></tr><tr><td>1.2</td><td>Internship goal</td><td>3</td></tr><tr><td>2</td><td>Task definition</td><td>3</td></tr><tr><td>2.1</td><td>Table Detection</td><td>4</td></tr><tr><td>2.2</td><td>Table Structure Recognition</td><td>4</td></tr><tr><td>3</td><td>Extraction methods</td><td>4</td></tr><tr><td>3.1</td><td>GROBID</td><td>5</td></tr><tr><td>3.1.1</td><td>GROBID: end-to-end model pipeline</td><td>5</td></tr><tr><td>3.2</td><td>LLM-Vision</td><td>6</td></tr><tr><td>3.2.1</td><td>LLM-Vision: end-to-end model pipeline</td><td>6</td></tr><tr><td>3.3</td><td>Table Transformer Extract (TATR-extract)</td><td>6</td></tr><tr><td>3.3.1</td><td>Detection Transformer (DETR)</td><td>7</td></tr><tr><td>3.3.2</td><td>TATR-detect</td><td>8</td></tr><tr><td>3.3.3</td><td>TATR-structure</td><td>8</td></tr><tr><td>3.3.4</td><td>TATR-extract: end-to-end model pipeline</td><td>9</td></tr><tr><td>3.4</td><td>Vision Grid Transformer + TATR-structure</td><td>11</td></tr><tr><td>3.4.1</td><td>Vision Grid Transformer (VGT)</td><td>11</td></tr><tr><td>3.4.2</td><td>VGT+TATR-structure: end-to-end model pipeline</td><td>12</td></tr><tr><td>3.5</td><td>XY-cut + Table Transformer Extract (XY+TATR-extract)</td><td>12</td></tr><tr><td>3.5.1</td><td>Motivations</td><td>14</td></tr><tr><td>3.5.2</td><td>XY+TATR-extract: end-to-end model pipeline</td><td>14</td></tr><tr><td>4</td><td>Evaluation</td><td>16</td></tr><tr><td>4.1</td><td>Evaluating Table Detection</td><td>17</td></tr><tr><td>4.1.1</td><td>Table detection evaluation based on bounding boxes(Bbox-based TD)</td><td>18</td></tr><tr><td>4.1.2</td><td>Table detection evaluation based on HTML mark-up (HTML-based TD)</td><td>24</td></tr><tr><td>4.1.3</td><td>Evaluation pipeline</td><td>25</td></tr><tr><td>4.2</td><td>Evaluating Table Structure Recognition</td><td>25</td></tr><tr><td>4.2.1</td><td>TEDS</td><td>26</td></tr><tr><td>4.2.2</td><td>GriTS</td><td>26</td></tr><tr><td>4.2.3</td><td>Evaluation pipeline</td><td>28</td></tr><tr><td>5</td><td>Results and analysis</td><td>30</td></tr><tr><td>5.1</td><td>Test dataset</td><td>30</td></tr><tr><td>5.2</td><td>Table Detection</td><td>30</td></tr><tr><td>5.2.1</td><td>Bounding box-based TD evaluation</td><td>30</td></tr><tr><td>5.2.2</td><td>HTML-based TD evaluation</td><td>34</td></tr><tr><td>5.3</td><td>Table Structure Recognition</td><td>36</td></tr><tr><td>5.3.1</td><td>Bounding box-based TSR evaluation</td><td>36</td></tr><tr><td>5.3.2</td><td>HTML-based TSR evaluation</td><td>38</td></tr><tr><td>5.4</td><td>Model confidence</td><td>40</td></tr><tr><td>5.5</td><td>Current pipeline limits: PDFAlto</td><td>40</td></tr><tr><td>5.6</td><td>Bounding box and HTML-based evaluation</td><td>41</td></tr><tr><td>6</td><td>Conclusion and outlook</td><td>42</td></tr><tr><td>6.1</td><td>Personal feedback</td><td>43</td></tr><tr><td>A</td><td>Dataset Test</td><td>44</td></tr><tr><td>B</td><td>Proofs</td><td>44</td></tr><tr><td>C</td><td>LLM-Vision hallucinations</td><td>46</td></tr></table>
# Abstract

## Résumé

Ce rapport synthétise mon stage réalisé au sein des équipes Inria Valda et Cedar, dans le cadre d'une collaboration avec le BRGM, le but étant d'extraire de manière structurée des tableaux provenant de données non structurées : des rapports scientifiques sous format PDF. L'enjeu est de mettre au point une extraction structurée tout en gardant en mémoire les incertitudes liées à celle-ci. L'extraction de table est divisée en deux sous-tâches : la détection de tables et la détection de leur structure. Nous avons comparé trois modèles d'extraction bout-en-bout, avec deux modèles de référence : un modèle d'extraction de PDF et un modèle de langue (avec vision). Les modèles essayés reposent sur la détection d'objets au sein d'images, et sont spécialisés sur les deux sous-tâches. Nous avons utilisé des métriques correspondant à nos attentes pour chacune de ces tâches, notamment concernant la calibration des probabilités des modèles, un point d'intérêt dans la construction d'une base de données probabiliste. Grâce aux modifications des chaînes de traitement et aux pré et post-traitements nous avons pu améliorer les performances obtenues. Les résultats sont globalement bons, mais le problème n'est pas entièrement résolu.

**Mots-clés** : Extraction de tables – Détection de tables – Reconnaissance de structures de tables – Apprentissage profond – Détection d'objets – PDF

## Abstract

This report summarizes my internship at Inria Valda and Cedar, which was the result of collaboration with BRGM to extract structured tables from unstructured data, specifically scientific reports in PDF format. The project aims at extracting structured information while conserving the uncertainty associated with it. Table extraction includes two subtasks: table detection and structure recognition. We compared three end-to-end models with two baselines: a PDF extractor and a large language model (with vision). The models we tried relied on object detection and had been trained on subtasks. We used metrics aligned with what we wanted to measure for each subtask. More specifically, we measured the model miscalibration, which is important for probabilistic knowledge bases. Through new pipelines and pre- and post-processing, we improved the models' performance. The results were relatively good, but table extraction is not yet a solved problem.

**Keywords:** Table extraction – Table detection – Table structured recognition – Deep learning – Computer vision – PDF
# 1 Introduction

## 1.1 Internship context

This internship took place in the context of a scientific partnership between BRGM, on one side, and Inria, on the other. BRGM is France's public reference institution in Earth Science applications for the management of surface and subsurface resources and risks. It also creates geological surveys of the French territory. BRGM reports to the ministries in charge of research, ecology, and economical matters. The National Institute for Research in Digital Science and Technology (Inria) is a French national research institution focusing on computer science and applied mathematics. Inria is under the double supervision of the French Ministry of National Education, Higher Education and Research and the Ministry of Economy, Finance and Industry.

As my supervisors belong to two different laboratories, I joined both the Inria team Valda (from Inria Paris) and also the Inria team CEDAR (from Inria Saclay). Valda is a joint research project-team between Inria Paris, École Normale Supérieure, and CNRS, within the Computer science department of École Normale Supérieure (DI/ENS, UMR 8548). The Valda team's focus is on both foundational and systems aspects of complex data management, especially human-centric data. CEDAR is a joint project-team of Inria Saclay and LIX (CNRS – UMR 7161 and École Polytechnique), focused on rich data analytics at cloud scale. My internship is co-supervised by Pierre Senellart, head of the Valda project-team and Ioana Manolescu, head of CEDAR project-team. Additionally, Cécile Gracianne, researcher data scientist at BRGM took part in my internship through weekly meetings and by linking up with BRGM and providing access to their data.

## 1.2 Internship goal

This internship is part of a larger project between BRGM and Inria: the "Augmented Geologist" (GeoAug) project. The GeoAug project aims at providing the scientific basis and the technological tools to support the creation, access, and interactive manipulation of data throughout the geologist's scientific process. The project brings together researchers from multiple domains at Inria - Data and Knowledge Representation and Processing; Geometry and Modeling; Interaction and Visualization - and multiple departments at BRGM. It is structured along three work packages: Exploitation and Structuring of Data and Knowledge; Terrain Modeling; and Interactive Data Access and Manipulation. This internship is a preliminary study for the "Exploitation and Structuring of Data and Knowledge" part.

For this internship, we focused on a specific data type: tabular data from unstructured documents (PDF file). Our goal is to extract data from tables present in PDF files. Indeed, tables in PDF reports have a specific structure that is understandable by the human eye (there is a specific way to read tables: row and column intersection means something) but not by simple automatic processing. To do so, we use table extraction methodology that has two subtasks: detecting tables in the file and understanding the table structure of each table. Thus we would be able to extract tabular information (content and structure) from unstructured documents to further store them into a knowledge base and exploit them. In other words, we want to store tabular data from unstructured data in a structured way, but automatic processing is harder task.

# 2 Task definition

The goal of our project is to extract the data content of tables, given an input document (typically a PDF file), while preserving their structure. This can be seen as a two-step process, consisting of two subtasks: detecting tables in the input, and recognizing each element in their structures. In some approaches, called *two-step models*, these tasks are independent, i.e., any combination of models could be used to solve the two tasks independently. In contrast, *one-shot models* may solve them together. In this work, we will consider the tasks separately (Section 2.1 and 2.2 respectively), which allows us to evaluate each aspect of our table information extraction independently.
## 2.1 Table Detection

The first step is table detection. It consists of detecting tables from an input page given as an image or as a PDF document. Some methods require specific types of input images. The simplest way to define a table is to define it as a rectangular region (bounding box) within the input image or document; indeed, this is how most models also represent tables. Specifically, we can model the input page as a bitmap, which is an image of size ($H, W$) $\in \mathbb{N}^{*2}$, where $H$ is the height and $W$ is the width of the image, that are the number of pixels along each axis. A table is typically recognized and output as a bounding box $(x_0, y_0)$, $(x_1, y_1) \in [1:H] \times [1:W]$, where $x_0, y_0, x_1$ and $y_1$ are the integer pixel coordinates of the top left corner and bottom right corner of the bounding box respectively. Some models do not output bounding boxes, but HTML mark-up that describe the extracted table structure at a logical level (table rows, columns, and cells). For these models, we evaluate the extraction quality on the detected HTML mark-up (Section 4.1.2).

## 2.2 Table Structure Recognition

Once a table is detected, we want to extract information from it, more specifically, its structure and content. The output format should be structured; we should be able to retrieve original table information from this structure. For instance, we could use the table's cell bounding boxes coordinates with their content, or a structure describing tables, i.e., lines, columns, cells, etc. In particular, following the HTML convention, a table can be seen as trees that have rows as nodes of the table, and the cells as nodes of each row (and the table's leaves). Both representations allow to describe a table. However, in the latter method, we use only essential information to define the structure: relative position, rather than absolute coordinates position. In our evaluation (Section 4.2), we will consider metrics to evaluate table structure recognition with their content.

# 3 Extraction methods

We now present the extraction methods that we have studied for our tasks.

An end-to-end model takes a document as input (or a page) and extracts tables (table detection, and table structure recognition). Different types of approaches have been used over time. First, heuristics relied on hard-coded patterns; more recently, trained Machine Learning models performed our tasks during inference, by generalizing what they learned. The first Machine Learning approaches required heavy feature engineering. This led to Deep Learning approaches based on deep neural networks with different types of architectures, that outperform previous approaches while also being more versatile. These advantages are due to: larger training dataset, bigger network and a more efficient architecture.

Concretely, we studied two baseline models: GROBID [13] and LLM-Vision. The former is a reference library using machine learning for extracting, parsing and re-structuring raw PDF documents (including table extraction), while the latter is a multi-modal large language model based on a transformer-decoder architecture. We wanted to check the raw capacities of an LLM with simple prompt to get a more challenging baseline, because of their increasing usage in various specialized tasks.

We also considered two-step models, which tackle each subtask as an object detection challenge. Here, one of the state-of-the-art model is TATR [32]. Specifically, TATR-extract (this is how we will call this end-to-end model) consists of a detection model TATR-detect, followed by a recognition structure model called TATR-structure, both based on the same architecture: a transformer encoder-decoder. Another interesting model is VGT [7], specialized in document layout detection that determines document components. Thus, we also built the pipeline VGT+TATR-structure, that replaces TATR-detect for the detection task by VGT, specialized in layout detection, while keeping TATR-structure for the table structure recognition. Finally, we also built the pipeline XY+TATR-extract, that is exactly TATR-extract but with a post-processing on input page done by the algorithm XY-cut that try naively to isolate layout component.

Table 1 synthesizes all end-to-end models studied, and raw characteristics: input and output type, something that will shape the way we evaluate models.

In the sequel, we describe GROBID in Section 3.1, LLM-Vision in Section 3.2, TATR in Section 3.3, VGT in Section 3.4 and XY in Section 3.5. We will provide more details on TATR and VGT since they are the best-known baseline methods.
<table><caption>Table 1: End-to-end models for table extraction.</caption><thead><tr><th rowspan="2">End-to-End Models</th><th rowspan="2">Input type</th><th colspan="3">Output</th><th rowspan="2">Steps</th></tr><tr><th>Location</th><th>Confidence score</th><th>HTML mark-up</th></tr></thead><tbody><tr><td>GROBID (baseline)</td><td>PDF</td><td>✓</td><td></td><td>✓</td><td>1</td></tr><tr><td>LLM-Vision (baseline)</td><td>Image</td><td></td><td></td><td>✓</td><td>1</td></tr><tr><td>TATR-extract</td><td>Image</td><td>✓</td><td>✓</td><td>✓</td><td>2</td></tr><tr><td>VGT+TATR-structure</td><td>Image</td><td>✓</td><td>✓</td><td>✓</td><td>2</td></tr><tr><td>XY+TATR-extract</td><td>Image</td><td>✓</td><td>✓</td><td>✓</td><td>2</td></tr></tbody></table>

## 3.1 GROBID

GeneRation Of BIibliographic Data (GROBID) is a machine learning library, from Inria, for extracting, parsing and re-structuring raw documents such as PDF into structured XML/TEI encoded documents with a particular focus on technical and scientific publications. The functionality used in our case is the ability to perform full text extraction and structuring from PDF articles. GROBID includes a model (that we will call GROBID from now on) for the overall document segmentation and models for the structuring of the text body (paragraph, section titles, reference and footnote callouts, figures, tables, data availability statements, etc.). In a complete PDF processing, GROBID manages 68 final labels used to build relatively fine-grained structures, from traditional publication metadata (title, author names, pages, etc.) to full text structures (section title, paragraph, figure captions, etc.). In other words, GROBID outputs a structured XML file that represents the input PDF file. Notice that GROBID is used in practice when uploading PDFs to HAL<fnref n="1" />. More particularly, GROBID extracts contents from PDFs tables and structures them with tags: <table>, <row> and <cell>. Table coordinates are <table> tag's attributes (page number, and bounding box point<fnref n="2" /> coordinate). In our case, we will only retrieve information related to tables, that are: table's location and structure.

GROBID's default configuration (the one we used) uses a feature-engineered machine learning approach with cascading of linear chain of Conditional Random Fields (CRF). CRF are a discriminative model for structured-output data.

### 3.1.1 GROBID: end-to-end model pipeline

GROBID takes as input a whole PDF document, contrary to other models that only handle individual bitmap images of PDF pages.

<figure><img src="image_4.png" /><figcaption>Figure 1: GROBID pipeline for table extraction. GROBID outputs a TEI.XML structured file from which we retrieve all tables extracted: table location (page number and bounding box coordinates $(x_0, y_0, x_1, y_1)$) and the table as XML mark-up</figcaption></figure>

<footnote n="1">¹https://hal.science/</footnote>

<footnote n="2">²The units of User Space (coordinate system on a PDF page) are called “points”. They are not pixels.</footnote>
Examine the image for any tables. If tables are present, write the HTML code to replicate them, starting each table with <table> and ending with </table>. Avoid using line breaks (\n) or additional text. Provide only the necessary HTML structure and content for the tables.

<caption>Figure 2: Sample prompt used for LLM-Vision.</caption>

## 3.2 LLM-Vision

The second baseline model chosen is a **multi-modal large language model** (text and vision) because they have showed versatility and strong availabilities on various benchmark. We use the closed-source model “GPT-4o mini” by OpenAI<fnref n="3" />: a cost-efficient small model that we will call “LLM-Vision” from now on. Table extraction is a very easy task for human, even tough annotation can be ambiguous, thus as most recent LLMs demonstrate human-matching capacities in some tasks, we wanted to see how good can the model be. Experiments are done with the API<fnref n="4" />. We evaluate LLM-Vision with zero-shot learning over the extraction task. “Zero-shot learning” simply means that we gave a simple prompt (a page as an image and a prompt), without any example of correct Input-Output (no few shot learning), just a simple prompt that asks the model to extract tables. The image size in input is set with parameter “high”: LLM-Vision looks at the input with low resolution image 512 × 512 pixels and then creates detail crops for each 512 × 512 pixels tile. The goal is to have a baseline with naive inference on LLM-Vision. The LLM-Vision has several limitations concerning vision capabilities: the model struggles with small text and tasks requiring precise spatial positioning.

Contrary to other models (see table 1), LLM-Vision does not output table locations (bounding box): it cannot return precise pixel output coordinates. This leads to a challenge that we will discuss in Section 4.1.2: how to evaluate table detection based only on HTML mark-up?

### 3.2.1 LLM-Vision: end-to-end model pipeline

LLM-Vision takes as input a page as an image with the prompt in Figure 2.

Figure 3 depicts the pipeline used for table extraction.

<figure><img src="image_5.png" /><figcaption>Figure 3: LLM-Vision pipeline for table extraction. A page as an image with the prompt in Figure 2 (zero-shot learning) is sent to the LLM-Vision API, then we can collect tables as HTML mark-up.</figcaption></figure>

## 3.3 Table Transformer Extract (TATR-extract)

TAble TRansformer (TATR) is composed of two models: one for table detection, and the other for table structure recognition. Both models use the DETection TRansformer (DETR) [5] architecture and have been trained on different datasets: a dataset containing report images for the table detection model, and table images for the structure recognition model.

<footnote n="3">³https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/?utm_source=chatgpt.com</footnote>

<footnote n="4">⁴https://platform.openai.com/docs/guides/vision</footnote>
### 3.3.1 Detection Transformer (DETR)

DETR is a transformer encoder-decoder based model for the object detection task. The architecture is divided into two parts: a **backbone model** (CNN) followed by a transformer. The former extracts image features, that are fed then into the latter; its neural architecture enables DETR to predict all objects at once (in parallel). Figure 4 outlines this architecture.

DETR has the advantage of suppressing the need for hand-designed components present in prior detection approaches. Specifically, the usage of a bipartite matching loss function supplants the non-maximum suppression (NMS) as a post-processing on candidate detections. NMS removes model predictions that overlap (in terms of bounding box) with other predictions with higher confidence score. Second, anchors (regions where to look for candidate objects to extract, used in previous object detection models) used in prior works are not needed in DETR, which, instead, learns a small fixed number of position encodings, so-called object queries, that it uses in order to extract.

<figure><img src="image_6.png" /><figcaption>Figure 4: DETR architecture, from [5]. DETR uses a conventional CNN backbone to learn a 2D representation of an input image. The model flattens it and supplements it with a positional encoding before passing it into a transformer encoder. A transformer decoder then takes as input a small fixed number of learned positional embeddings, which we call object queries, and additionally attends to the encoder output. We pass each output embedding of the decoder to a shared feed forward network (FFN) that predicts either a detection (class and bounding box) or a “no object” class.</figcaption></figure>

In details, the bipartite loss function used in DETR is $\mathcal{L}_{\text{Hungarian}}$ defined as shown below. We recall the details because we will reuse the idea of bipartite matching in order to evaluate the quality of the extractions:

$$ \hat{\sigma} = \arg \min_{\sigma \in \mathcal{S}_N} \sum_{i=1}^{N} \mathcal{L}_{\text{match}}(y_i, \hat{y}_{\sigma(i)}) \quad (1) $$

$$ \mathcal{L}_{\text{match}}(y_i, \hat{y}_{\sigma(i)}) = -\mathbb{1}_{[c_i \neq \emptyset]} \hat{p}_{\sigma(i)}(c_i) + \mathbb{1}_{[c_i \neq \emptyset]} \mathcal{L}_{\text{box}}(b_i, \hat{b}_{\sigma(i)}) \quad (2) $$

$$ \mathcal{L}_{\text{box}}(b_i, \hat{b}_{\sigma(i)}) = \lambda_{\text{iou}} \mathcal{L}_{\text{iou}}(b_i, \hat{b}_{\sigma(i)}) + \lambda_{\text{L1}} \|b_i - \hat{b}_{\sigma(i)}\|_{1} \quad (3) $$

$$ \mathcal{L}_{\text{Hungarian}}(y, \hat{y}) = \sum_{i=1}^{N} \left[ -\log \hat{p}_{\hat{\sigma}(i)}(c_i) + \mathbb{1}_{[c_i \neq \emptyset]} \mathcal{L}_{\text{box}}(b_i, \hat{b}_{\hat{\sigma}(i)}) \right] \quad (4) $$

where:

* As commonly used, the $\hat{\sigma}$ designates predictions;

* $N$ is the number of predictions (recognized objects); in DETR, this is always at least as large as the number of ground truth objects;

* $\sigma \in \mathcal{S}_N$ is a permutation over the $N$ predicted objects in each output; the permutation corresponds to a certain bipartite match; thus, $\hat{\sigma}$ is the permutation corresponding to the optimal one-to-one match, determined by the Hungarian algorithm [16];

* Each $\hat{y}_{\sigma(i)}$ is a prediction of the model, and $y_i$ is the ground truth (GT) assigned by the permutation to the detection at $i$;

* Each $c_i$ is a predicted class (we will discuss below which classes are used in TATR for table detection, respectively, table structure understanding).

* $\hat{p}_{\sigma(i)}(c_i)$ is the probability assigned to class $c_i$ in the prediction as position $\sigma(i)$;
* $\hat{b}_{\sigma(i)}$ is the bounding box of the prediction $\hat{y}_{\sigma(i)}$ and $b_i$, the bounding box ground truth assigned by the permutation to the detection at $i$;
* $\mathcal{L}_{iou}$ is the intersection-over-union distance between two bounding boxes (based on the surfaces that they share);
* $\mathcal{L}_{box}$ is a distance between two bounding boxes computed as a linear combination of $\mathcal{L}_{iou}$ and the $l^1$ distance of the boxes;
* $\lambda_{iou}, \lambda_{L1} \in \mathbb{R}$ are hyperparameters.

DETR outputs a fixed number of predictions that include bounding boxes, object labels and confidence score distribution. We get two TATR models (DETR ones) trained on different tasks: table detection and table structure recognition. Let's discuss about the two models more specifically.

### 3.3.2 TATR-detect

From now on, we will call TATR-detect the table detection model from TATR. TATR-detect performs object detection with 3 classes: {“table”, “table rotated”, “no object”} where the category “table” is used for those whose text has the same orientation as the text in the rest of the document (page), while “table rotated” designates tables which have been rotated by 90 degrees (anticlockwise). During inference (over a page as an image), TATR-detect outputs 15 predictions, each of them contains bounding box coordinates and confidence score distribution over labels. These confidence scores express “How confident is the model that this bounding box contains the object i?”. Authors chose 15 predictions per inference (at most 15 tables within an input page). Let $c_i$ denote the confidence score for label $i \in \{1, 2, 3\}$. Note that authors use $\hat{i} \in \arg\max_i c_i$ to chose a label and that for inference, this technique is used to display one label per bounding box. This may be considered as a loss of information; as we will see (Section 3.3.4), an alternative is to keep the predictions whose confidence is above a certain threshold.

### 3.3.3 TATR-structure

From now on, we will call TATR-structure the structure recognition model of TATR. TATR-structure performs object detection with 7 classes: {“table”, “table column”, “table row”, “table column header”, “table projected row header”, “table spanning cell”, “no object”}. During inference, TATR-structure takes a cropped table image as input and outputs 125 predictions, each of them containing bounding box coordinates and confidence score distribution over labels. Generally, there is a padding $p_{px,s}$ added to the strict “cropped table image” to improve model performance (input data should look like the training data used on DETR, that created TATR-structure). From this raw model’s output, there are several post-processing steps applied that will transform the output into a real table structure, but also that complete the actual extraction with additional data like textual information.

1. Once we obtained TATR-structure's predictions (bounding boxes and confidence score), we process them into a consistent set of table structures (rows, columns, panning cells, headers), that include resolving conflicts/overlaps, and ensuring the boxes meet certain alignment conditions (for example: rows should all have the same width, etc.).

2. Now that we have the table structure, by providing a set of words (tokens) we fill the table with its actual textual content (from the input image). Retrieving these data (which can be done with OCR) is not addressed by the TATR repository. We decided to extract tokens (content and localization with bounding box) from the PDF from which the page image was taken, with a PDF parser tool, PDFAlto<fnref n="5" />, used in GROBID. We fill cells with tokens that appears within the cells box comparing bounding boxes overlap between cells and tokens.

3. Finally, table structures are processed into table cells, which are later exported to HTML mark-up.

To sum up, we provide in input a cropped table image and a set of textual data (tokens content and location within the input image) into TATR-structure, that outputs (after some post-processing steps) the table in HTML mark-up.

<footnote n="5">⁵https://github.com/kermitt2/pdfalto.</footnote>
### 3.3.4 TATR-extract: end-to-end model pipeline

We use a slightly different end-to-end pipeline that the one provided for inference with visualization (downstream task).

For each prediction of TATR-detect output, they used to assign a unique label with a confidence score: by choosing the label with highest confidence score over all the labels: $\hat{i} \in \arg\max_i c_i$. However, we noticed that sometimes tables failed to be “detected” even though their table bounding box had been found by the model, because the label assignment failed: the “no object” label has a slightly higher confidence score than the “table” one (see Figure 9a). Thus, we rather decided to use another post-processing, that assigns a label that is not “no object”: either “table” or “rotated table”$^6$. This eliminated the previous failure cases. Nevertheless, we obtain more “tables” predictions than before: among 15 predictions only few of them were assigned as tables before ($\hat{i} \in \arg\max_{i \in \{1,2,3\}} c_i$), whereas now we have 15 tables predictions, rotated or not ($\hat{i} \in \arg\max_{i \in \{1,2\}} c_i$).

As all the predictions thus obtained may not be relevant, we use the Non Maximum Suppression (NMS) algorithm, widely used in object detection models post-processing. It allows to keep predictions with higher confidence score among those with overlapping areas.

For the remaining predictions, before feeding them into TATR-structure, we remove the predicted tables that do not contain any tokens within their predicted bounding box. As explained in Section 3.3.3, TATR-structure takes as input a cropped table image and a set of tokens (with positions). To obtain the token set, we run PDFAlto on the whole PDF document from which the input image is taken. From the PDFAlto output, we can retrieve the tokens located inside each “cropped table as an image”, which is the output of TATR-detect and an input to the TATR-structure. Thus, if no token is found, the table in HTML mark-up is empty: it is a structure with mark-up without any content. Thanks to this post-TATR-detect processing, or more precisely pre-TATR-structure processing, we can exclude tables detected that are actually empty (we consider that empty tables are useless). However, some tables in PDF documents can actually be present in the PDF in the form of bitmap images; our PDF parser (PDFAlto) cannot find the tokens in these tables. Thus, some image-tables that have been detected would not be recognized as tables any more. Intuitively, with this processing, we will obtain a higher precision but a slightly lower recall (see Section 4.1). For what concerns the parameters, we set $p_{px,s} = 100$ pixels, that determines how much padding in pixels will be added around a detected table before outputting a cropped image of the table. Figure 5 synthesizes the key steps pipeline of the end-to-end TATR-extract: TATR-detect+TATR-structure. Post-processing steps (NMS and empty table check) are not displayed.

<footnote>$^6$If it is a rotated table, the cropped table image and tokens within are rotated by 270 degrees (anticlockwise) before TATR-structure.</footnote>
<figure><img src="image_7.png" /><figcaption>Figure 5: TATR-extract pipeline for table extraction. TATR-detect takes a page as an image in input and predicts table bounding boxes with confidence score. Post-processing is not shown. Then, the initial image is cropped with bounding box predicted. These cropped tables as images are individually, send into TATR-structure with tokens content and localization from the new input. Then TATR-structure outputs table as HTML mark-up. Finally, we add TATR-detect outputs to obtain the final output of TATR-extract, the end-to-end model.</figcaption></figure>
## 3.4 Vision Grid Transformer + TATR-structure

The next end-to-end model is VGT+TATR-structure, where VGT replaces TATR-detect for the table detection.

Instead of using a model specialized in table detection, we can use a model specialized on a more general task, that includes table detection. Some models are specialized in Document Layout Analysis [2] (DLA) which consists of transforming documents into structured representations in order to then perform document information extraction. Different approaches have been used over time: some based on visual information (like TATR-detect, that only performs table detection and not the whole layout segmentation), but more recently, models also use textual information<sup>7</sup> from document. This is the case for our next model.

### 3.4.1 Vision Grid Transformer (VGT)

Vision Grid Transformer (VGT) is a multi-modal document layout analysis that uses both a vision model and a grid model. Contrary to previous TATR models that exclusively rely only on visual information, VGT also relies on textual information trough 2D semantic representations. We use the VGT architecture with weights fine-tuned on the PubLayNet [37] benchmark, that performs object detection on images and classifies among six classes, including a “table” label (the only that matters for us). This model achieves state-of-the-art results with the best Average Precision score on Table detection (see Section 4.1 for definition) over PubLayNet validation dataset. VGT have been pre-trained simultaneously with visual and textual supervision unlike previous multi-modal DLA model like VSR [35] (only visual supervision). As shown in Figure 6, the model is composed of two parts:

<figure><img src="image_8.png" /><figcaption>Figure 6: The model architecture of Vision Grid Transformer (VGT) with pre-training objectives for the GiT branch, from [7].</figcaption></figure>

1. A Vision Transformer (ViT), that models visual information, inspired by ViT [10] and DiT [19]. The input image is split into non-overlapping $P \times P$ patches that are flattened and linearly projected to get a sequence of tokens embeddings $X \in \mathbb{R}^{N \times P^2 C}$ that is then feed into ViT (where C is channel size and N the number of tokens).

2. A Grid Transformer (GiT), that models 2D language information. The input image is seen as a 2D token-level grid which feeds GiT (architecture similar to ViT). GiT has been pre-trained with Masked Grid Language Modeling (MGLM), inspired by BERT's Masked Language Modeling [9] in order to learn better token-level semantics for grid features.

Both encoder transformers have been modified in order to become multi-scale framework: they output the input image embeddings at different scales. These transformers outputs 4 visual features $V_i \in \mathbb{R}^{H/2^i \times W/2^i \times D}$ (with different scale feature $i \in \{2, \dots, 5\}$ and $(H, W)$ input image size), and

<footnote n="7"><sup>7</sup>TATR-structure uses tokens position only for post-processing tables, not during the actual detection task (row, columns, header, etc.)</footnote>
textual features $S_i$ (same shape as $V_i$) are then combined with element-wise sum $Z_i = V_i + S_i$ to form multi-scale multi-modal features. Then, we employ a Feature Pyramidal Network [22] (FPN) to refine pyramid features $Z_i$ further, by extracting multi-scale features. Finally, we can extract RoI features from different levels of the feature pyramid according to their scales for later object detection.

To sum up, the goal of the whole two-stream framework is to be the feature backbone that generates better embeddings from the input image page in order to feed detection framework (with FPN) that performs object detections with 6 classes: {“title”, “text”, “figure”, “table”, “list”, “no object”}. We used as detection framework the Cascade R-CNN [4] detector with FPN, which is implemented based on the Detectron2 [33].

### 3.4.2 VGT+TATR-structure: end-to-end model pipeline

To create an end-to-end model for table extraction based on VGT (table detector), we used the previous structure recognition model TATR-structure. Thus, the model pipeline is very similar to the previous one (TATR-extract, Section 3.3.4), except that VGT requires two inputs: a page as an image and a page as a grid. Then, VGT outputs predictions that are bounding boxes with a confidence score distribution over labels, we keep the “table” score. However, for a detected table, we don’t know whether it is rotated or not, and that matters in order to use TATR-structure that takes as input only non-rotated table. Thus, we need to determine whether or not the table detected is a rotated table. To do so, we simply look at the average shape<fnref n="8" /> (length and width) of tokens within the detected table. If $ΔL < ΔW$, we estimate that the table is rotated, with $ΔL$ is the average length of bounding box tokens within the table detected, and $ΔW$ the average width. Once a rotated table is detected, we use the same post-processing as TATR-detect: rotate it again. Before sending cropped table images into TATR-structure, we make sure that those tables contain tokens (see pre-TATR structure processing discussed Section 3.3.4). Then the pipeline is similar to the one of TATR-extract. As for the parameters values, we set $p_{px,s} = 100$ pixels.

In more details concerning the experiments, we used VGT’s pre-trained GiT weights, DiT-base weights for ViT, BERT tokenizer with LayoutLM [34] for word embeddings grid model. To obtain a grid input (tokens position and content on pages), we used the VGT repository [8]’s PDF parser; alternatively, PDFAlto could have been used.

## 3.5 XY-cut + Table Transformer Extract (XY+TATR-extract)

We have noticed that TATR-detect was less efficient on pages containing multiple tables. Even when it managed to detect them, their confidence scores were relatively low. In this section, we describe an alternative method where we use XY-cut as a pre-processing step for the TATR method. In Section 3.5.1 we explain the reasoning behind our choices, then Section 3.5.2 outlines our processing pipeline.

<footnote n="8">⁸Most of the time, tokens are multi-character words that are longer than they are wide (counting pixels).</footnote>
<figure><img src="image_9.png" /><figcaption>Figure 7: VGT+TATR-structure pipeline for table extraction. VGT takes a page as an image and a grid (obtained through PDF parser, tokenizer and embedding model) as input and predicts table bounding boxes with confidence score. Post-processing is not shown. Then, the initial image is cropped with bounding box previously predicted. These cropped table images are individually sent into TATR-structure with tokens content and localization from the new input. Finally, we add VGT outputs to obtain the final output of VGT+TATR-structure, the end-to-end model.</figcaption></figure>
### 3.5.1 Motivations

We wanted to go further by observing more precisely how TATR-detect “sees” and where it looks at, in order to improve detection capabilities. As seen (Section 3.3.1), DETR does not use region proposal network nor anchors for bounding box predictions: they come from the transformer decoder output. Object queries (transformer decoder input, see Figure 4) are positional embeddings learned during training and contain abstract information about spatial location or “where should the model look?” Thus the transformer decoder performs cross attention between object queries and transformer encoder output to output 15 embeddings. These output embeddings are fed into predictions heads that output bounding boxes and classes. In consequence, each output, which is directly associated to an object query, is roughly specialized in an area (Figure 8).

<figure><img src="image_10.png" /><figcaption>Figure 8: Object queries location for a white and black a input image. Above each “query i”, we plot the activation map of attention weights from the last transformer decoder layer. This corresponds to visualizing, for each detected object, which part of the image the model was looking at to predict this specific bounding box and class. Under each heatmap, we visualize the output: bounding box with dominant class (“tab” or “no”) and its confidence score. Approximately 5 queries are focused on the page bottom, 2 on the top, and 8 in the middle.</figcaption></figure>

As queries pay attention to different regions, we tried to provide image chunks (sub images) instead of whole pages as input for TATR-detect. For example, we cut a page into two chunks and gave each of them to the model. Small experiments have shown improved results: tables were detected with higher confidence score (Figure 9). We have also experimented by keeping the initial page shape (by adding padding to the image chunk), but results without padding were more promising.

To leverage this technique, we wanted to automatically cut pages (as an image) into chunks of interest. We used the recursive X-Y cut [15], a top-down page segmentation technique that decomposes a document image recursively into a set of rectangular blocks using image pixels. By counting the number of black pixels<fnref n="9" /> along each axis, X and Y, we obtain profile pixel distributions, and isolate components rectangle that create a sub-image. This process is recursively applied until stop criteria are met (a minimum area is found).

### 3.5.2 XY+TATR-extract: end-to-end model pipeline

We use the TATR-extract pipeline with the XY-cut algorithm to pre-process input images. Cropped images (XY-cut output) are now inputs of TATR-detect; we add a padding $p_{px,d} = 10$ pixels before

<footnote n="9">⁹Assuming that PDF reports are written in black on white background.</footnote>
<figure><img src="image_11.png" /><figcaption>(a) Whole page as input. Notice that for the first table "Tableau 33", TATR-detect confidence scores are: $c_{table} = 0.37$, $c_{no object} = 0.63$. With regular TATR-extract pipeline (Section 3.3.4), this table would not have been detected.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) Top page as input</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(c) Bottom page as input</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 10: XY cut algorithm project black pixels over the axis X and Y. From these profiles, we can detect gaps and create sub-image.</figcaption></figure>
TATR-detect. Instead of keeping 15 predictions (as we did for TATR-extract), we keep only the 2 most confident predictions because the new input (image chunk) might already contain at most one or two tables, no more. Image chunks contain almost no white space gap, there are titles, paragraphs, tables, figures, etc. This selection prevents the model from outputting too many tables. We apply the pre-TATR-structure processing, that is, we remove predicted table bounding boxes that do not contain any tokens. For what concerns the parameters, we set $p_{px,s}$ = 100 pixels.

<figure><img src="image_15.png" /><figcaption>Figure 11: XY+TATR-extract pipeline for table extraction. We pre-process a page as an image with the XY-cut algorithm that creates sub-images. TATR-detect takes these image chunks one by one in input and predicts table bounding boxes with confidence score. Only the top 2 predictions (with the highest confidence scores) are kept (not shown in the figure). Then, the initial image is cropped with the predicted bounding box (adding the XY-cut offset). These cropped tables as images are individually sent into TATR-structure with tokens content and localization from the new input. Finally, we add the XY+TATR-detect outputs to obtain the final output of TATR-extract, the end-to-end model.</figcaption></figure>

# 4 Evaluation

We now experimentally compare the performance of end-to-end models presented above. We organize the evaluation according to the sub-tasks: Table Detection (Section 4.1), Table Structure Recognition (Section 4.2). An overview of the evaluated methods for the various tasks appears in Table 2.
<table><caption>Table 2: Methods experimentally compared for the Table Detection and/or the Table Structure Recognition tasks.</caption><thead><tr><th rowspan="2">End-to-end pipeline</th><th colspan="2">Table Detect.</th><th rowspan="2">Table Struct. Reco.</th></tr><tr><th>Bbox</th><th>HTML</th></tr></thead><tbody><tr><td>GROBID (baseline)</td><td>GROBID</td><td>GROBID</td><td>GROBID</td></tr><tr><td>LLM-Vision (baseline)</td><td>LLM-Vision</td><td>LLM-Vision</td><td>LLM-Vision</td></tr><tr><td>TATR-extract</td><td>TATR-detect</td><td>TATR-extract</td><td>TATR-extract</td></tr><tr><td>VGT+TATR-structure</td><td>VGT</td><td>VGT+TATR-structure</td><td>VGT+TATR-structure</td></tr><tr><td>XY+TATR-extract</td><td>XY+TATR-detect</td><td>XY+TATR-extract</td><td>XY+TATR-extract</td></tr></tbody></table>

## 4.1 Evaluating Table Detection

The Table Detection task can be seen as an object detection task, well-known in the area of Computer Vision. Objects are located by their bounding box, and their label confidence score are used for measuring performance. However, as seen above (Table 1) not all models output bounding box location (and also confidence score). Thus we must use two different way to define tables and compare positive prediction with ground truth annotation, that is why we used two evaluation methods:

1. **Bounding boxes-based table detection evaluation** (Section 4.1.1) For TD based on bounding boxes, we can compare GROBID, TATR-detect, XY+TATR-detect and VGT<fnref n="10" />. We evaluate table detection with table location through their bounding boxes.

2. **HTML mark-up-based table detection evaluation** (Section 4.1.2) For what concerns TD based on HTML tables, we can compare GROBID, LLM-Vision, TATR-extract, XY+TATR-extract and VGT+TATR-structure. We evaluate table detection with table content through their HTML mark-up representations.

Having modeled our task as an object detection, similar to the problems tackled in information retrieval, we can evaluate performance by computing the traditional metrics such as Precision and Recall for the label “table”. Specifically, a True Positive (TP) is a table correctly recognized, a False Positive (FP) is a table found by the model where a human user would not consider a table exists, and a False Negative (FN) is a table recognized by human users but missed by the model.

**Notation** We need two thresholds, one for defining positive predictions $\theta_c \in [0, 1]$, and the other $\theta_J \in [0, 1]$ to distinguish between TP and FP.

* Let $\mathcal{P} := \{(\hat{y}, c)_i\}$ the set of model predictions, with $\hat{y}$ the output (bounding box or/and HTML mark-up) and c confidence score distribution over labels (if the model does not output scores, we consider that $c_{table} = 1$).

* Let $\mathcal{P}_{\theta_c}$ the subset of predictions such that $c_{table} > \theta_c$: $\mathcal{P}_{\theta_c} := \{(\hat{y}, c) \in \mathcal{P} | c_{table} > \theta_c\}$, with confidence score threshold $\theta_c \in [0, 1]$.

* Some metrics are functions of decisions, other are functions of probabilities. For the former, we define $\mathcal{P}^+ \subseteq \mathcal{P}$ the set of positive predictions (predicted tables), by using $\mathcal{P}^+ = \{\hat{y} | (\hat{y}, c) \in \mathcal{P}, c_{table} > \theta_c\}$, given a score threshold $\theta_c$. The set is now denoted $\mathcal{P}_{\theta_c}^+$. For the latter, we use a subset of $\mathcal{P}$.

* Let $\mathcal{P}^{++} \subseteq \mathcal{P}^+$ the set of True Positive predictions (predicted tables). We use $\mathcal{P}_{\theta_J}^{++} = \mathcal{P}_{\theta_J}^+ \cup \{\hat{y} \in \mathcal{P}^+ | \exists y, f(\hat{y}, y) > \theta_J\}$, with f computing the similarity between prediction and ground truth table.

**Threshold values** Let's take a closer look at the roles played by $\theta_c$ and $\theta_J$.

* **Confidence score threshold $\theta_c$** is used for models like TATR-detect and VGT that output *confidence scores*. Such model output (elements in $\mathcal{P}$) contain a bounding box (or HTML mark-up in the HTML-based TD) and a confidence score for the label “table”. This threshold $\theta_c$ sets

<footnote n="10">¹⁰For two-steps models for TD evaluation, we can compare their TD model independently from their TSR. However, for TSR evaluation, we consider the TD-model because it conditions the TSR-model input.</footnote>
a decision function, to filter out predictions, in order to obtain $P^+ = P_{\theta_c} \subseteq P$. Note that for models that do not output confidence scores (GROBID, LLM-Vision), we consider that $P^+ = P$. The threshold choice is important because it will influence the count of TP, FP and FN during evaluation. A lower threshold implies many predictions kept with poorer quality (low confidence score predictions would be considered as positives) which decreases Precision. A higher threshold implies a small number of prediction kept but we might miss some relevant predictions, which decreases Recall; an appropriate trade-off must be found. Some metrics (like AP, see below) overcome this issue by considering an aggregation over all values of $\theta_c$. However, in downstream tasks, we still need to fix a threshold $\theta_c$, in order to be able to output “relevant tables”. For instance, in two stage models (the table detection part followed by the table structure recognition part), $\theta_c$ is used to select those positive predicted tables that will be (if TP) the input of the TSR model.

* **Jaccard index threshold** $\theta_J$ is used during evaluation in order to distinguish a positive prediction between a TP and a FP (to define $P^{++} = P_{\theta_J}^+ \subseteq P^+$). This threshold $\theta_J$ sets a decision function: is the positive prediction related to a ground truth? Again there is trade-off between low and high $\theta_J$.

### 4.1.1 Table detection evaluation based on bounding boxes (Bbox-based TD)

From now on, we will discuss metrics that will use bounding box data (tables location with bounding box), but those metrics will also be used for HTML-based TD (similarity computation will be different).

**Basics** We want to count the number of TP, FP and FN in the prediction of a given model. First, we need to define the set of positive predictions $P^+$ given all $\mathcal{P}$ that we have. To do so, we use the set $P_{\theta_c}^+ = \{B_P \in \mathcal{P} | c_{\text{table}} > \theta_c\}$. Second, for a predicted table (a positive) bounding box, we need to find the associated ground truth table (GT) bounding box.

Let $B_P \in P^+$, and $B_{GT}$ be the ground truth bounding box (target) associated with this prediction. To distinguish if the prediction is a TP or an FP, the approach commonly used is to compute the Jaccard index (also called intersection over union, IoU in short<fnref n="11" />) between the areas of prediction and target, and to check whether this is above a given threshold $\theta_J$. Thus:

1. We compute IoU($B_P, B_{GT}$) = $\frac{A(B_P \cap B_{GT})}{A(B_P \cup B_{GT})}$ and consider the prediction a TP if $\text{IoU}(B_P, B_{GT}) > \theta_J$, and FP otherwise. A function returns input rectangle area.

2. Then, we can count the number of TP, FP and FN, using the threshold $\theta_J = \theta$, respectively denoted as $TP_\theta, FP_\theta, FN_\theta$.

3. The precision $P_\theta$ is the proportion of predictions that are TP among all prediction by the model: $P_\theta = \frac{TP_\theta}{TP_\theta+FP_\theta}$.

4. The recall $R_\theta$ is the proportion of predictions that are TP among all ground truth: $R_\theta = \frac{TP_\theta}{TP_\theta+FN_\theta}$.

5. We use the $F_{1,\theta}$-score to compare the overall performance of models on the table detection task, which is the harmonic mean of $P_\theta$ and $R_\theta$.

We used raw Precision, Recall and $F_1$-score (denoted $X_{\theta_J}^{\theta_c}$). These metrics are functions of decisions $P^+$, which requires $\theta_c$ for probabilistic models, and $\theta_J$ to define $P^{++}$.

**Ground truth assignment** Let's consider an image in input with $g$ ground truth tables in the image. Let $p_0$ be the number of tables predicted by the model. We keep only tables predicted with confidence score higher than a fixed confidence score $\theta_c$, the number of remaining tables is denoted $p \le p_0$. Then we want to count how many TP, FP and FN there are. Note that we don't have necessarily $p = g$; there might be FPs (whenever $p > g$) or FNs (whenever $p < g$). The main insight leveraged here is to view the problem of counting TP, FP, FN as an **assignment** problem between predictions and ground truths.

Specifically, we use a cost matrix of shape $p \times g$ that holds the $l^1$ distance between predictions and ground truth bounding boxes that are in $\mathbb{R}^4$. Then, based on the cost matrix, we create a one-to-one matching between $\min(p, g)$ predictions and ground truths. Here are the main steps of the algorithm used to count the TP, FP and FN:

<footnote n="11">¹¹For HTML-based TD, we will use Jaccard index too, that is why we keep the $\theta_J$ notation instead of $\theta_{IoU}$.</footnote>
1. On all images, we run the model to get the tables predicted. We keep only those with high confidence score (above $\theta_c$). Let $p = |\mathcal{P}_{\theta_c}^+|$ be the number of predictions remaining and $g$ the number of ground truth tables in one image.

2. We create matching (prediction, target) pairs, by minimizing the $l^1$ distance between their bounding boxes.

3. If $p > g$, predictions that have no ground truth associated to them are FP. If $p < g$, ground truths that have no prediction associated are FN.

4. For each matching pair (that is, a prediction bounding box, and a ground truth bounding box), we compute the Jaccard index $J$ between them. If $J > \theta_J$, we consider the prediction as a TP, otherwise it is a FP.

The process is formalized in Algorithm 1. Note that this algorithm is already implemented in library `pycocotools`<fnref n="12" /> for object detection evaluation, however we will use it for the HTML-based evaluation. Each prediction is an element of $\mathbb{R}^4$ as bounding box.

```
TP, FP, FN ← 0, 0, 0;
    for image ∈ dataset do
        $\hat{Y}$ ← predicted bounding boxes of objects with confidence score > $\theta_c$, $\hat{Y} \in \mathbb{R}^{p \times 4}$;
        $Y$ ← ground truth bounding boxes $Y \in \mathbb{R}^{g \times 4}$;
        $C = (\|\hat{Y}_i - Y_j\|_1)_{i,j} \in \mathbb{R}^{p \times g}$ cost matrix;
        I, J ← Hungarian algorithm(C) // There are $\min(p, g)$ matches
        FN ← FN + $g - \min(p, g)$ // Tables not detected
        FP ← FP + $p - \min(p, g)$ // Tables detected that have no ground truths
        for $(\hat{y}, y) \in (\hat{Y}[I], Y[J])$ do
            // Matching the predictions with the ground truth
            if IoU($\hat{y}, y$) > $\theta_J$ then
                TP ← TP + 1;
            else
                FP ← FP + 1;
                FN ← FN + 1;
            end
        end
    end
```

**Algorithm 1:** Computing Precision and Recall for the Table Detection task.

## Expected values of Precision and Recall: aggregating different Jaccard index threshold

Setting ground truth tables can be ambiguous: Where does a table really start and end? Should we include also the borders, or just the cell contents? That is why, rather than fixing a fixed threshold $\theta_J$, we would like to have a more global appreciation of results.

So far, we have used a fixed Jaccard index threshold $\theta_J$ during evaluation in order to distinguish a prediction (a positive) between a TP and a FP: $\mathcal{P}_{\theta_J}^{++} = \{\hat{y} \in \mathcal{P}^+ | \text{Jaccard}(\hat{y}, y) > \theta_J\}$. $\theta_J = 0.5$ is a natural choice, because if we consider non-overlapping predictions (NMS usually applied), we can get only one prediction with IoU greater than 50% (TP) per ground truth. Nevertheless, sometimes we want to count TP, FP with different values for $\theta_J$ in order to have stricter conditions on TP. For instance, the competition ICDAR 2019 cTDaR [12] computes $F_1$-scores with $\theta_J \in \{0.6, 0.7, 0.8, 0.9\} =: \mathcal{J}$ in order to get stricter metrics. The higher the threshold $\theta_J$ is, the harder it is for a positive to be considered as a TP. Then, they compute the metric WAvg($F_1$) that aggregates previous fixed threshold scores: it is a weighted average of $F_1$-scores, with weights proportional to the threshold. WAvg($F_1$) assigns a weight to each $F_1$ value of the corresponding Jaccard index threshold. As a result, the $F_1$-score with higher thresholds are given more importance than those with lower thresholds. Let $X_\theta$ be a metric such as

<footnote n="12">¹²https://github.com/cocodataset/cocoapi</footnote>
Precision, Recall or $F_1$-score at $\theta_J = \theta$, we defined:

$$ \text{WAvg}(X) := \sum_{\theta \in \mathcal{J}} \underbrace{\frac{\theta}{\sum_{\theta \in \mathcal{J}} \theta}}_{\text{Proportional weights.}} X_{\theta} \qquad (5) $$

The idea of proportional weights is intuitive (regarding the increasing severity)<fnref n="13" />. However, the choice of the threshold value set $\mathcal{J}$ is an arbitrary choice. Why specifically choose these four values?

It turns out that WAvg($X$) can be interpreted as a discrete approximation of a weighted average over a continuous range of values, and the weighted average as the expected value of the random variable $X_\theta$ according to a piecewise linear probability density function $f_s$, where $s$ a parameter: $f_s: x \mapsto \alpha_s x \mathbb{1}_{[s,1]}(x)$ with $\alpha_s > 0$ a normalization factor ($\int_{\mathbb{R}} f_s d\lambda = 1$).

Formally, let $\theta_J$ be the threshold real random variable with values in $[0, 1]$ (we will also use the range $[0.5, 1]$). $\theta_J$ follows the law $\theta_J \sim f_0(\theta) \propto \theta \mathbb{1}_{[0,1]}(\theta) = 2\theta \mathbb{1}_{[0,1]}(\theta)$ (and $f_{.5}(\theta) \propto \theta \mathbb{1}_{[0.5,1]}(\theta) = \frac{8}{3}\theta \mathbb{1}_{[0.5,1]}(\theta)$).

Let $p, g$ be respectively the number of predicted tables (the number of positives) and the number of ground truth tables. The assignment problem between prediction and ground truth is solved with the Hungarian algorithm: $\min(p, g)$ predictions are assigned to $\min(p, g)$ ground truths, and their Jaccard index score is denoted $J_k$. For $f_s = f_0$, we obtain the expected values of metrics under the assumption that $\theta_J$ is sampled according to $f_s$:

**Proposition 4.1.** Let $P_{\theta_J}, R_{\theta_J}, F_{1,\theta_J}$ be the random variable Precision, Recall and $F_1$-score obtained with the random variable $\theta_J$. Under the assumption that $\theta_J$ is sampled according to $f$, we have:

$$ \begin{aligned} \mathbb{E}_{\theta_J \sim f}[P_{\theta_J}] &= \frac{1}{p} \sum_{k=1}^{\min(p,g)} \mathbb{E}_{\theta_J \sim f}[\mathbb{1}_{[J_k > \theta_J]}] \\ \mathbb{E}_{\theta_J \sim f}[R_{\theta_J}] &= \frac{1}{g} \sum_{k=1}^{\min(p,g)} \mathbb{E}_{\theta_J \sim f}[\mathbb{1}_{[J_k > \theta_J]}] \end{aligned} \qquad (6) $$

$$ \mathbb{E}_{\theta_J \sim f}[F_{1,\theta_J}] = \frac{2}{p+g} \sum_{k=1}^{\min(p,g)} \mathbb{E}_{\theta_J \sim f}[\mathbb{1}_{[J_k > \theta_J]}] $$

See proof on page 44. We use these formulas with two probability density functions that are proportional to their input on $[0, 1]$ and $[0.5, 1]$.

**Proposition 4.2.** For probability density functions that are proportional to their input on $[0, 1]$ and $[0.5, 1]$, denoted $f_0$ and $f_{.5}$, we have:

$$ \begin{aligned} \mathbb{E}_{\theta_J \sim f_0}[\mathbb{1}_{[J_k > \theta_J]}] &= J_k^2 \\ \mathbb{E}_{\theta_J \sim f_{.5}}[\mathbb{1}_{[J_k > \theta_J]}] &= \frac{4}{3} \left(J_k^2 - \frac{1}{4}\right) \mathbb{1}_{[J_k > 0.5]}(J_k) \end{aligned} \qquad (7) $$

See proof on page 45. Thus, for each one of the following metric: Precision, Recall and $F_1$ we will compute them with a fixed Jaccard index threshold $\theta_J = 0.5$, but also these expected values, according a probability density function $f_0$ or $f_{.5}$.

**Average precision** In order to better compare models with confidence score and not choosing only one $\theta_c$, the metric Average Precision is useful for considering aggregation over $\theta_c$ threshold values. The TATR repository<fnref n="14" /> provides a built-in evaluation tool that allows computing the COCO metrics for the table detection task. The original COCO paper [23] introduced a large dataset for object detection, and evaluation code allowing to compute 12 metrics characterizing the performance of an object detector. More specifically, it includes Average Precision (AP) and Average Recall (AR) at multiple IoU threshold values (IoU threshold 0.50, 0.75 or a range) and across scales (objects areas).

<footnote n="13">¹³We could also discuss the choice of a linear difficulty scale.</footnote>

<footnote n="14">¹⁴https://github.com/microsoft/table-transformer</footnote>
The main metric used is $AP_\theta$, the average precision at IoU threshold $\theta$ that was used previously in the Pascal VOC [11] challenge, and became popular as an object detection metric.

There exist different definitions of $AP_\theta$. For models that output a confidence score $c_k$ on each prediction $k$ (that is, bounding box plus label), we can rank predictions in the decreasing order of their confidence score. Thus $(c_{(k)})_k$ is ordered, with $c_{(k)} \le c_{(k+1)}$ ($k \le n-1$, $n = |\mathcal{P}|$). Thus we can compute the Precision and Recall, denoted $P_{\theta_j}^{\theta_c}$ and $R_{\theta_j}^{\theta_c}$, at confidence score $\theta_c$ (positive predictions are elements from $\mathcal{P}_{\theta_c}^+$). Thus we obtain a list of pairs $((P_{\theta_j}^{c(k)}, R_{\theta_j}^{c(k)})_k$ that defines a Precision-Recall curve where the horizontal coordinate is given by $R$ and whose $y$ values form a “staircase” curve dictated by the values $P_{\theta_j}(R)$. More formally:

$$ \{(R, P_{\theta_j}(R)) \mid R \in \{R_{\theta_j}^{c(k)} \mid 1 \le k \le n\}\} \qquad (8) $$

1. Based on the Precision-Recall curve, $AP_\theta$ is defined as the Area Under the Curve (AUC):

$$ AP_{\theta}^{\text{AUC}} = \int_{0}^{1} P_{\theta}(R_{\theta}) dR_{\theta} \qquad (9) $$

In particular, in the Scikit-learn library [29], $AP_\theta$ is computed as:

$$ AP_{\theta}^{\text{AUC}} = \sum_{k=1}^{n} \tilde{P}_{\theta}^{c(k)} \Delta R_{\theta}^{c(k)} \qquad (10) $$

with $\Delta R_{\theta}^{c(k)} = R_{\theta}^{c(k-1)} - R_{\theta}^{c(k)}$, $n$ the total number of prediction and $c_{(0)} = 0$ and the precision function $\tilde{P}_{\theta}(r) = \max_{\tilde{r} \ge r} P_{\theta}(\tilde{r})$ chosen as to reduce the impact of “wiggles” in the Precision/Recall curve.

2. In the Pascal VOC Challenge [11], precision is averaged over a fixed set of equidistant recall levels $\mathcal{R} = \{0, 0.1, \dots, 1\}$.

$$ AP_{\theta}^{\text{VOC}} = \frac{1}{11} \sum_{r \in \mathcal{R}} \tilde{P}_{\theta}(r) \qquad (11) $$

Finally, as this definition of Average Precision $AP_{\theta}^{\text{VOC}}$ depends on a specific $\theta_J = \theta$, they also introduced an AP metric that is an average over multiple Intersection-over-Union (IoU) thresholds, denoted $AP_{\mathcal{J}}^{\text{VOC}}$, with $\mathcal{J} = \{0.5, 0.55, \dots, 0.95\}$:

$$ AP_{\mathcal{J}}^{\text{VOC}} = \frac{1}{9} \sum_{\theta_J \in \mathcal{J}} AP_{\theta_J}^{\text{VOC}} \qquad (12) $$

We note that the above metrics, as well as others currently supported in the library `pycocotools`, suffer from the following limitations:

* While evaluating the model, the evaluation module considers only images containing ground truth annotation. For the pages (images) containing no ground truth tables, there is no annotation file (using Pascal VOC XML annotation format) and then these images aren't loaded: we don't perform inference over these images because we already know that there is nothing to detect. Instead of considering the whole dataset for evaluation, only a subset of the dataset is used. By removing negative samples, in such pages (without ground truth), model predictions that would have accounted for FP aren't considered, which leads to an **overestimation of Precision**.

* As we said, Average Precision metric is based on the Precision-Recall curve, thus this metric considers all the predictions made by the model, regardless of their confidence score. However, we would also like to obtain access to Precision and Recall at a certain confidence threshold. For the Table Detection task (followed by table structure recognition), we set a threshold confidence score on the detected tables in order to have a usable set of boxes for TSR. AP is a ranking metric, hence even object with very low confidence may end up being True Positives and thus **boosting the AP score**.

To solve the abovementioned shortcomings, we computed $X_{\theta_j}^{\theta_c}$ aiming for more intuitive results, that have the following properties:
1. While evaluating the model, we consider all the images of the test dataset, including ones without ground truth annotation (also called negative examples).

2. We set a confidence score threshold $\theta_c$, useful for downstream tasks.

3. Either we set a IoU threshold $\theta_J$, or we compute Precision and Recall over a range of $\theta_J$ values: $\mathbb{E}_{\theta_J}[X]$

As a result, we will use AP as an aggregating score for probabilistic models, but $X^{\theta_c}$ will be also useful for downstream tasks whenever a fixed $\theta_c$ is needed.

**Confidence score calibration** Until now, we used metrics $(X_{\theta_J}^{\theta_c}, \mathbb{E}_{\theta_J}[X], AP)$ based on Precision and Recall that are based on binary choices: a prediction is either a positive either a negative, there is no in-between. However, we are losing interesting information for models that output confidence score: “How confident was the model that this bounding box contains a table?” That is why, we want to use another metric that, this time, is function of probabilities (confidence score), not just decisions (positive/negative). Actually, we aim to evaluate model miscalibration, as the model is seen as a classifier. A model is said **calibrated** when the probabilities (confidence scores) of the model are aligned with the real-world outcomes. A calibrated model is desirable because we might be more confident in its predictions (prediction includes confidence score).

Formally, a model is perfectly calibrated [14] when $\mathbb{P}[\hat{Y} = Y | \hat{P} = p] = p$, with $[\hat{Y} = Y]$ the event “the model prediction is right” (score of 1 when there is a table, 0 otherwise) and $[\hat{P} = p]$ the event “the confidence score (r.v.) is equal to $p \in [0, 1]$”. Here, the random variable $\hat{P}$ perfectly reflects how trustworthy the prediction is. Most research focused on measuring and resolving miscalibration for classification tasks. In this scope, the expected calibration error (ECE) [25] is commonly used in conjunction with the Brier score [3] and negative log likelihood (NLL) loss to measure miscalibration [17]. We chose to consider the well-established Brier score and the ECE.

**Brier score** The Brier score measures the accuracy of probabilistic predictions. In binary classification, accuracy is the proportion of correct predictions (both true positives and true negatives) among the total number of cases examined. Notice that we did not mention the number of TN because in object detection, there is an infinity of negative locations: countless potential bounding boxes exist, that do not contain any ground truth. Let $\mathcal{P}$ be a set of predictions and $c_k$ a confidence score that there is a table in the prediction $k$. Then, the Brier score $B$ is defined by:

$$B = \frac{1}{|\mathcal{P}|} \sum_{k \in \mathcal{P}} (c_k - o_k)^2 \quad (13)$$

* $k \in \mathcal{P}$ is an index of a prediction

* $c_k \in [0, 1]$ are confidence score from prediction $k$ (prediction in $\mathcal{P}$); the higher $c_k$, the more confident the model is about the fact that is a table in the bounding box $k$.

* $o_k \in \{0, 1\}$ is a binary value: either there is a ground truth (table) associated to the prediction $k$ or not.

In our case, which amounts to a binary classification problem (“table” or “no table”), $B$ is equivalent to $\ell^2$ distance $\frac{1}{p} \|c - o\|_2^2$, the mean square error. The lower $B \in [0, 1]$ is, the more accurate model predictions are.

We set $o_i = \delta_i \mathbb{1}_{[\text{Jaccard}(\hat{Y}_i, Y_i)>\theta_J]}$ where:

* $\delta_i = 1$ if we were able to associate prediction $i$ to a ground truth (during one-to-one matching), else $\delta_i = 0$

* If $\delta_i = 1$, we can compute the Jaccard index between the prediction $i$ and its ground truth associated, denoted $J_i$.

To sum up, for each prediction $i$ we look at how far the confidence score (probability to have a table according to the model) is from the ground truth:

* Either there is no GT table associated ($\delta_i = 0$);

* Or, there is a GT table associated ($\delta_i = 1$), but they do not overlap enough ($J_i < \theta_J$ so $o_i = 0$), or they are sufficiently similar ($o_i = 1$).
Note that usually accuracy is not the most suitable measure, especially for imbalanced classes. However, we will compute the Brier score not only on $\mathcal{P}$, but also on $\mathcal{P}_{\theta_c}$ sets<fnref n="15" /> in order to have, for downstream tasks, a set of positive samples $\mathcal{P}_{\theta_c}^+$, but also a measure of confidence score accuracy.

$$B^{\theta_c} = \frac{1}{|\mathcal{P}_{\theta_c}|} \sum_{k \in \mathcal{P}_{\theta_c}} (c_k - o_k)^2 \quad (14)$$

Note that for models without confidence score, such as GROBID and LLM-Vision, given that $\mathcal{P} = \mathcal{P}^+$ and $c_{\text{table}} = 1$, we obtain the result: $B = 1 - P$ due to the confidence of 1. Note that $B$ should be denoted $B_{\theta_J}$ because the vector $o \in \{0, 1\}^p$ depends on $\theta_J$ as Jaccard index threshold. Similarity to the previous metrics, we can then compute the expected value of $B_{\theta_J}$:

**Proposition 4.3.** Under the assumption that $\theta_J$ is sampled according to $f$, the excepted value of $B_{\theta_J}$ is given by:

$$\mathbb{E}_{\theta_J \sim f}[B_{\theta_J}] = \frac{1}{|\mathcal{P}|} \sum_{k \in \mathcal{P}} c_k^2 + (1 - 2c_k)\mathbb{E}[1_{[J_k > \theta_J]}] \quad (15)$$

See proof on page 45. Precision and Recall reflect a boolean decision of whether a prediction is correct or not; in contrast, the Brier score more generally considers probabilities.

**Expected Calibration Error (ECE)** The ECE is defined by:

$$ECE = \mathbb{E}_{\hat{P}} \left[ \left| \mathbb{P} \left[ \hat{Y} = Y \mid \hat{P} = p \right] - p \right| \right] \quad (16)$$

Equation 17 approximates ECE by partitioning predictions into $M$ equally-spaced bins and taking a weighted average of the bins' accuracy/confidence difference. Where $n$ is the number of predictions, $B_m$, the set of indices of predictions whose confidence falls between $\frac{m-1}{M}$ and $\frac{m}{M}$, acc and the conf functions compute respectively the average accuracy and confidence:

$$ECE \approx \sum_{m=1}^{M} \frac{|B_m|}{m} |\text{acc}(B_m) - \text{conf}(B_m)| \quad (17)$$

We compute this score over $\mathcal{P}$ to obtain an overall model calibration error.

**Summary** Table 3 summarizes the metrics explained above and which we will use in our evaluation. As a reminder, we denote:

* $X_{\theta_J}^{\theta_c}$, the metric $X$ (that can be Precision, Recall and $F_1$-score) computed with positive samples that have confidence scores greater than $\theta_c$. A positive sample is a TP if the Jaccard similarity (with its associated GT) is greater than $\theta_J$.

* $\mathbb{E}_{\theta_J}[X_{\theta_J}^{\theta_c}] = \int_{\mathbb{R}} X^{\theta_c}(\theta_J) f(\theta_J) d\theta_J$, the expected value of the metric $X$ (that can be Precision, Recall and $F_1$-score) computed with positive samples that have confidence scores greater than $\theta_c$ over a known probability density function $f$, with $X^{\theta_c}(\theta_J) = X_{\theta_J}^{\theta_c}$.

* $AP_{\theta}$, the Average Precision, that is the area under the curve of the Precision-Recall curve obtained with IoU threshold $\theta_J = \theta$.

* $A\mathbb{E}_{\theta_J}[P_{\theta}]$, the Average expected Precision: the area under the curve of the $\mathbb{E}_{\theta_J}[P_{\theta_J}] - \mathbb{E}_{\theta_J}[R_{\theta_J}]$ curve obtained.

* $B_{\theta_J}^{\theta_c}$, the Brier score over the subset of predictions $\mathcal{P}_{\theta_c}$. For samples that have Jaccard similarity scores (with their associated GT) greater than $\theta_J$, we consider that their target are tables.

* $\mathbb{E}_{\theta_J}[B_{\theta_J}^{\theta_c}] = \int_{\mathbb{R}} B_{\theta_J}^{\theta_c}(\theta_J) f(\theta_J) d\theta_J$, the expected Brier score.

Those metrics take as input either a set of positive prediction $\mathcal{P}^+$, or a set of prediction with their confidence scores, $\mathcal{P}$. $\mathcal{P}^+$ is defined with the score threshold $\theta_c$: $\mathcal{P}_{\theta_c}^+ = \{\hat{y} | (\hat{y}, c) \in \mathcal{P}, c > \theta_c\}$, whereas $\mathcal{P}_{\theta_c}$ is just the restriction of prediction by their confidence scores (samples are neither positive nor negative, we retain their scores): $\mathcal{P}_{\theta_c} = \{(\hat{y}, c) \in \mathcal{P} | c > \theta_c\}$. I table 3, the Jaccard threshold $\theta_J$ is either “Fixed” at 0.5, or “All”: we consider all possible values through the excepted-based metrics

<footnote n="15">¹⁵We do not talk about $\mathcal{P}^+$ set here: there is no decision function positive/negative.</footnote>
($\theta_J \in [0, 1]$ or $[.5, 1]$). Finally, the confidence threshold score is either “Fixed” at $\theta_c$, or “All”: we consider all possible values through area under the $P^{\theta_c} - R^{\theta_c}$ curves. For metrics $Y^{\theta_c}$ are computed with fixed $\theta_c$, we will plot $(Y^{\theta_c})_{\theta_c}$, as a function of $\theta_c$ in the section 5.2.

<table><caption>Table 3: Metric used in table detection evaluation overview.</caption><thead><tr><th>Metric</th><th>Input</th><th>θ<sub>J</sub></th><th>θ<sub>c</sub></th><th>Results (Section 5.2)</th></tr></thead><tbody><tr><td>X<sup>θ<sub>c</sub></sup><sub>θ<sub>J</sub></sub></td><td>P<sup>+</sup><sub>θ<sub>c</sub></sub></td><td>Fixed</td><td>Fixed</td><td rowspan="2">Fig. 15, 18 Tab. 6, 8</td></tr><tr><td>E<sub>θ<sub>J</sub></sub>[X<sup>θ<sub>c</sub></sup><sub>θ<sub>J</sub></sub>]</td><td>P<sup>+</sup><sub>θ<sub>c</sub></sub></td><td>All</td><td>Fixed</td></tr><tr><td>AP<sub>θ</sub></td><td>(P<sup>+</sup><sub>θ<sub>c</sub></sub>)<sub>θ<sub>c</sub></sub></td><td>Fixed</td><td>All</td><td rowspan="2">Tab. 7</td></tr><tr><td>A E<sub>θ<sub>J</sub></sub>[P<sub>θ</sub>]</td><td>(P<sup>+</sup><sub>θ<sub>c</sub></sub>)<sub>θ<sub>c</sub></sub></td><td>All</td><td>All</td></tr><tr><td>B<sup>θ<sub>c</sub></sup><sub>θ<sub>J</sub></sub></td><td>P<sub>θ<sub>c</sub></sub></td><td>Fixed</td><td>Fixed</td><td rowspan="2">Fig. 17</td></tr><tr><td>E<sub>θ<sub>J</sub></sub>[B<sup>θ<sub>c</sub></sup><sub>θ<sub>J</sub></sub>]</td><td>P<sub>θ<sub>c</sub></sub></td><td>All</td><td>Fixed</td></tr></tbody></table>

### 4.1.2 Table detection evaluation based on HTML mark-up (HTML-based TD)

Some methods do not recognize a table as a bounding box, but return it as a structured HTML mark-up table. This is the case for the model LLM-Vision (Section 3.2). We need to measure the performance of such models on table detection using only the predicted and ground truth tables HTML mark-up (as it is the only information that we have). We use the previous metrics (Section 4.1.1), except that the distance function (IoU for bounding boxes) has to change. A simple way to do this is to compare the tokens (strings) in the prediction, respectively, in the ground truth, via an n-gram (tuple of n tokens), a technique commonly used in natural language processing.

We considered approaches like ROUGE [21] or BLEU [27], which are respectively mainly used for summary and translation evaluation (Recall and Precision-based metrics), in order to obtain similarity score. Notice that BLEU-4 was used as Table Structure Recognition metric [20] (see Section 4.2). However, these metrics focus on only one aspect (Precision or Recall), and don't totally measure the similarity. Instead, inspired by the Intersection-over-Union with bounding boxes, we decided to use the Jaccard similarity between multisets of n-grams from the prediction and the ground truth, respectively denoted $S_P$ and $S_{GT}$.

$$ \text{Jaccard}(S_P, S_{GT}) = \frac{|S_P \cap \text{multi } S_{GT}|}{|S_P \cup \text{multi } S_{GT}|} \qquad (18) $$

where $| \cdot |$ denotes the cardinality of a multiset.

To adopt this, we need to choose a value $n$, a definition of tokens definition, and rely on multisets instead of regular sets. Here are the reasons behind our choices:

* $n = 2$-gram. We chose $n = 2$, so the elements of 2-gram set are tuple containing 2 tokens. The metric is stricter for $n = 2$ than $n = 1$. $S_P = \{(t_1, t_2), (t_2, t_3), (t_3, t_4), \dots\}$, where $t_i$ denotes a token.

* To derive n-grams, HTML tags from the table structure are not considered as tokens, because our measure needs to reflect how much cell *content* are in common between prediction and ground truth. Thus, our tokens will only be derived from cell contents. Further, we did not use words as tokens, because we noticed that the LLM-based model introduced character-level errors, which degraded our word-based metrics drastically (see examples at Appendix C). That is why we chose a smaller token size: 2-characters (a token is a sequence of 2 characters).

* The Jaccard distance is computed over multisets: we use 2-grams multisets. We choose to use multisets because it allows elements repetition: table cells content are not necessarily unique and might contain duplicates. For instance, if a table is filled with "0." that is a 2-character string, it makes sense to consider $S_P$ as a 2-grams multiset with repetitions $\{(\text{"0."}, \text{"0."}), (\text{"0."}, \text{"0."}), \dots\}$ rather than regular set with a single tuple: $\{(\text{"0."}, \text{"0."})\}$.

From now on the “Jaccard distance over 2-gram-2-character multiset” is denoted HTML Jaccard. For example, assume the predicted HTML mark-up is:

```html
<table><tr><td>Location</td><td>Time</td><td>Times</td></tr></table>
```
This leads to:

$$S_p = \{ ("Lo", "ca"), ("ca", "ti"), ("ti", "on"), ("on", "Ti"), ("Ti", "me"), ("me", "Ti"), ("Ti", "me"), \dots \}$$

Accordingly, we adapt Algorithm 1 to replace coordinate-based computations, in particular the $l^1$ distance, with the above Jaccard score for the cost matrix, as follows.

* $\hat{\mathbf{Y}}_i$ are now lists of tokens for each prediction.

* We compute $\mathbf{C} = (1 - \text{Jaccard}(S_P^i, S_{GT}^j))_{i,j} \in \mathbb{R}^{p \times g}$.

* As above, we solve an assignment problem, minimizing $\mathbf{C}$ to obtain pairs (prediction, target), using the Hungarian algorithm.

* We replace the bounding box-based Jaccard index (IoU) by this HTML-based Jaccard index (HTML Jaccard), to distinguish, given a positive prediction, whether it is a TP or an FP.

This naive approach is a good approximation for TD quality evaluation. Thus, we can compute $X_{\theta_j}^{\theta_c}$ that is Precision, Recall and $F_1$-score, over $P_{\theta_j}^+$ with fixed thresholds $\theta_j$ and $E_{\theta_j}[X_{\theta_j}^{\theta_c}]$. We did not compute metrics using confidence scores, such as AP and Brier, because LLM-Vision does not provide it. Instead, we compute these metrics with bounding boxes, which is a natural choice. Experimentally, we observed that HTML-based metrics (especially $X_{.5}$) were highly correlated to the bounding box-based ones (see Section 5.6).

### 4.1.3 Evaluation pipeline

For bbox-based TD evaluation, we run detection models (GROBID, VGT, (XY+)TATR-detect) on pages, and compare predictions with annotation bounding box ground truths. For HTML-based TD evaluation, we run end-to-end models (GROBID, LLM-Vision, VGT+TATR-structure, (XY+)TATR-extract) on pages, and compare predictions with the HTML ground truth.

We used the Label Image<fnref n="16" /> popular application to annotate bounding boxes ourselves. It allows to generate annotation files for every image containing tables, through a graphic interface. For each image, we select as many bounding boxes as there are tables (or rotated tables) in the image. We can select the output format among PascalVOC or YOLO [30]. We chose the PascalVOC format since it is the one used in the TATR pipeline.

## 4.2 Evaluating Table Structure Recognition

Here, we are concerned with the evaluation of the quality of results in the second task. Specifically, we check that the predicted table has the correct table shape (topology, structure), and that cells have the expected content. Extracted tables can be defined through table's cell bounding boxes coordinates with their content, or with a more structured format. For example, considering the way tables are described in HTML, a table can be seen as a tree having rows as children of the table, and the cells as children of each row (and table's leaves).

A naive approach for TSR evaluation could be to compare predicted cells with GT cells by their absolute positions, but this approach suffers from the shift problem: a slight shift along one or both coordinates would disproportionately penalize the models. Usually authors compute Average Precision for TSR: as an object detection problem with classes: {"row", "column", ...} (as TATR-structure, Section 3.3.3). Essential table information is in its relative structure/position, rather than absolute coordinates position. Therefore, we prefer **metrics that rely on the structure**, not on absolute coordinates.

The 4-gram BLEU score (BLEU-4) metric can be applied by comparing HTML tables as strings. It allows to compare at the same time mark-up structure (tags) and content between predictions and ground truth. However, BLEU-4 evaluates exact (not partial) match cell correctness and does not catch table spatial representation as they are reduced to sequence of HTML tokens. The DAR metric [6], another metric that does not rely on absolute coordinates, is based on relative cell positions (which cells are above/below/left/right of a given cell), but it does not capture the global 2D structure. We will rather use GriTS [31] metrics, which evaluate tables directly in their matrix form and create a similarity between matrices, and TEDS [36] that computes similarity between tables seen as trees of HTML tags. We describe below in more details these two metrics for TSR, that satisfies our requirements.

<footnote n="16">¹⁶https://github.com/HumanSignal/labelImg</footnote>
### 4.2.1 TEDS

The metric Tree-Edit-Distance-Based Similarity (TEDS, in short) measures the similarity of tables viewed as trees. Specifically, each HTML table can be described in a standardized way as follows.

* There is a root `<table>` with children `<tr>` (nodes are table rows).

* The leaves of the tree are cells `<td>` with attributes “colspan”, “rowspan” and “content”.

* Alternatively, we could also describe tables with children `<thead>`, `<tbody>`. This description depends on the ability of the model to perform *function analysis*, i.e., to determine if a cell is a header or not.

TEDS builds upon the previous Tree-Edit Distance [28]. This metric is inspired by the Levenshtein Distance, which is a string metric for measuring the difference between two sequences [18]. It counts the minimum number of insertions, deletions and substitutions to transform a string into another string (minimum number of single-character edits). This problem is usually solved with Dynamic Programming. TEDS can be seen as generalizing string edit distance: instead of counting characters to be deleted, inserted or replaced, it counts the node deletions, insertions, relabeling and content modification operations needed to transform a tree (HTML table) into another.

* The cost to delete or insert a node is 1.

* The relabeling of a structure node also costs 1.

* The relabeling of a cell content incurs a substitution cost that is the normalized Levenshtein similarity between the two strings.

Putting it all together, the TEDS score is defined by Equation (19):

$$ TEDS(T_P, T_{GT}) = 1 - \frac{\text{EditDist}(T_P, T_{GT})}{\max(|T_P|, |T_{GT}|)} \quad (19) $$

where $| \cdot |$ is the number of nodes. TEDS takes into account the tag structure of the tree but also the tag names, thus it performs Function Analysis: the type of the cells (header or not) matter.

### 4.2.2 GriTS

Grid Table Similarity is a family of metrics for Table Structure Recognition using the natural form of a table, namely a matrix. GriTS distinguishes three different levels at which table structure can be identified (understood): Topology, describing the rows and columns each cell spans over in a two-dimensional grid; Content, which refers to the textual content within each cell; and Location, denoting the rectangular span in a pixel matrix that is occupied by each cell. Accordingly, there are three GriTS metrics, for Topology, Content, and Location respectively; each of them can be seen as a function to call over two matrices, the prediction and the ground truth.

Each metric computes matrix similarity based on their 2-dimensional Longest Common Subsequence (2D-LCS, in short), whose computation is NP-Hard [1]. Entries from input matrices depend on the subtasks that we are facing. There are three steps while calculating similarity for GriTS metric:

1. According to the subtask choice (Content, Topology, Location), we generate A and B matrices with entries as shown in Table 4; these entries represent the input tables that we want to compare.

2. We compute two-dimensional most similar substructures (denoted $\tilde{A}$, $\tilde{B}$) between two matrices, specific to the subtask.

3. We compute similarity score between $\tilde{A}$ and $\tilde{B}$, again depending on the subtask.

<table><caption>Table 4: GriTS metrics characteristics.</caption><thead><tr><th>Type</th><th>Input</th><th>Matrix entries</th><th>Entries space E</th><th>Function f</th></tr></thead><tbody><tr><td>Content</td><td>Cells layout + text content</td><td>Cell content (string)</td><td>V<sup>N</sup></td><td>LCS</td></tr><tr><td>Topology</td><td>Cells layout</td><td>Relative cell position</td><td>Z<sup>4</sup></td><td>IoU</td></tr><tr><td>Location</td><td>Cells layout + absolute coordinates</td><td>Bounding boxes location</td><td>R<sup>4</sup></td><td>IoU</td></tr></tbody></table>
The entry-wise penalty function $f: E \times E \rightarrow [0, 1]$ in the rightmost column is used to measure the so-called partial correctness between two entries with the same coordinates. It replaces the 0 or 1 value used for the vanilla 2D-LCS computation.

<figure><img src="image_16.png" /><figcaption>(a) An example presentation table from the PubTables-1M dataset.</figcaption></figure>

<table><caption>(b) GriTS<sub>Con</sub></caption><thead><tr><th>Group</th><th>Sequence of Administration</th><th>Sequence of Administration</th><th>Sequence of Administration</th></tr></thead><tbody><tr><td>Group</td><td>Phase I</td><td>Phase II</td><td>Phase III</td></tr><tr><td>I</td><td>C</td><td>A</td><td>B</td></tr><tr><td>II</td><td>B</td><td>C</td><td>A</td></tr><tr><td>III</td><td>A</td><td>B</td><td>C</td></tr></tbody></table>

<table><caption>(c) GriTS<sub>Top</sub></caption><thead><tr><th>[0, 0, 1, 2]</th><th>[0, 0, 3, 1]</th><th>[-1, 0, 2, 1]</th><th>[-2, 0, 1, 1]</th></tr></thead><tbody><tr><td>[0, -1, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td></tr><tr><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td></tr><tr><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td></tr><tr><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td><td>[0, 0, 1, 1]</td></tr></tbody></table>

<table><caption>(d) GriTS<sub>Loc</sub></caption><thead><tr><th>[136.42, 477.25, 160.62, 501.45]</th><th>[185, 477.25, 470.89, 487.22]</th><th>[185, 477.25, 470.89, 487.22]</th><th>[185, 477.25, 470.89, 487.22]</th></tr></thead><tbody><tr><td>[136.42, 477.25, 160.62, 501.45]</td><td>[185, 491.48, 271.9, 501.45]</td><td>[284.5, 491.48, 371.39, 501.45]</td><td>[384, 491.48, 470.89, 501.45]</td></tr><tr><td>[136.42, 505.82, 160.62, 515.72]</td><td>[185, 505.82, 271.9, 515.72]</td><td>[284.5, 505.82, 371.39, 515.72]</td><td>[384, 505.82, 470.89, 515.72]</td></tr><tr><td>[136.42, 515.73, 160.62, 525.63]</td><td>[185, 515.73, 271.9, 525.63]</td><td>[284.5, 515.73, 371.39, 525.63]</td><td>[384, 515.73, 470.89, 525.63]</td></tr><tr><td>[136.42, 525.64, 160.62, 535.53]</td><td>[185, 525.64, 271.9, 535.53]</td><td>[284.5, 525.64, 371.39, 535.53]</td><td>[384, 525.64, 470.89, 535.53]</td></tr></tbody></table>

Figure 12: An example presentation table from the PubTables-1M dataset, from [31], along with corresponding ground truth grid cell matrices for different GriTS metrics (Table 4). Each matrix entry corresponds to one grid cell. Cells obtained from originally fused (merged) HTML cells are shaded darker.

Figure 12 presents an example: one table from a PDF file, a prediction, and the corresponding matrices generated for the Content, Topology and Location scores. Entries in each matrix follow the types indicated in Table 4: $V^N$ (strings) for Content, $Z^4$ for Topology<fnref n="17" /> and $\mathbb{R}^4$ for the Location (bounding box corner coordinates).

The set of substructures of a matrix $M$ with rows $R = [r_1, \dots, r_n]$ and columns $C = [c_1, \dots, c_m]$ is defined as the set of matrices $M'$ that have rows $R'$ and columns $C'$ respectively included in $R$ and $C$. The set of substructures of $M$ is denoted $\mathcal{M}$; its cardinality is $2^{n+m}$. Then, the 2D-Longest Common Subsequence (or 2D-MSS) takes as input two matrices $A, B$ and outputs two substructure matrices $\tilde{A}, \tilde{B}$ (of the same shape) that are the most similar according to the entry-wise penalty function $f$. To avoid actually computing 2D-LCS, [31] proposed a **heuristic** approach to determine the 2D-MSS by factoring the problem (using DP), **considering the optimal subsequence of rows and columns independently**. This reduces the complexity from $2^{n+m}$ to $2^n + 2^m$. Thus, the process is:

1. Generate $(A, B) \in (E^{n_A \times m_A}, E^{n_B \times m_B})$, according to the subtask choice (Content, Topology, Location).

2. Identify the 2D most similar substructures $\tilde{A}, \tilde{B} = 2\text{D-MSS}_f(A, B) = \arg\max_{A \times B} \sum_{i,j} f(A'_{i,j}, B'_{i,j})$, knowing the subtask choice $f$.

3. Compute similarity score $\text{GriTS}_f(A, B)$, knowing the subtask choice $f$.

The GriTS score, for a subtask $f$, is defined as:

$$ \text{GriTS}_f(A, B) = 2 \frac{\sum_{i,j} f(\tilde{A}_{i,j}, \tilde{B}_{i,j})}{|A| + |B|} \qquad (20) $$

<footnote n="17">¹⁷The GriTS encoding uses positive, 0 and negative integers for this encoding; see [31] for the details.</footnote>
where $| \cdot |$ is the matrix size (number of grid cells), $f$ the similarity function between the grid cells' properties ($0 \le f \le 1$), and $\tilde{A}, \tilde{B}$ are the most similar two-dimensional substructures of A and B: $\tilde{A}, \tilde{B} = 2\text{D-MSS}_f(A, B)$. GriTS$_f$ can be interpreted as an $F_1$-score coming from GriTS-Rec$_f$ and GriTS-Prec$_f$ (taking A and B to be the prediction, respectively, the ground truth).

**GriTS metrics in our context** To compute GriTS metrics for our problem, two methods can be envisioned, depending on whether we represent a table (and each of its cells) based on bounding boxes, or HTML mark-up:

* Using the bounding box location of every single row and column requires a significant annotation effort.

* Using the HTML mark-up output would be tedious to create by hand, but we can start from a model's output which can then be manually corrected to obtain a good-quality GT. In particular, we use the HTML output of the TATR model (Section 3.3), which we then manually review and improve.

Thus, we used GriTS Content and Topology, and not GriTS Location (since we chose not to rely on bounding boxes for each table element).

**HTML mark-up vocabulary to be used in our evaluation** One last aspect needs to be settled with respect to the metrics based on HTML mark-up (TEDS and GriTS). Different models used slightly different sets of HTML tags to represent the tables they understood. For instance, TATR-structure uses additional tags such as `<thead>`, `<tbody>`, and also predicts whether a cell is a header or not (`<td>`, `<th>` tags), while GROBID does not. To avoid being penalized by such inconsistencies, we normalize each output preserving only three tags: `<table>`, `<tr>`, `<td>`. From now on, we implicitly consider that each metric applies over such normalized outputs (and normalized GT) only.

### 4.2.3 Evaluation pipeline

End-to-end models are evaluated according to TD and TSR subtasks. For TD, we evaluate either with decision-based metrics (input in $P^+$), or with probability-based metrics (input in $P^c$). In both cases, we use the same pipeline (Figure 13) for evaluation (except that we keep confidence score), with $\theta_c = 0$ to get $P^c$ (for probability-based metric).

Figure 14 shows how positive predictions are matched with the ground truth to create the set $P^{++}$. For TSR, we evaluate tables extracted (HTML mark-up tables) with their associated ground truth tables. The choice of the set containing tabled extracted matters for TSR evaluation:

* We could consider $P^+$, that are tables considered as positives. For each positive sample, we try to associate a ground truth, but we could face cases where the sample is actually not a table (there are more predictions than GT): there is no GT to associate with.

* We could consider $P^{++}$, that are tables considered as True Positives. With this set, we have pairs of extracted tables and GT. Actually, we use $P^{++} = P_{\theta_J}^+$, with threshold $\theta_J$ that distinguishes TP of FP according to bbox or HTML-based Jaccard similarity score.

Hence, we used $P^{++}$ set for TSR evaluation.
<figure><img src="image_17.png" /><figcaption>Figure 13: Simple evaluation pipeline. We run end-to-end model inference on page X, and obtain: the predicted tables set $\mathcal{P}$ that contains HTML mark-up tables, table confidence scores (equal to 1 for non probabilistic models), and bounding boxes in some cases. X is annotated with ground truth tables Y. We filter out predictions by the confidence threshold $\theta_c$ to get $\mathcal{P}^+$. Finally, $\hat{Y}$ represents end-to-end positive predictions, containing bounding boxes or HTML mark-up tables, depending on the TD based metric.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Figure 14: Details on the “Evaluation” module from Figure 13. Inputs are $(\hat{Y}, Y)$. We compute pairwise distance with the matrix $C$ based on a distance function, $d_B$ denotes the distance function for bounding box-based TD and $d_H$ for HTML-based. With the Hungarian algorithm, we solve the assignment problem: in the example shown, $p > g$ (three predictions, two ground truth tables), thus there are *g* matching pairs $(\sigma(j), j)$ with $\sigma: \llbracket 1, \min(p, g) \rrbracket \to \llbracket 1, \max(p, g) \rrbracket$ creating matches. For matching pairs, depending on the Jaccard similarity, the sample is classified as FP or TP. For TP, we can compute TSR metric with the GT found. Predictions without GT are considered FP, and a GT without prediction leads to a FN.</figcaption></figure>
# 5 Results and analysis

## 5.1 Test dataset

The test dataset is composed of 499 pages from BRGM<fnref n="18" /> PDF reports, manually annotated for table detection (bounding boxes, see Section 4.1.3) and table structure recognition (manual review of TATR-extract inference, see Section 4.2.3) tasks. For more characteristics see Appendix A. We evaluate models performance on TD and TSR tasks. As discussed earlier (Section 4), we have two evaluation methods, bounding-box based or HTML-based.

* **BBox-based** The set of positive tables $\mathcal{P}^{B+}$ is defined through bounding box coordinates. In this case, we can compare GROBID, TATR-extract, XY+TATR-extract and VGT+TATR-structure. On each page, for each ground truth table we assign a predicted table by using bounding box coordinates. More specifically, we use a cost matrix containing $\ell^1$ pairwise distances between predictions and ground truths, then the Hangarian algorithm solves the assignment problem. Thus we can count the number of TP, FP and FN. Then, we can use the set of True Positive tables $\mathcal{P}^{B++}$, to compute TSR metrics. That is why the number of samples on which TSR metrics are computed depends on the TD evaluation, that determines the set of TP, $\mathcal{P}^{B++}$.

* **HTML-based** The set of positive tables $\mathcal{P}^{H+}$ is defined through HTML mark-up. In this case, we can compare GROBID, LLM-Vision, TATR-extract, XY+TATR-extract and VGT+TATR-structure. On each page, for each ground truth table we assign a predicted table by using HTML mark-up tables. More specifically, we use a cost matrix containing pairwise HTML-Jaccard distance between predictions and ground truths, then the Hangarian algorithm solves the assignment problem. Thus we can count the number of TP, FP and FN. Then, we can use the set of True Positive tables $\mathcal{P}^{H++}$, to compute TSR metrics. That is why the number of samples on which TSR metrics are computed depends on the TD evaluation, that determines the set of TP, $\mathcal{P}^{H++}$.

Table 5 summarizes results (reference) according to task (TD, TSR) and method (Bbox, HTML-based).

<table><caption>Table 5: Methods used for subtask evaluation.</caption><thead><tr><td>Evaluation</td><td>TD</td><td>TSR</td></tr></thead><tbody><tr><td>Bounding box-based</td><td>Tab. 6, Fig 15</td><td>Tab. 9, Fig 19</td></tr><tr><td>HTML-based</td><td>Tab. 8, Fig 18</td><td>Tab. 10, Fig 20</td></tr></tbody></table>

## 5.2 Table Detection

We evaluate table detection performance with the following metrics: Precision (P), Recall (R) and $F_1$-score. Let X be a metric, we denote:

* $X_{.5}$ the metric X with Jaccard index threshold at $\theta_J = 0.5$ (either IoU threshold, for bbox-based metric, or HTML-Jaccard index, for HTML-based metric);

* $\mathbb{E}_0[X]$ as shorthand for $\mathbb{E}_{\theta_J \sim f_0(\theta_J)}[X]$, where $f_0: \theta \mapsto 2\theta\mathbb{1}_{[0,1]}(\theta)$. It is the expected score X (as a random variable) under the probability density function $f_0$ that weighted the probability of outcomes.

* $\mathbb{E}_{.5}[X]$ as shorthand for $\mathbb{E}_{\theta_J \sim f_{.5}(\theta_J)}[X]$, where $f_{.5}: \theta \mapsto \frac{8}{3}\theta\mathbb{1}_{[0.5,1]}(\theta)$. It is the expected score X (as a random variable) under the probability density function $f_{.5}$ that weighted the probability of outcomes.

In tables 6, 8, we copy the score values for arbitrary confidence score thresholds: 5%, 50% and 95%. They are not best (not $\arg\max_{\theta_c} F_{1,\theta_c}$) but they do show trends. Note that baselines do not depend on threshold to define positives because in their case $\mathcal{P}^+ = \mathcal{P}$.

### 5.2.1 Bounding box-based TD evaluation

The baseline model GROBID is largely outperformed by object detection approaches regarding Precision and Recall scores (IoU at 50 % and excepted values). During evaluation, we encountered problems in

<footnote n="18">¹⁸https://infoterre.brgm.fr/rechercher/</footnote>
Table detection evaluation based on bounding box

<figure><img src="image_19.png" /><figcaption>Figure 15: Results for bounding box-based table detection evaluation according to confidence score threshold $\theta_c$ that defines $P_{\theta_c}^+$. For each metric $X$, there are 3 sub-metrics, $X_{.5}$ for IoU threshold at 50 %, and $E_0[X]$, $E_{.5}[X]$ for expected value over IoU distribution. Models are distinguished by their colors and metric types by their line style.</figcaption></figure>
<table><caption>Table 6: Results for bounding box-based table detection evaluation. For each metric X, there are 3 sub-metrics, X.5 for IoU threshold at 50 %, and E₀[X], E.5[X] for expected value over IoU distribution. For probabilistic models, we use 3 arbitrary confidence score thresholds to define P<sub>θc</sub><sup>+</sup>.</caption><thead><tr><th colspan="2">Model</th><th colspan="3">Precision</th><th colspan="3">Recall</th><th colspan="3">F<sub>1</sub></th></tr><tr><th>Name</th><th>θ<sub>c</sub></th><th>P<sub>.5</sub></th><th>E<sub>0</sub>[P]</th><th>E<sub>.5</sub>[P]</th><th>R<sub>.5</sub></th><th>E<sub>0</sub>[R]</th><th>E<sub>.5</sub>[R]</th><th>F<sub>1</sub><sub>.5</sub></th><th>E<sub>0</sub>[F<sub>1</sub>]</th><th>E<sub>.5</sub>[F<sub>1</sub>]</th></tr></thead><tbody><tr><td>GROBID</td><td></td><td>0.23</td><td>0.16</td><td>0.12</td><td>0.12</td><td>0.09</td><td>0.06</td><td>0.16</td><td>0.11</td><td>0.08</td></tr><tr><td>LLM-Vision</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="3">VGT</td><td>0.05</td><td>0.45</td><td>0.41</td><td>0.40</td><td><strong>0.98</strong></td><td><strong>0.89</strong></td><td><strong>0.86</strong></td><td>0.61</td><td>0.56</td><td>0.54</td></tr><tr><td>0.50</td><td>0.91</td><td>0.83</td><td>0.81</td><td>0.66</td><td>0.60</td><td>0.59</td><td>0.76</td><td><strong>0.70</strong></td><td><strong>0.68</strong></td></tr><tr><td>0.95</td><td>0.91</td><td><strong>0.88</strong></td><td><strong>0.87</strong></td><td>0.16</td><td>0.16</td><td>0.16</td><td>0.28</td><td>0.27</td><td>0.26</td></tr><tr><td rowspan="3">TATR-d</td><td>0.05</td><td>0.17</td><td>0.13</td><td>0.11</td><td>0.97</td><td>0.70</td><td>0.62</td><td>0.29</td><td>0.21</td><td>0.19</td></tr><tr><td>0.50</td><td>0.27</td><td>0.20</td><td>0.17</td><td>0.89</td><td>0.65</td><td>0.57</td><td>0.41</td><td>0.30</td><td>0.27</td></tr><tr><td>0.95</td><td><strong>0.97</strong></td><td>0.74</td><td>0.66</td><td>0.76</td><td>0.58</td><td>0.51</td><td>0.85</td><td>0.65</td><td>0.58</td></tr><tr><td rowspan="3">XY+TATR-d</td><td>0.05</td><td>0.07</td><td>0.05</td><td>0.04</td><td>0.97</td><td>0.64</td><td>0.53</td><td>0.13</td><td>0.08</td><td>0.07</td></tr><tr><td>0.50</td><td>0.15</td><td>0.10</td><td>0.08</td><td>0.94</td><td>0.63</td><td>0.53</td><td>0.26</td><td>0.17</td><td>0.15</td></tr><tr><td>0.95</td><td><strong>0.97</strong></td><td>0.65</td><td>0.54</td><td>0.92</td><td>0.62</td><td>0.51</td><td><strong>0.94</strong></td><td>0.63</td><td>0.53</td></tr></tbody></table>

table page location: GROBID managed to extract tables but it positioned them at the wrong page (page location is used for matching with ground truth tables during evaluation). We have tried another pipeline: we split multi-pages PDF file into single-page PDF files. That way, we were sure about tables extracted location. However, this attempt did not have a significant impact; worse, some tables were not detected anymore. This can be explained by the GROBID training: the model expects as a first page a front cover with titles, authors, etc. not a regular report page. Finally, we kept the original pipeline (Figure 1) with the whole PDF as input.

Table 6 and Figure 15 underline the fact that the threshold choice θ<sub>c</sub>, that determines P<sub>θc</sub><sup>+</sup> has different impact on VGT and TATR-detect. TATR-detect-based models increase their Precisions for very high threshold, contrary to VGT that have better trade-off with for low confidence score threshold. TATR with XY achieves higher Precision for high θ<sub>c</sub>. XY+TATR-detect Precisions are lower than TATR-detect (for low threshold) because they are more predictions with XY-cut (without the 2-top post-processing, see Section 3.5.2, there would be even more). We can find trade-off on this validation set (we don't have test set to properly chose best thresholds) for high F<sub>1</sub>-score: around 30% for VGT and 95% for TATR-detect-based models. TATR-detect is specialized in TD, whereas VGT also classifies other layout document elements. Non informative confidence score (uniform scores on classes), for these two models, are 0.66<sup>19</sup> and 0.17 respectively, which explains the differing behavior in relation to thresholds. We also notice that input image resolution have a significant impact, for example VGT requires lower resolution (according to VGT repository that includes pre-processing pipeline) than TATR-detect does. We tried to use the same image resolution for both models, but the results were not as good. Also, due to pre-TATR-structure processing, that removes detected tables that contain no tokens, all TATR-structure based model saw their recall ceiled by non-empty tables that are actually images which do not contain tokens.

Expected metrics used (E₀[X], E.5[X]) follow the X.5 trend, those metrics are similar, but different: there are stricter (lower scores) and they also add information about IoU. For instance, we notice that expected Recalls are closer to Recall at IoU 50 % for VGT than for TATR-detect-based models. Since E[R] = 1/9 Σₖ Jₖ², this means that VGT predicts bounding boxes that are closer (pixel-wise) to the GT annotations used. As the number of GT, g, is constant, Σₖ Jₖ² is higher for VGT, that underlines the fact that VGT delimits a table right up to its borders, as we did during annotation, contrary to TATR-detect that stops to the written elements. So for higher IoU threshold, some TATR-detect predictions would have become FP (and this is measured by E[R]).

For probabilistic models, we plot the Precision-Recall curve (Figure 16) for precise range of value θ<sub>c</sub> ∈ {0, 0.25, 0.5, 0.1, 0.15, ..., 0.9, 0.95, 0.975, 1}<sup>20</sup> at IoU threshold 50 % and expected Precision-Recall

<footnote n="19"><sup>19</sup>TATR-detect classifies into 3 classes, but for evaluation we consider “rotated table” as “table”.</footnote>

<footnote n="20"><sup>20</sup>This is an approximation, we should have used all (c<sub>(k)</sub>)<sub>k</sub>.</footnote>
<figure><img src="image_20.png" /><figcaption>Figure 16: Precision-Recall curves for TATR-detect, VGT and XY+TATR-detect, depending on IoU (50%, expected precision).</figcaption></figure>

<table><caption>Table 7: Average Precision for probabilistic models computed as area under the curves from Figure 16.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">AP</th></tr><tr><th>AP<sub>.5</sub></th><th>A<sup>E</sup><sub>0</sub>[P]</th><th>A<sup>E</sup><sub>.5</sub>[P]</th></tr></thead><tbody><tr><td>VGT</td><td>0.84</td><td><strong>0.72</strong></td><td><strong>0.68</strong></td></tr><tr><td>TATR-detect</td><td>0.79</td><td>0.45</td><td>0.36</td></tr><tr><td>XY+TATR-detect</td><td><strong>0.90</strong></td><td>0.40</td><td>0.28</td></tr></tbody></table>
curves. The higher you are on the right, the better the $F_1$-score is. Best performance are achieved by XY-TATR-detect with IoU threshold 50 %, but it is VGT for expected curves. Finally, we compute Average Precision scores with $P_{.5}$, $E_0[P]$, $E_{.5}[P]$, that is area under the PR-curve (Table 7).

<figure><img src="image_21.png" /><figcaption>Figure 17: Brier scores ↓ computed over $P_{\theta_c}$, for each metric type (fixed and expected IoU).</figcaption></figure>

Figure 17 shows Brier scores with $P_{\theta_c}$ as input (that are samples with confidence score greater than $\theta_c$, we keep their confidence scores), that compute prediction accuracy (binary classification). Notice that for low $\theta_c$, VGT outputs many bounding boxes with low confidence score, that are TN, which improves the accuracy, that decreases Brier scores. Overall, VGT has the lowest scores (best), which means that confidence scores are more trustworthy than TATR-detect-based models.

### 5.2.2 HTML-based TD evaluation

<table><caption>Table 8: Results for HTML-based table detection evaluation. For each metric $X$, there are 3 sub-metrics, $X_{.5}$ for HTML-Jaccard index threshold at 50 %, and $E_0[X]$, $E_{.5}[X]$ for expected value over IoU distribution. For probabilistic models, we use 3 arbitrary confidence score thresholds to define $P_{\theta_c}^+$.</caption><thead><tr><th colspan="2">Model</th><th colspan="3">Precision</th><th colspan="3">Recall</th><th colspan="3">F<sub>1</sub></th></tr><tr><th>Name</th><th>θ<sub>c</sub></th><th>P<sub>.5</sub></th><th>E<sub>0</sub>[P]</th><th>E<sub>.5</sub>[P]</th><th>R<sub>.5</sub></th><th>E<sub>0</sub>[R]</th><th>E<sub>.5</sub>[R]</th><th>F<sub>1..5</sub></th><th>E<sub>0</sub>[F<sub>1</sub>]</th><th>E<sub>.5</sub>[F<sub>1</sub>]</th></tr></thead><tbody><tr><td>GROBID</td><td></td><td>0.15</td><td>0.08</td><td>0.03</td><td>0.08</td><td>0.04</td><td>0.01</td><td>0.11</td><td>0.06</td><td>0.02</td></tr><tr><td>LLM-Vision</td><td></td><td>0.16</td><td>0.10</td><td>0.08</td><td>0.68</td><td>0.48</td><td>0.36</td><td>0.25</td><td>0.18</td><td>0.13</td></tr><tr><td rowspan="3">VGT+TATR-s</td><td>0.05</td><td>0.42</td><td>0.31</td><td>0.27</td><td><strong>0.93</strong></td><td>0.67</td><td>0.58</td><td>0.58</td><td>0.42</td><td>0.37</td></tr><tr><td>0.50</td><td>0.89</td><td>0.64</td><td>0.56</td><td>0.64</td><td>0.47</td><td>0.40</td><td>0.75</td><td>0.54</td><td>0.47</td></tr><tr><td>0.95</td><td>0.91</td><td><strong>0.84</strong></td><td><strong>0.82</strong></td><td>0.16</td><td>0.15</td><td>0.15</td><td>0.28</td><td>0.26</td><td>0.25</td></tr><tr><td rowspan="3">TATR-s</td><td>0.05</td><td>0.16</td><td>0.12</td><td>0.10</td><td>0.90</td><td>0.68</td><td>0.59</td><td>0.27</td><td>0.20</td><td>0.18</td></tr><tr><td>0.50</td><td>0.25</td><td>0.19</td><td>0.17</td><td>0.82</td><td>0.64</td><td>0.56</td><td>0.38</td><td>0.30</td><td>0.26</td></tr><tr><td>0.95</td><td><strong>0.92</strong></td><td>0.75</td><td>0.68</td><td>0.72</td><td>0.59</td><td>0.53</td><td>0.80</td><td>0.66</td><td><strong>0.60</strong></td></tr><tr><td rowspan="3">XY+TATR-s</td><td>0.05</td><td>0.06</td><td>0.05</td><td>0.04</td><td>0.91</td><td><strong>0.69</strong></td><td><strong>0.60</strong></td><td>0.12</td><td>0.09</td><td>0.08</td></tr><tr><td>0.50</td><td>0.14</td><td>0.11</td><td>0.09</td><td>0.89</td><td>0.67</td><td>0.58</td><td>0.24</td><td>0.18</td><td>0.16</td></tr><tr><td>0.95</td><td>0.90</td><td>0.68</td><td>0.59</td><td>0.86</td><td>0.67</td><td>0.56</td><td><strong>0.88</strong></td><td><strong>0.67</strong></td><td>0.58</td></tr></tbody></table>

Thanks to HTML-based evaluation we can compare our second baseline model, LLM-Vision, with the others. We needed this HTML-based TD to properly compare LLM-Vision scores, that use $P^{H++}$ with other model scores obtained from $P^{H++}$ with the same HTML-base method, and not from $P^{B++}$.

LLM-Vision has lower Precision than object-detection based models because it outputs a large number of tables and many of them are FP. LLM-Vision Recall is relatively high and constitutes a challenging baseline. However, TATR-detect-based models perform better on all Recall related metrics. Now expected value based metrics are computed using $J_k$ that now denotes HTML-Jaccard index. Once again, GROBID baseline model have been largely outperformed by previous models, but also by LLM-Vision.
Table detection evaluation based on HTML mark-up

<figure><img src="image_22.png" /><figcaption>Figure 18: Results for HTML-based table detection evaluation according to confidence score threshold $\theta_c$ that defines $P_{\theta_c}^+$. For each metric $X$, there are 3 sub-metrics, $X_{.5}$ for IoU threshold at 50 %, and $E_0[X]$, $E_{.5}[X]$ for expected value over IoU distribution. Models are distinguished by their colors and metric types by their line style.</figcaption></figure>
We notice that bbox-based scores (Figure 15) are similar to HTML-based (Figure 18), we will discuss this in Section 5.6.

## 5.3 Table Structure Recognition

We evaluate table structure recognition performance with the following metrics: GriTS Topology, GriTS Content and TEDS. All these metrics are normalized (denoted by *): HTML tables (prediction and ground truth) contain only the following tags: <table>, <tr>, <td> (no function analysis).

Scores are obtained by averaging scores from pairs (True Positives, GT), TP are in $P^{++} = P_{\theta_j}^+$, that is obtained by setting $\theta_J = 0.5$. As said, $P^+$ elements do not have necessarily GT associated. For positives that have a GT associated, if their the Jaccard index is too low, this means that the table has not been correctly detected. We want to evaluate the TSR task without being penalized by TD capacities. However, TSR and TD are not totally independent as we consider end-to-end models. That is why we decided to use, as test set, the TP tables detected by the TD part model $P^{++}$. Contrary to TD evaluation, here the number of input samples (denoted $|P_{0.5}^{++}|$) depends on the model performance over previous TD task. As a result, we cannot directly compare numbers columns by columns (that is why none of the scores are in bold in Tables 9, 10).

### 5.3.1 Bounding box-based TSR evaluation

<table><caption>Table 9: Results for bounding box-based table structure recognition evaluation over the set of TP samples $P_{0.5}^{++}$, with normalized metrics. For probabilistic models, we use 3 arbitrary confidence score thresholds to define $P_{\theta_c}^+$.</caption><thead><tr><th>Model</th><th>θ<sub>c</sub></th><th>GriTS<sup>*</sup><sub>Top</sub></th><th>GriTS<sup>*</sup><sub>Con</sub></th><th>TEDS<sup>*</sup></th><th>|P<sup>++</sup><sub>0.5</sub>|</th></tr></thead><tbody><tr><td>GROBID</td><td></td><td>0.63</td><td>0.57</td><td>0.45</td><td>15</td></tr><tr><td>LLM-Vison</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="3">VGT+TATR-structure</td><td>0.05</td><td>0.90</td><td>0.86</td><td>0.85</td><td>120</td></tr><tr><td>0.50</td><td>0.91</td><td>0.88</td><td>0.86</td><td>81</td></tr><tr><td>0.95</td><td>0.95</td><td>0.94</td><td>0.94</td><td>20</td></tr><tr><td rowspan="3">TATR-extract</td><td>0.05</td><td>0.90</td><td>0.86</td><td>0.85</td><td>119</td></tr><tr><td>0.50</td><td>0.91</td><td>0.87</td><td>0.85</td><td>109</td></tr><tr><td>0.95</td><td>0.93</td><td>0.90</td><td>0.89</td><td>93</td></tr><tr><td rowspan="3">XY-TATR-extract</td><td>0.05</td><td>0.89</td><td>0.86</td><td>0.85</td><td>119</td></tr><tr><td>0.50</td><td>0.89</td><td>0.86</td><td>0.84</td><td>116</td></tr><tr><td>0.95</td><td>0.89</td><td>0.86</td><td>0.84</td><td>113</td></tr></tbody></table>
Table structure recognition evaluation based on bounding box

<figure><img src="image_23.png" /><figcaption>Figure 19: Results for bounding box-based table structure evaluation according to confidence score threshold $\theta_c$ that defines $\mathcal{P}_{\theta_c}^+$. Metrics are computed over normalized HTML mark-up tables. GROBID is out of the frame, see Table 9.</figcaption></figure>
### 5.3.2 HTML-based TSR evaluation

<table><caption>Table 10: Results for HTML-based table structure recognition evaluation over the set of TP samples $P_{0.5}^{++}$, with normalized metrics For probabilistic models, we use 3 arbitrary confidence score thresholds to define $P_{\theta_c}^+$.</caption><thead><tr><th>Model</th><th>θ<sub>c</sub></th><th>GriTS<sup>*</sup><sub>Top</sub></th><th>GriTS<sup>*</sup><sub>Con</sub></th><th>TEDS<sup>*</sup></th><th>|P<sup>++</sup><sub>0.5</sub>|</th></tr></thead><tbody><tr><td>GROBID</td><td></td><td>0.74</td><td>0.71</td><td>0.58</td><td>10</td></tr><tr><td>LLM-Vison</td><td></td><td>0.94</td><td>0.85</td><td>0.86</td><td>84</td></tr><tr><td rowspan="3">VGT+TATR-structure</td><td>0.05</td><td>0.92</td><td>0.89</td><td>0.87</td><td>114</td></tr><tr><td>0.50</td><td>0.91</td><td>0.88</td><td>0.86</td><td>79</td></tr><tr><td>0.95</td><td>0.95</td><td>0.94</td><td>0.94</td><td>20</td></tr><tr><td rowspan="3">TATR-extract</td><td>0.05</td><td>0.91</td><td>0.88</td><td>0.87</td><td>111</td></tr><tr><td>0.50</td><td>0.92</td><td>0.89</td><td>0.88</td><td>101</td></tr><tr><td>0.95</td><td>0.94</td><td>0.92</td><td>0.91</td><td>88</td></tr><tr><td rowspan="3">XY-TATR-extract</td><td>0.05</td><td>0.90</td><td>0.88</td><td>0.87</td><td>112</td></tr><tr><td>0.50</td><td>0.90</td><td>0.88</td><td>0.87</td><td>109</td></tr><tr><td>0.95</td><td>0.90</td><td>0.88</td><td>0.87</td><td>106</td></tr></tbody></table>

GROBID is largely outperformed by other models, even over a small set (between 10 and 15 TP tables). GROBID struggles with structure recognition because it sees tables line by line and fail to understand spanning rows. Beyond GROBID case, scores (Figure 10) are very close to each other. We naturally observe that the more TP a model has, the lower are TSR metric scores, because with more tables correctly detected, the probability to make a mistake during TSR increases. That is the case for XY, that handles more TP.

LLM-Vision achieves strong Topology recognition, but lower scores on content related metrics (GriTS content and TEDS) because of hallucinations (Appendix C). For LLM-Vision, finding the write prompt that aligns with our expectation was tough. For instance, LLM-Vision transposes tables when their first column is in bold (usually the fist row is bold because it contains column names), the model extracts the data but modify its structure, something that we decided to penalize (metrics could have been invariant by transpositions). Furthermore, for table structure recognition evaluation, metrics use as input table represented as HTML mark-up string, and might fail to parse LLM-Vision output HTML mark-up (e.g. `<` should be encoded with `&lt;`).

TATR-structure-based models obtain similar results as excepted. These models, cannot achieve perfect GriTS<sub>Con</sub>, nor TEDS, score because they used tokens (textual information) provided by the parser PDFAlto that does not retrieve all text from the document due to its limitations (Section 5.5).
Table structure recognition evaluation based on HTML mark-up

<figure><img src="image_24.png" /><figcaption>Figure 20: Results for HTML-based table structure evaluation according to confidence score threshold $\theta_c$ that defines $P_{\theta_c}^+$. Metrics are computed over normalized HTML mark-up tables. GROBID is out of the frame, see Table 10.</figcaption></figure>
## 5.4 Model confidence

For probabilistic models, we observe miscalibration with the following Figure 21, from predictions in $P_{0.05}$. Top histograms (Figure 21) show the distribution of confidence scores. We observe that VGT outputs a large number of sample with low table confidence score, which confirms our analysis about low Brier score (accuracy measure improved by TN). Reliability diagrams [26] show accuracy as a function of confidence by using interval (width 0.1). For a perfect calibration, we obtain the identity function (hatched histogram). The gap between model outputs and identity histogram underlines how much the model is miscalibrated. TATR-detect-based models are poorly calibrated compare to VGT, that obtains lower ECE (↓).

<figure><img src="image_25.png" /><figcaption>Figure 21: Confidence histograms (top) and reliability diagrams (bottom) for TATR-detect (left), VGT (center) and XY+TATR-detect (right) on validation dataset.</figcaption></figure>

## 5.5 Current pipeline limits: PDFAlto

PDFAlto (PDF parser) provides TATR-structure with a list of tokens (words) and their bounding boxes in image coordinates. Hence, issues with PDFAlto penalize TD and TSR performance (for models using it). There are two main limits in our pipeline:

* PDFAlto does not retrieve all text on PDF page, especially stylized text like equations. Which implies partial table extraction for TATR-structure-based models.

* As PDF parser, PDFAlto can only extract the text content from the PDF, not the one from images. So there is a drawback that is not due to the table detection models but the parser. For example, we remove detected tables that do not contain tokens, as post-processing (for TATR-detect and VGT).

To solve this issue, we could run OCR on images (within PDF file), or OCR over the whole file.
## 5.6 Bounding box and HTML-based evaluation

We created a HTML-based evaluation in order to compare LLM-Vision with other models. Thanks to models that were evaluated with both methods, we can compare results obtained with the two strategies. We notice that bbox-based scores (Figure 15) are similar to HTML-based (Figure 18), which is confirmed with Figure 22 that shows correlation between bbox-based $X_{.5}$ and HTML-based ones. This demonstrates that the HTML-Jaccard index built is globally efficient at measuring similarity between tables and reaches similar results.

<figure><img src="image_26.png" /><figcaption>Figure 22: HTML-based Precision, Recall, $F_1$ as a function of bounding box-based ones with $\theta_J = 0.5$.</figcaption></figure>

We evaluate HTML-Jaccard index performance as a classifier that distinguishes a TP from a FP, according to ground truth set by IoU. We obtain a confusion matrix by setting $\theta_J = 0.5$ for both Jaccard index. Accuracy (TP+TN over all samples) is relatively high, but there are some FNs. We notice that many samples achieve perfect HTML-Jaccard score, these are cases where PDFAlto does not lose any information. For VGT, we notice that IoU are usually very high (bounding boxes match the GT), but HTML-Jaccard spread from 1 to 0.5, due to PDFAlto limits discussed (Section 5.5).
<figure><img src="image_27.png" /><figcaption>Figure 23: HTML-Jaccard index distribution over bounding box Jaccard index (IoU) distribution. With $\theta_J = 0.5$, we evaluate HTML-Jaccard classifier with respect to IoU, to obtain a confusion matrix.</figcaption></figure>

# 6 Conclusion and outlook

The table extraction task is not a solved problem (neither TD nor TSR). Various deep learning based approaches have improved *state-of-the-art* results, however, there is no perfect solution. Some models are focusing on certain datatypes, for example scans of paper documents with distortion [24], other on robustness. We tackled the problem for a specific input type: PDF reports (BRGM data). Nevertheless, GROBID was the only model that leveraged the PDF format as input: other models handle images as they rely on image object detection. Thanks to the PDF format, we have easily retrieved tokens (needed for TATR-structure) through the PDF parser, PDFAlto, even if the token extraction was limited (internal limits and failed extractions), preventing better results. We manage to achieve good performance on both subtasks with:

* TATR-extract by modifying its pipeline (consider non dominant score, add post-processing: NMS and remove empty tables).

* VGT+TATR-structure, by using another TD model, VGT, to create a new end-to-end more calibrated model that provides more confidence in prediction.

* XY+TATR-extract, by adding additional pre-processing to use sub-images as input, thanks to XY-cut method.

On the one hand, GROBID obtains the worst results in every aspect: prediction precision and ability to detect all tables; and for structure recognition; this is not a suitable solution for table extraction. On the other hand, LLM-Vision obtains honorable results, especially with relatively competitive Recall and TSR scores, but the overall performance does not reach our best models. XY+TATR-extract appears to be the most promising model, with respect to PR-curve (Figures 15 and 16, with IoU threshold at 0.5), as we observe the best precision-recall trade-off. However, expected value based metrics offer another way of analyzing and nuancing results. In fact, regarding AP-curve for expect values (mean over IoU threshold), we observe that VGT+TATR-structure takes the lead with better IoU distribution. VGT is the best calibrated model, compared to (XY+)TATR-detect according to Brier score results and reliability diagrams. On downstream task, the $\theta_c$ choice matters: either we set a high threshold and expect to get Positive predictions, or we set low threshold and confidence scores and rely on
confidence scores as probabilities. We could try reconfidencing TATR-detect based models to obtain better calibration and more interpretable scores. For TATR-structure, we could also use confidence score over structure recognition labels: “row”, “column”, etc. for better structure recognition track, instead of just using the table’s confidence score. We would also like to evaluate these models with larger data sets. The PhD following this internship will tackle other data formats (like geological maps, data basis, schemas) and heterogeneity of data formats within a document. The project will also aim at representing spatial and temporal dimensions of data, while tracking the data provenance.

## 6.1 Personal feedback

I really enjoyed my 5-month internship at Inria Valda and Inria Cedar, whether it was for the daily work, the project goals, or the environment at ENS Paris. I have enjoyed working with my supervisors and appreciate their support throughout this project. Moreover, the project led by Inria and BRGM is meaningful to me and inspiring. During my internship, I had the opportunity to visit the BRGM at Orléans to meet Cécile Gracianne and geologists in order to learn more about their work and our project goals. This visit was very interesting and allowed me to better understand the context in which the internship took place.

I discovered what research daily work looks like, and it turned out to be exactly what I was searching for: with plenty of freedom to explore ideas. I studied what table extraction actually means, how to correctly evaluate the task, and what the state-of-the-art models are. I reviewed papers and experimented with open-source models. I am happy with the work we produced within the given time, even though there are many ideas that I wish I had the time to try. Weekly meetings with supervisors were the opportunity to improve my oral expression skills, by prioritizing my ideas during my presentation. This internship was rewarding and reinforced my idea of continuing in academic research. Finally, I am happy to continue this project with a PhD with my current internship supervisors.
# Appendix

## A Dataset Test

<table><caption>Table 11: Files used for TD and TSR evaluation.</caption><thead><tr><th>Name File</th><th>Pages</th><th>Tables</th></tr></thead><tbody><tr><td>crgeos20211464151_v1</td><td>5</td><td>4</td></tr><tr><td>CRGEOS_2022_354_S2_317_0</td><td>26</td><td>5</td></tr><tr><td>RP_53832_FR_AleaSismique_Lycee_Chirongui</td><td>62</td><td>17</td></tr><tr><td>Rapport_RP-71365-FR_finale_corrNumRapport</td><td>168</td><td>23</td></tr><tr><td>Noteamiro-BCSF-RENASS-Mayotte-13-07-2018</td><td>62</td><td>21</td></tr><tr><td>RP-61170-FR_Etude_sismique</td><td>176</td><td>54</td></tr><tr><td>Total</td><td>499</td><td>124</td></tr></tbody></table>

## B Proofs

*Proof of Proposition 4.1.* Let $\theta_J$ be a threshold real random variable with probability density function $f$. We tackle the Precision case and formally compute $P_{\theta_J}$. Define $P_{\theta_J}$, $TP_{\theta_J}$, and $FP_{\theta_J}$ as random variables at threshold $\theta_J$. Let $p$ and $g$ represent the number of predictions (positives) and the number of ground truths, respectively. The assignment problem between predictions and ground truths is solved using the Hungarian algorithm: $min(p, g)$ predictions are assigned to $min(p, g)$ ground truths, with their Jaccard index scores denoted as $J_k = \text{Jaccard}(\hat{Y}_k, Y_k)$. Among $p$ predictions, only $min(p, g)$ can be assigned to a ground truth, leaving the rest as false positives.

$$P_{\theta_J} = \frac{TP_{\theta_J}}{TP_{\theta_J} + FP_{\theta_J}} \qquad (21)$$

$$= \frac{TP_{\theta_J}}{p} \qquad (22)$$

$$= \frac{1}{p} \sum_{k=1}^{\min(p,g)} 1_{[J_k > \theta_J]} \qquad (23)$$

Taking the expected value over the random variable $\theta_J$ (linearly), we obtain:

$$\mathbb{E}_{\theta_J \sim f}[P_{\theta_J}] = \frac{1}{p} \sum_{k=1}^{\min(p,g)} \mathbb{E}_{\theta_J \sim f}[1_{[J_k > \theta_J]}] \qquad (25)$$

Since $R_{\theta_J} = \frac{TP_{\theta_J}}{g}$ and $F_{1,\theta_J} = 2 \frac{TP_{\theta_J}}{g+p}$, we derive the following results:

$$\mathbb{E}_{\theta_J \sim f}[P_{\theta_J}] = \frac{1}{p} \sum_{k=1}^{\min(p,g)} \mathbb{E}_{\theta_J \sim f}[1_{[J_k > \theta_J]}]$$

$$\mathbb{E}_{\theta_J \sim f}[R_{\theta_J}] = \frac{1}{g} \sum_{k=1}^{\min(p,g)} \mathbb{E}_{\theta_J \sim f}[1_{[J_k > \theta_J]}] \qquad (27)$$

$$\mathbb{E}_{\theta_J \sim f}[F_{1,\theta_J}] = \frac{2}{p+g} \sum_{k=1}^{\min(p,g)} \mathbb{E}_{\theta_J \sim f}[1_{[J_k > \theta_J]}]$$

□
Proof of Proposition 4.2. We use two probability density functions:

$$f_0(\theta) \propto \theta \mathbb{1}_{[0,1]}(\theta), \qquad (28)$$

$$f_{0.5}(\theta) \propto \theta \mathbb{1}_{[0.5,1]}(\theta). \qquad (29)$$

After normalization, ensuring $\int_{\mathbb{R}} f \, d\lambda = 1$, we obtain:

$$f_0(\theta) = 2\theta \mathbb{1}_{[0,1]}(\theta), \qquad (30)$$

$$f_{0.5}(\theta) = \frac{8}{3} \theta \mathbb{1}_{[0.5,1]}(\theta). \qquad (31)$$

Let $J_k \in [0, 1]$ be the Jaccard score for prediction $k$. We compute:

$$\mathbb{E}_{\theta_J \sim f_0} [\mathbb{1}_{[J_k > \theta_J]}] = \int_{\mathbb{R}} \mathbb{1}_{[J_k > \theta]} f_0(\theta) \, d\theta \qquad (32)$$

$$= \int_{0}^{J_k} f_0(\theta) \, d\theta \qquad (33)$$

$$= \int_{0}^{J_k} 2\theta \, d\theta \qquad (34)$$

$$= J_k^2. \qquad (35)$$

Similarly, for $f_{0.5}$:

$$\mathbb{E}_{\theta_J \sim f_{0.5}} [\mathbb{1}_{[J_k > \theta_J]}] = \int_{\mathbb{R}} \mathbb{1}_{[J_k > \theta]} f_{0.5}(\theta) \, d\theta \qquad (36)$$

$$= \int_{0}^{J_k} f_{0.5}(\theta) \, d\theta \qquad (37)$$

$$= \int_{0}^{J_k} \frac{8}{3} \theta \mathbb{1}_{[J_k > 0.5]}(J_k) \, d\theta \qquad (38)$$

$$= \frac{4}{3} \left( J_k^2 - \frac{1}{4} \right) \mathbb{1}_{[J_k > 0.5]}(J_k). \qquad (39)$$

Hence the results.

□

Proof of Proposition 4.3. The Brier score $B_{\theta_J}$ evaluates the accuracy of binary classification, where $c_k$ are confidence scores (probabilities) and $o_k$ boolean ground truth values (depending on $\theta_J$). Since $o_k$ depends on the $\theta_J$ parameter, we compute the expected value of $B_{\theta_J}$ assuming $\theta_J$ is sampled according to $f$. With, $J_k = \delta_k \times \text{Jaccard}(\hat{Y}_k, Y_k)$, we obtain:

$$B_{\theta_J} = \frac{1}{|\mathcal{P}|} \sum_{k \in \mathcal{P}} (c_k - o_k)^2 \qquad (40)$$

$$= \frac{1}{|\mathcal{P}|} \sum_{k \in \mathcal{P}} (c_k - \mathbb{1}_{[J_k > \theta_J]})^2 \qquad (41)$$

(42)

Taking expectation:

$$\mathbb{E}[B_{\theta_J}] = \frac{1}{|\mathcal{P}|} \sum_{k \in \mathcal{P}} \mathbb{E}[(c_k - \mathbb{1}_{[J_k > \theta_J]})^2] \qquad (43)$$

(44)

Expanding the expectation:

$$\mathbb{E}[(c_k - \mathbb{1}_{[J_k > \theta_J]})^2] = c_k^2 - 2c_k \mathbb{E}[\mathbb{1}_{[J_k > \theta_J]}] + \mathbb{E}[\mathbb{1}_{[J_k > \theta_J]}^2] \qquad (45)$$

$$= c_k^2 + (1 - 2c_k) \mathbb{E}[\mathbb{1}_{[J_k > \theta_J]}] \qquad (46)$$

(47)
Thus, we obtain:

$$ \mathbb{E}_{\theta_J \sim f}[B_{\theta_J}] = \frac{1}{|\mathcal{P}|} \sum_{k \in \mathcal{P}} c_k^2 + (1 - 2c_k) \mathbb{E}[1_{[J_k > \theta_J]}] \quad (48) $$

**Remark: Relationship between Brier score and Precision** We observe that Precision and Brier score (accuracy) are related when computed using the same confidence score threshold. We express the Precision with the notation of the Brier score:

$$ P_{\theta_J}^{\theta_c} = \frac{1}{p} \sum_{k=1}^{\min(p,g)} 1_{[J_k > \theta_J]} \quad (49) $$

$$ = \frac{1}{|\mathcal{P}_{\theta_c}^+|} \sum_{k=1}^{\min(p,g)} 1_{[J_k > \theta_J]} \quad \text{Since } |\mathcal{P}_{\theta_c}^+| = p \quad (50) $$

$$ = \frac{1}{|\mathcal{P}_{\theta_c}^+|} \sum_{k \in \mathcal{P}_{\theta_c}^+} o_k \quad o_k = \delta_k 1_{[J_k > \theta_J]} \quad (51) $$

(52)

Rearranging, we obtain:

$$ 1 - P_{\theta_J}^{\theta_c} = \frac{1}{|\mathcal{P}_{\theta_c}^+|} \sum_{k \in \mathcal{P}_{\theta_c}^+} 1 - o_k \quad (53) $$

$$ = \frac{1}{|\mathcal{P}_{\theta_c}^+|} \sum_{k \in \mathcal{P}_{\theta_c}^+} (1 - o_k)^2 \quad \text{Binary values.} \quad (54) $$

(55)

By definition, the Brier score $B_{\theta_J}^{\theta_c}$ for inputs in $\mathcal{P}_{\theta_c}$ is given by:

$$ B_{\theta_J}^{\theta_c} = \frac{1}{|\mathcal{P}_{\theta_c}|} \sum_{k \in \mathcal{P}_{\theta_c}} (c_k - o_k)^2 \quad (56) $$

These equations are similar, except that in the Brier score, we use confidence score in [0, 1], whereas in Precision, the score is either 0 or 1. Additionally, the input predictions are not taken from exactly the same set.

* The set $\mathcal{P}_{\theta_c}^+ = \{\hat{y} | (\hat{y}, c) \in \mathcal{P}, c > \theta_J\}$ denotes positive samples. Elements in this set are considered as tables with 100% certainty.

* The set $\mathcal{P}_{\theta_c} = \{(\hat{y}, c) \in \mathcal{P} | c > \theta_J\}$ denotes a subset of predictions. Elements in this set are tuples composed of $\hat{y}$ (representing table with bounding boxes or HTML markup) and the confidence score $c$ indicating the probability that $\hat{y}$ is a table.

# C LLM-Vision hallucinations

We have noticed that when recognizing table structure, LLMs may hallucinate, i.e., report tokens that were not present in the actual input. Figure 24 shows a sample GT table, for which Figure 25a provides the image given as input to the LLM. As Figure 25b shows, the LLM:

* Erroneously replaces some I (capital i) with the digit 1;

* Reformats the numbers from the European to the US format (this is arguably not problematic); but also

* “Fills in” a missing value in an empty cell, and

* Changes and makes up many numeric values.
Supplementary Table S1. (continued)

<table><thead><tr><th>Dates TU</th><th>M<sub>lv</sub></th><th>Hypocenter (km)</th><th>Latitude</th><th>Longitude</th><th>Records</th></tr></thead><tbody><tr><td>04/06/2018 21:20:00</td><td>55,700</td><td>3800</td><td>-12,7933</td><td>45,6365</td><td>YTMZ, MILA</td></tr><tr><td>05/06/2018 23:00:00</td><td>52,900</td><td>2700</td><td>-12,8490</td><td>45,6352</td><td>YTMZ, MILA</td></tr><tr><td>06/06/2018 09:33:00</td><td>51,400</td><td>323</td><td>-12,8413</td><td>45,6363</td><td>YTMZ, MILA</td></tr><tr><td>07/06/2018 13:07:00</td><td>49,800</td><td>2998</td><td>-12,8577</td><td>45,6135</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018 00:56:00</td><td>48,600</td><td>000</td><td>-12,8173</td><td>45,6232</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018 03:31:00</td><td>48,200</td><td>3600</td><td>-12,8303</td><td>45,5622</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018 12:03:00</td><td>49,700</td><td>3600</td><td>-12,8117</td><td>45,5628</td><td>YTMZ, MILA</td></tr><tr><td>09/06/2018 22:36:00</td><td>50,900</td><td>2298</td><td>-12,9020</td><td>45,6558</td><td>YTMZ, MILA</td></tr><tr><td>12/06/2018 17:17:00</td><td>55,800</td><td>4000</td><td>-12,8888</td><td>45,6375</td><td>YTMZ, MILA</td></tr><tr><td>18/06/2018 13:43:00</td><td>52,100</td><td>2000</td><td>-12,8830</td><td>45,7238</td><td>YTMZ, MILA</td></tr><tr><td>19/06/2018 21:36:00</td><td>47,500</td><td>2000</td><td>-12,8428</td><td>45,7028</td><td>YTMZ, MILA</td></tr><tr><td>22/06/2018 12:50:00</td><td>48,100</td><td>2000</td><td>-12,7503</td><td>45,7673</td><td>YTMZ, MILA</td></tr><tr><td>23/06/2018 19:45:30</td><td>53,400</td><td>2000</td><td>-12,8900</td><td>45,6707</td><td>YTMZ, MILA</td></tr><tr><td>25/06/2018 17:42:00</td><td>55,100</td><td>800</td><td>-12,8488</td><td>45,7088</td><td>YTMZ, MILA</td></tr><tr><td>27/06/2018 06:41:00</td><td>55,700</td><td>000</td><td>-12,8532</td><td>45,6530</td><td>YTMZ, MILA</td></tr><tr><td>25/05/2019 13:08:06</td><td>43,700</td><td>3200</td><td>-12,7480</td><td>45,3963</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>28/05/2019 10:53:20</td><td>45,300</td><td>3233</td><td>-12,8122</td><td>45,4205</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>06/06/2019 14:53:31</td><td>43,800</td><td>2533</td><td>-12,7462</td><td>45,4207</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>14/06/2019 02:05:15</td><td>46,300</td><td>3200</td><td>-12,8128</td><td>45,4045</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>15/06/2019 03:45:08</td><td>50,500</td><td>3900</td><td>-12,8208</td><td>45,5338</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/06/2019 07:31:39</td><td>47,100</td><td>2589</td><td>-12,8143</td><td>45,4000</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>30/08/2019 00:32:47</td><td>49,700</td><td>3800</td><td>-12,8135</td><td>45,5498</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>20/10/2019 01:42:05</td><td>42,700</td><td>3346</td><td>-12,7880</td><td>45,5962</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>23/10/2019 22:44:29</td><td>44,800</td><td>2700</td><td>-12,7975</td><td>45,4537</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>07/11/2019 17:29:58</td><td>44,300</td><td>3000</td><td>-12,7863</td><td>45,5880</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>15/12/2019 19:13:34</td><td>39,600</td><td>2600</td><td>-12,7723</td><td>45,4267</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/12/2019 04:13:26</td><td>35,800</td><td>3600</td><td>-12,8405</td><td>45,5517</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/12/2019 12:09:23</td><td>30,700</td><td>4000</td><td>-12,7990</td><td>45,4095</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>17/12/2019 17:45:18</td><td></td><td>4000</td><td>-12,7890</td><td>45,3122</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>21/12/2019 03:06:12</td><td>36,800</td><td>3300</td><td>-12,7818</td><td>45,4060</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>24/12/2019 23:27:25</td><td>37,900</td><td>3100</td><td>-12,7923</td><td>45,6215</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>26/12/2019 08:22:44</td><td>29,000</td><td>2949</td><td>-12,8617</td><td>45,3810</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>02/01/2020 16:00:33</td><td>53,100</td><td>3642</td><td>-12,8022</td><td>45,5577</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>04/01/2020 09:47:49</td><td>34,500</td><td>3187</td><td>-12,7903</td><td>45,4343</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>05/01/2020 06:06:30</td><td>36,600</td><td>3542</td><td>-12,1962</td><td>45,8258</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>06/01/2020 00:05:42</td><td>36,000</td><td>3753</td><td>-12,7975</td><td>45,5217</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>07/01/2020 08:33:59</td><td>35,700</td><td>3300</td><td>-12,8428</td><td>45,5382</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>08/01/2020 05:04:58</td><td>38,600</td><td>2800</td><td>-12,8208</td><td>45,4197</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>09/01/2020 11:47:27</td><td>34,100</td><td>3418</td><td>-12,8072</td><td>45,6012</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>23/01/2020 09:56:40</td><td>34,600</td><td>2432</td><td>-12,8487</td><td>45,4072</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr></tbody></table>

(continued on next page)

<caption>Figure 24: Page containing a table ground truth.</caption>
<table><thead><tr><th>Dates TU</th><th>M lv</th><th>Hypocenter (km)</th><th>Latitude</th><th>Longitude</th><th>Records</th></tr></thead><tbody><tr><td>04/06/2018<br>21:20:00</td><td>55,700</td><td>3800</td><td>-12,7933</td><td>45,6365</td><td>YTMZ, MILA</td></tr><tr><td>05/06/2018<br>23:00:00</td><td>52,900</td><td>2700</td><td>-12,8490</td><td>45,6352</td><td>YTMZ, MILA</td></tr><tr><td>06/06/2018<br>09:33:00</td><td>51,400</td><td>323</td><td>-12,8413</td><td>45,6363</td><td>YTMZ, MILA</td></tr><tr><td>07/06/2018<br>13:07:00</td><td>49,800</td><td>2998</td><td>-12,8577</td><td>45,6135</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018<br>00:56:00</td><td>48,600</td><td>000</td><td>-12,8173</td><td>45,6232</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018<br>03:31:00</td><td>48,200</td><td>3600</td><td>-12,8303</td><td>45,5622</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018<br>12:03:00</td><td>49,700</td><td>3600</td><td>-12,8117</td><td>45,5628</td><td>YTMZ, MILA</td></tr><tr><td>09/06/2018<br>22:36:00</td><td>50,900</td><td>2298</td><td>-12,9020</td><td>45,6558</td><td>YTMZ, MILA</td></tr><tr><td>12/06/2018<br>17:17:00</td><td>55,800</td><td>4000</td><td>-12,8888</td><td>45,6375</td><td>YTMZ, MILA</td></tr><tr><td>18/06/2018<br>13:43:00</td><td>52,100</td><td>2000</td><td>-12,8830</td><td>45,7238</td><td>YTMZ, MILA</td></tr><tr><td>19/06/2018<br>21:36:00</td><td>47,500</td><td>2000</td><td>-12,8428</td><td>45,7028</td><td>YTMZ, MILA</td></tr><tr><td>22/06/2018<br>12:50:00</td><td>48,100</td><td>2000</td><td>-12,7503</td><td>45,7673</td><td>YTMZ, MILA</td></tr><tr><td>23/06/2018<br>19:45:30</td><td>53,400</td><td>2000</td><td>-12,8900</td><td>45,6707</td><td>YTMZ, MILA</td></tr><tr><td>25/06/2018<br>17:42:00</td><td>55,100</td><td>800</td><td>-12,8488</td><td>45,7088</td><td>YTMZ, MILA</td></tr><tr><td>27/06/2018<br>06:41:00</td><td>55,700</td><td>000</td><td>-12,8532</td><td>45,6530</td><td>YTMZ, MILA</td></tr><tr><td>25/05/2019<br>13:08:06</td><td>43,700</td><td>3200</td><td>-12,7480</td><td>45,3963</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>28/05/2019<br>10:53:20</td><td>45,300</td><td>3233</td><td>-12,8122</td><td>45,4205</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>06/06/2019<br>14:53:31</td><td>43,800</td><td>2533</td><td>-12,7462</td><td>45,4207</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>14/06/2019<br>02:05:15</td><td>46,300</td><td>3200</td><td>-12,8128</td><td>45,4045</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>15/06/2019<br>03:45:08</td><td>50,500</td><td>3900</td><td>-12,8208</td><td>45,5338</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/06/2019<br>07:31:39</td><td>47,100</td><td>2589</td><td>-12,8143</td><td>45,4000</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>30/08/2019<br>00:32:47</td><td>49,700</td><td>3800</td><td>-12,8135</td><td>45,5498</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>20/10/2019<br>01:42:05</td><td>42,700</td><td>3346</td><td>-12,7880</td><td>45,5962</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>23/10/2019<br>22:44:29</td><td>44,800</td><td>2700</td><td>-12,7975</td><td>45,4537</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>07/11/2019<br>17:29:58</td><td>44,300</td><td>3000</td><td>-12,7863</td><td>45,5880</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>15/12/2019<br>19:13:34</td><td>39,600</td><td>2600</td><td>-12,7723</td><td>45,4267</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/12/2019<br>04:13:26</td><td>35,800</td><td>3600</td><td>-12,8405</td><td>45,5517</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/12/2019<br>12:09:23</td><td>30,700</td><td>4000</td><td>-12,7990</td><td>45,4095</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>17/12/2019<br>17:45:18</td><td>4000</td><td></td><td>-12,7890</td><td>45,3122</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>21/12/2019<br>03:06:12</td><td>36,800</td><td>3300</td><td>-12,7818</td><td>45,4060</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>24/12/2019<br>23:27:25</td><td>37,900</td><td>3100</td><td>-12,7923</td><td>45,6215</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>26/12/2019<br>08:22:44</td><td>29,000</td><td>2949</td><td>-12,8617</td><td>45,3810</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>02/01/2020<br>16:00:33</td><td>53,100</td><td>3642</td><td>-12,8022</td><td>45,5577</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>04/01/2020<br>09:47:49</td><td>34,500</td><td>3187</td><td>-12,7903</td><td>45,4343</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>05/01/2020<br>06:06:30</td><td>36,600</td><td>3542</td><td>-12,1962</td><td>45,8258</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>06/01/2020<br>00:05:42</td><td>36,000</td><td>3753</td><td>-12,7975</td><td>45,5217</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>07/01/2020<br>08:33:59</td><td>35,700</td><td>3300</td><td>-12,8428</td><td>45,5382</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>08/01/2020<br>05:04:58</td><td>38,600</td><td>2800</td><td>-12,8208</td><td>45,4197</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>09/01/2020<br>11:47:27</td><td>34,100</td><td>3418</td><td>-12,8072</td><td>45,6012</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>23/01/2020<br>09:56:40</td><td>34,600</td><td>2432</td><td>-12,8487</td><td>45,4072</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr></tbody></table>

<table><thead><tr><th>Dates TU</th><th>Mlw</th><th>Hypocenter (km)</th><th>Latitude</th><th>Longitude</th><th>Records</th></tr></thead><tbody><tr><td>04/06/2018<br>21:20:00</td><td>55,700</td><td>3800</td><td>-12,7933</td><td>45,6365</td><td>YTMZ, MILA</td></tr><tr><td>05/06/2018<br>23:00:00</td><td>52,900</td><td>2700</td><td>-12,8490</td><td>45,6352</td><td>YTMZ, MILA</td></tr><tr><td>06/06/2018<br>09:33:00</td><td>48,600</td><td>323</td><td>-12,8413</td><td>45,6363</td><td>YTMZ, MILA</td></tr><tr><td>07/06/2018<br>13:07:00</td><td>49,800</td><td>2998</td><td>-12,8577</td><td>45,6135</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018<br>06:56:00</td><td>48,600</td><td>000</td><td>-12,8173</td><td>45,6232</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018<br>03:13:00</td><td>48,200</td><td>3600</td><td>-12,8303</td><td>45,5622</td><td>YTMZ, MILA</td></tr><tr><td>08/06/2018<br>12:03:00</td><td>49,700</td><td>3600</td><td>-12,8117</td><td>45,5628</td><td>YTMZ, MILA</td></tr><tr><td>09/06/2018<br>22:36:00</td><td>50,900</td><td>2298</td><td>-12,9020</td><td>45,6558</td><td>YTMZ, MILA</td></tr><tr><td>12/06/2018<br>17:17:00</td><td>55,800</td><td>4000</td><td>-12,8888</td><td>45,6375</td><td>YTMZ, MILA</td></tr><tr><td>18/06/2018<br>13:43:00</td><td>52,100</td><td>2000</td><td>-12,8830</td><td>45,7238</td><td>YTMZ, MILA</td></tr><tr><td>19/06/2018<br>21:36:00</td><td>47,500</td><td>2000</td><td>-12,8428</td><td>45,7028</td><td>YTMZ, MILA</td></tr><tr><td>22/06/2018<br>12:50:00</td><td>48,100</td><td>2000</td><td>-12,7503</td><td>45,7673</td><td>YTMZ, MILA</td></tr><tr><td>23/06/2018<br>19:45:30</td><td>53,400</td><td>2000</td><td>-12,8900</td><td>45,6707</td><td>YTMZ, MILA</td></tr><tr><td>25/06/2018<br>17:42:00</td><td>55,100</td><td>800</td><td>-12,8488</td><td>45,7088</td><td>YTMZ, MILA</td></tr><tr><td>27/06/2018<br>06:41:00</td><td>55,700</td><td>2000</td><td>-12,8532</td><td>45,6530</td><td>YTMZ, MILA</td></tr><tr><td>25/05/2019<br>13:06:00</td><td>43,700</td><td>3200</td><td>-12,7480</td><td>45,3963</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>28/05/2019<br>15:32:00</td><td>43,800</td><td>2533</td><td>-12,8122</td><td>45,4205</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>06/06/2019<br>14:53:31</td><td>43,800</td><td>2533</td><td>-12,7462</td><td>45,4207</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>14/06/2019<br>02:05:15</td><td>46,300</td><td>3200</td><td>-12,8128</td><td>45,4045</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>15/06/2019<br>03:45:08</td><td>50,500</td><td>3900</td><td>-12,8208</td><td>45,5338</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/06/2019<br>07:31:39</td><td>47,100</td><td>2589</td><td>-12,8143</td><td>45,4000</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>30/08/2019<br>00:32:47</td><td>49,700</td><td>3800</td><td>-12,8135</td><td>45,5498</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>20/10/2019<br>01:42:05</td><td>42,700</td><td>3346</td><td>-12,7808</td><td>45,5962</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>21/10/2019<br>22:44:29</td><td>44,800</td><td>2700</td><td>-12,7975</td><td>45,4537</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>07/11/2019<br>12:59:58</td><td>34,300</td><td>3000</td><td>-12,7863</td><td>45,5880</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>15/12/2019<br>19:13:34</td><td>39,600</td><td>2600</td><td>-12,7723</td><td>45,4267</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>16/12/2019<br>04:13:26</td><td>35,800</td><td>3000</td><td>-12,8405</td><td>45,5171</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>17/12/2019<br>12:09:23</td><td>30,700</td><td>4000</td><td>-12,7990</td><td>45,4095</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>17/12/2019<br>17:45:18</td><td>30,600</td><td>4000</td><td>-12,7890</td><td>45,3122</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>21/12/2019<br>03:06:12</td><td>36,800</td><td>3300</td><td>-12,7818</td><td>45,4060</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>24/12/2019<br>23:27:25</td><td>37,900</td><td>3100</td><td>-12,7923</td><td>45,6215</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>26/12/2019<br>08:22:44</td><td>29,000</td><td>2949</td><td>-12,8617</td><td>45,3810</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>02/01/2020<br>10:03:33</td><td>53,100</td><td>3642</td><td>-12,8022</td><td>45,5577</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>04/01/2020<br>09:47:49</td><td>34,500</td><td>3187</td><td>-12,7903</td><td>45,4343</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>05/01/2020<br>06:06:30</td><td>36,600</td><td>3542</td><td>-12,9162</td><td>45,8258</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>06/01/2020<br>00:54:32</td><td>36,000</td><td>3753</td><td>-12,7975</td><td>45,5217</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>08/01/2020<br>08:35:59</td><td>33,000</td><td>3300</td><td>-12,8482</td><td>45,5382</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>08/01/2020<br>05:48:58</td><td>38,200</td><td>2128</td><td>-12,8202</td><td>45,4197</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>09/01/2020<br>11:47:27</td><td>34,100</td><td>3418</td><td>-12,8076</td><td>45,6012</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr><tr><td>23/01/2020<br>09:56:40</td><td>34,000</td><td>2432</td><td>-12,8487</td><td>45,4072</td><td>YTMZ, MCHI, MILA, MTSB, KNKL, PMZI, R1EE2</td></tr></tbody></table>

(a) Table ground truth (as HTML mark-up) from the sample Figure 24

(b) LLM-Vision prediction (as HTML mark-up) from the image in Figure 24.

<caption>Figure 25: On the left (Figure 25a), the ground truth table, and on the right (Figure 25b) LLM-Vision's prediction containing errors.</caption>
List of Figures

<table>
  <tbody>
    <tr>
      <td>1</td>
      <td>GROBID pipeline for table extraction. GROBID outputs a TEI.XML structured file from which we retrieve all tables extracted: table location (page number and bounding box coordinates (x<sub>0</sub>, y<sub>0</sub>, x<sub>1</sub>, y<sub>1</sub>)) and the table as XML mark-up</td>
      <td>5</td>
    </tr>
    <tr>
      <td>2</td>
      <td>Sample prompt used for LLM-Vision.</td>
      <td>6</td>
    </tr>
    <tr>
      <td>3</td>
      <td>LLM-Vision pipeline for table extraction. A page as an image with the prompt in Figure 2 (zero-shot learning) is sent to the LLM-Vision API, then we can collect tables as HTML mark-up.</td>
      <td>6</td>
    </tr>
    <tr>
      <td>4</td>
      <td>DETR architecture, from [5]. DETR uses a conventional CNN backbone to learn a 2D representation of an input image. The model flattens it and supplements it with a positional encoding before passing it into a transformer encoder. A transformer decoder then takes as input a small fixed number of learned positional embeddings, which we call object queries, and additionally attends to the encoder output. We pass each output embedding of the decoder to a shared feed forward network (FFN) that predicts either a detection (class and bounding box) or a "no object" class.</td>
      <td>7</td>
    </tr>
    <tr>
      <td>5</td>
      <td>TATR-extract pipeline for table extraction. TATR-detect takes a page as an image in input and predicts table bounding boxes with confidence score. Post-processing is not shown. Then, the initial image is cropped with bounding box predicted. These cropped tables as images are individually, send into TATR-structure with tokens content and localization from the new input. Then TATR-structure outputs table as HTML mark-up. Finally, we add TATR-detect outputs to obtain the final output of TATR-extract, the end-to-end model.</td>
      <td>10</td>
    </tr>
    <tr>
      <td>6</td>
      <td>The model architecture of Vision Grid Transformer (VGT) with pre-training objectives for the GiT branch, from [7].</td>
      <td>11</td>
    </tr>
    <tr>
      <td>7</td>
      <td>VGT+TATR-structure pipeline for table extraction. VGT takes a page as an image and a grid (obtained through PDF parser, tokenizer and embedding model) as input and predicts table bounding boxes with confidence score. Post-processing is not shown. Then, the initial image is cropped with bounding box previously predicted. These cropped table images are individually sent into TATR-structure with tokens content and localization from the new input. Finally, we add VGT outputs to obtain the final output of VGT+TATR-structure, the end-to-end model.</td>
      <td>13</td>
    </tr>
    <tr>
      <td>8</td>
      <td>Object queries location for a white and black a input image. Above each "query i", we plot the activation map of attention weights from the last transformer decoder layer. This corresponds to visualizing, for each detected object, which part of the image the model was looking at to predict this specific bounding box and class. Under each heatmap, we visualize the output: bounding box with dominant class ("tab" or "no") and its confidence score. Approximately 5 queries are focused on the page bottom, 2 on the top, and 8 in the middle.</td>
      <td>14</td>
    </tr>
    <tr>
      <td>9</td>
      <td>Inference visualization for different image chunk size input. For each prediction, we plot the bounding box (in pink) and confidence score distribution: p<sub>0</sub>, p<sub>1</sub>, p<sub>2</sub>. If the dominant class is "tables" (p<sub>1</sub> or p<sub>2</sub>), the distribution is surrounded by a frame, otherwise ("no object" dominant) it is an ellipse. We notice that reducing the image size (Figures 9b, 9c) allows better prediction: higher confidence score over tables.</td>
      <td>15</td>
    </tr>
    <tr>
      <td>10</td>
      <td>XY cut algorithm project black pixels over the axis X and Y. From these profiles, we can detect gaps and create sub-image.</td>
      <td>15</td>
    </tr>
    <tr>
      <td>11</td>
      <td>XY+TATR-extract pipeline for table extraction. We pre-process a page as an image with the XY-cut algorithm that creates sub-images. TATR-detect takes these image chunks one by one in input and predicts table bounding boxes with confidence score. Only the top 2 predictions (with the highest confidence scores) are kept (not shown in the figure). Then, the initial image is cropped with the predicted bounding box (adding the XY-cut offset). These cropped tables as images are individually sent into TATR-structure with tokens content and localization from the new input. Finally, we add the XY+TATR-detect outputs to obtain the final output of TATR-extract, the end-to-end model.</td>
      <td>16</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>12</td>
      <td>An example presentation table from the PubTables-1M dataset, from [31], along with corresponding ground truth grid cell matrices for different GriTS metrics (Table 4). Each matrix entry corresponds to one grid cell. Cells obtained from originally fused (merged) HTML cells are shaded darker.</td>
      <td>27</td>
    </tr>
    <tr>
      <td>13</td>
      <td>Simple evaluation pipeline. We run end-to-end model inference on page X, and obtain: the predicted tables set P that contains HTML mark-up tables, table confidence scores (equal to 1 for non probabilistic models), and bounding boxes in some cases. X is annotated with ground truth tables Y. We filter out predictions by the confidence threshold θ<sub>c</sub> to get P<sup>+</sup>. Finally, Ŷ<sub>Y</sub> represents end-to-end positive predictions, containing bounding boxes or HTML mark-up tables, depending on the TD based metric.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>14</td>
      <td>Details on the “Evaluation” module from Figure 13. Inputs are (ŷ<sub>Y</sub>, Y). We compute pairwise distance with the matrix C based on a distance function, d<sub>B</sub> denotes the distance function for bounding box-based TD and d<sub>H</sub> for HTML-based. With the Hungarian algorithm, we solve the assignment problem: in the example shown, p &gt; g (three predictions, two ground truth tables), thus there are g matching pairs (σ(j), j) with σ: [1, min(p, g)] → [1, max(p, g)] creating matches. For matching pairs, depending on the Jaccard similarity, the sample is classified as FP or TP. For TP, we can compute TSR metric with the GT found. Predictions without GT are considered FP, and a GT without prediction leads to a FN.</td>
      <td>29</td>
    </tr>
    <tr>
      <td>15</td>
      <td>Results for bounding box-based table detection evaluation according to confidence score threshold θ<sub>c</sub> that defines P<sub>θ<sub>c</sub></sub><sup>+</sup>. For each metric X, there are 3 sub-metrics, X<sub>.5</sub> for IoU threshold at 50 %, and E<sub>0</sub>[X&#773;], E<sub>.5</sub>[X] for expected value over IoU distribution. Models are distinguished by their colors and metric types by their line style.</td>
      <td>31</td>
    </tr>
    <tr>
      <td>16</td>
      <td>Precision-Recall curves for TATR-detect, VGT and XY+TATR-detect, depending on IoU (50%, expected precision).</td>
      <td>33</td>
    </tr>
    <tr>
      <td>17</td>
      <td>Brier scores ↓ computed over P<sub>θ<sub>c</sub></sub>, for each metric type (fixed and expected IoU).</td>
      <td>34</td>
    </tr>
    <tr>
      <td>18</td>
      <td>Results for HTML-based table detection evaluation according to confidence score threshold θ<sub>c</sub> that defines P<sub>θ<sub>c</sub></sub><sup>+</sup>. For each metric X, there are 3 sub-metrics, X<sub>.5</sub> for IoU threshold at 50 %, and E<sub>0</sub>[X&#773;], E<sub>.5</sub>[X] for expected value over IoU distribution. Models are distinguished by their colors and metric types by their line style.</td>
      <td>35</td>
    </tr>
    <tr>
      <td>19</td>
      <td>Results for bounding box-based table structure evaluation according to confidence score threshold θ<sub>c</sub> that defines P<sub>θ<sub>c</sub></sub><sup>+</sup>. Metrics are computed over normalized HTML mark-up tables. GROBID is out of the frame, see Table 9.</td>
      <td>37</td>
    </tr>
    <tr>
      <td>20</td>
      <td>Results for HTML-based table structure evaluation according to confidence score threshold θ<sub>c</sub> that defines P<sub>θ<sub>c</sub></sub><sup>+</sup>. Metrics are computed over normalized HTML mark-up tables. GROBID is out of the frame, see Table 10.</td>
      <td>39</td>
    </tr>
    <tr>
      <td>21</td>
      <td>Confidence histograms (top) and reliability diagrams (bottom) for TATR-detect (left), VGT (center) and XY+TATR-detect (right) on validation dataset.</td>
      <td>40</td>
    </tr>
    <tr>
      <td>22</td>
      <td>HTML-based Precision, Recall, F<sub>1</sub> as a function of bounding box-based ones with θ<sub>J</sub> = 0.5.</td>
      <td>41</td>
    </tr>
    <tr>
      <td>23</td>
      <td>HTML-Jaccard index distribution over bounding box Jaccard index (IoU) distribution. With θ<sub>J</sub> = 0.5, we evaluate HTML-Jaccard classifier with respect to IoU, to obtain a confusion matrix.</td>
      <td>42</td>
    </tr>
    <tr>
      <td>24</td>
      <td>Page containing a table ground truth.</td>
      <td>47</td>
    </tr>
    <tr>
      <td>25</td>
      <td>On the left (Figure 25a), the ground truth table, and on the right (Figure 25b) LLM-Vision's prediction containing errors.</td>
      <td>48</td>
    </tr>
  </tbody>
</table>
List of Tables

<table><tr><td>1</td><td>End-to-end models for table extraction.</td><td>5</td></tr><tr><td>2</td><td>Methods experimentally compared for the Table Detection and/or the Table Structure Recognition tasks.</td><td>17</td></tr><tr><td>3</td><td>Metric used in table detection evaluation overview.</td><td>24</td></tr><tr><td>4</td><td>GriTS metrics characteristics.</td><td>26</td></tr><tr><td>5</td><td>Methods used for subtask evaluation.</td><td>30</td></tr><tr><td>6</td><td>Results for bounding box-based table detection evaluation. For each metric X, there are 3 sub-metrics, X.5 for IoU threshold at 50 %, and E<sub>0</sub>[X], E<sub>.5</sub>[X] for expected value over IoU distribution. For probabilistic models, we use 3 arbitrary confidence score thresholds to define P<sub>θ<sub>c</sub></sub><sup>+</sup>.</td><td>32</td></tr><tr><td>7</td><td>Average Precision for probabilistic models computed as area under the curves from Figure 16.</td><td>33</td></tr><tr><td>8</td><td>Results for HTML-based table detection evaluation. For each metric X, there are 3 sub-metrics, X.5 for HTML-Jaccard index threshold at 50 %, and E<sub>0</sub>[X], E<sub>.5</sub>[X] for expected value over IoU distribution. For probabilistic models, we use 3 arbitrary confidence score thresholds to define P<sub>θ<sub>c</sub></sub><sup>+</sup>.</td><td>34</td></tr><tr><td>9</td><td>Results for bounding box-based table structure recognition evaluation over the set of TP samples P<sub>0.5</sub><sup>++</sup>, with normalized metrics. For probabilistic models, we use 3 arbitrary confidence score thresholds to define P<sub>θ<sub>c</sub></sub><sup>+</sup>.</td><td>36</td></tr><tr><td>10</td><td>Results for HTML-based table structure recognition evaluation over the set of TP samples P<sub>0.5</sub><sup>++</sup>, with normalized metrics. For probabilistic models, we use 3 arbitrary confidence score thresholds to define P<sub>θ<sub>c</sub></sub><sup>+</sup>.</td><td>38</td></tr><tr><td>11</td><td>Files used for TD and TSR evaluation.</td><td>44</td></tr></table>
References

[1] Amihood Amir et al. "Generalized LCS". In: vol. 409. Sept. 2007, pp. 50-61. ISBN: 978-3-540-75529-6. DOI: 10.1007/978-3-540-75530-2_5.

[2] Galal M. Binmakhashen and Sabri A. Mahmoud. "Document Layout Analysis: A Comprehensive Survey". In: *ACM Comput. Surv.* 52.6 (Oct. 2019). ISSN: 0360-0300. DOI: 10.1145/3355610. URL: https://doi.org/10.1145/3355610.

[3] Glenn W Brier. "Verification of forecasts expressed in terms of probability". In: *Monthly weather review* 78.1 (1950), pp. 1-3.

[4] Zhaowei Cai and Nuno Vasconcelos. *Cascade R-CNN: Delving into High Quality Object Detection*. 2017. arXiv: 1712.00726 [cs.CV]. URL: https://arxiv.org/abs/1712.00726.

[5] Nicolas Carion et al. *End-to-End Object Detection with Transformers*. 2020. arXiv: 2005.12872 [cs.CV]. URL: https://arxiv.org/abs/2005.12872.

[6] Bertrand Coüasnon and Aurélie Lemaitre. "Recognition of Tables and Forms". In: *Handbook of Document Image Processing and Recognition* (Jan. 2014). DOI: 10.1007/978-0-85729-859-1_20.

[7] Cheng Da et al. *Vision Grid Transformer for Document Layout Analysis*. 2023. arXiv: 2308.14978 [cs.CV]. URL: https://arxiv.org/abs/2308.14978.

[8] Cheng Da et al. "Vision Grid Transformer for Document Layout Analysis". In: *ICCV*. 2023.

[9] Jacob Devlin et al. *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. 2019. arXiv: 1810.04805 [cs.CL]. URL: https://arxiv.org/abs/1810.04805.

[10] Alexey Dosovitskiy et al. *An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale*. 2021. arXiv: 2010.11929 [cs.CV]. URL: https://arxiv.org/abs/2010.11929.

[11] Mark Everingham et al. "The pascal visual object classes (voc) challenge". In: *International journal of computer vision* 88 (2010), pp. 303-338.

[12] Liangcai Gao et al. "ICDAR 2019 Competition on Table Detection and Recognition (cTDaR)". In: *2019 International Conference on Document Analysis and Recognition (ICDAR)*. 2019, pp. 1510-1515. DOI: 10.1109/ICDAR.2019.00243.

[13] GROBID. https://github.com/kermitt2/grobid. 2008. swh: 1 : dir : dab86b296e3c3216e2241968f0d63b68e8209d3c.

[14] Chuan Guo et al. *On Calibration of Modern Neural Networks*. 2017. arXiv: 1706.04599 [cs.LG]. URL: https://arxiv.org/abs/1706.04599.

[15] Jaekyu Ha, R.M. Haralick, and I.T. Phillips. "Recursive X-Y cut using bounding boxes of connected components". In: *Proceedings of 3rd International Conference on Document Analysis and Recognition*. Vol. 2. 1995, 952-955 vol.2. DOI: 10.1109/ICDAR.1995.602059.

[16] H. W. Kuhn. "The Hungarian method for the assignment problem". In: *Naval Research Logistics Quarterly* 2.1-2 (1955), pp. 83-97. DOI: https://doi.org/10.1002/nav.3800020109. eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/nav.3800020109. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/nav.3800020109.

[17] Fabian Küppers et al. "Confidence Calibration for Object Detection and Segmentation". In: *Deep Neural Networks and Data for Automated Driving*. Springer International Publishing, 2022, pp. 225-250. ISBN: 978301012334. DOI: 10.1007/978-3-031-01233-4_8. URL: http://dx.doi.org/10.1007/978-3-031-01233-4_8.

[18] Vladimir I. Levenshtein. "Binary codes capable of correcting deletions, insertions, and reversals". In: *Soviet physics. Doklady* 10 (1965), pp. 707-710. URL: https://api.semanticscholar.org/CorpusID:60827152.

[19] Junlong Li et al. *DiT: Self-supervised Pre-training for Document Image Transformer*. 2022. arXiv: 2203.02378 [cs.CV]. URL: https://arxiv.org/abs/2203.02378.

[20] Minghao Li et al. "TableBank: Table Benchmark for Image-based Table Detection and Recognition". eng. In: *Proceedings of the Twelfth Language Resources and Evaluation Conference*. Ed. by Nicoletta Calzolari et al. Marseille, France: European Language Resources Association, May 2020, pp. 1918-1925. ISBN: 979-10-95546-34-4. URL: https://aclanthology.org/2020.lrec-1.236/.
[21] Chin-Yew Lin. "ROUGE: A Package for Automatic Evaluation of Summaries". In: *Text Summarization Branches Out*. Barcelona, Spain: Association for Computational Linguistics, July 2004, pp. 74-81. URL: https://aclanthology.org/W04-1013/.

[22] Tsung-Yi Lin et al. *Feature Pyramid Networks for Object Detection*. 2017. arXiv: 1612.03144 [cs.CV]. URL: https://arxiv.org/abs/1612.03144.

[23] Tsung-Yi Lin et al. "Microsoft COCO: Common Objects in Context". In: *CoRR* abs/1405.0312 (2014). arXiv: 1405.0312. URL: http://arxiv.org/abs/1405.0312.

[24] Chixiang Ma et al. "Robust Table Detection and Structure Recognition from Heterogeneous Document Images". In: *Pattern Recognition* 133 (Jan. 2023), p. 109006. ISSN: 0031-3203. DOI: 10.1016/j.patcog.2022.109006. URL: http://dx.doi.org/10.1016/j.patcog.2022.109006.

[25] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. "Obtaining well calibrated probabilities using bayesian binning". In: *Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence*. AAAI'15. Austin, Texas: AAAI Press, 2015, pp. 2901-2907. ISBN: 0262511290.

[26] Alexandru Niculescu-Mizil and Rich Caruana. "Predicting good probabilities with supervised learning". In: *Proceedings of the 22nd International Conference on Machine Learning*. ICML '05. Bonn, Germany: Association for Computing Machinery, 2005, pp. 625-632. ISBN: 1595931805. DOI: 10.1145/1102351.1102430. URL: https://doi.org/10.1145/1102351.1102430.

[27] Kishore Papineni et al. "Bleu: a Method for Automatic Evaluation of Machine Translation". In: *Annual Meeting of the Association for Computational Linguistics*. 2002. URL: https://api.semanticscholar.org/CorpusID:11080756.

[28] Mateusz Pawlik and Nikolaus Augsten. "Tree edit distance: Robust and memory-efficient". In: *Information Systems* 56 (2016), pp. 157-173. ISSN: 0306-4379. DOI: https://doi.org/10.1016/j.is.2015.08.004. URL: https://www.sciencedirect.com/science/article/pii/S0306437915001611.

[29] F. Pedregosa et al. "Scikit-learn: Average precision score". In: *Journal of Machine Learning Research* 12 (2011), pp. 2825-2830. URL: https://scikit-learn.org/1.5/modules/generated/sklearn.metrics.Average_precision_score.html.

[30] Joseph Redmon et al. *You Only Look Once: Unified, Real-Time Object Detection*. 2016. arXiv: 1506.02640 [cs.CV]. URL: https://arxiv.org/abs/1506.02640.

[31] Brandon Smock, Rohith Pesala, and Robin Abraham. *GriTS: Grid table similarity metric for table structure recognition*. 2023. arXiv: 2203.12555 [cs.LG]. URL: https://arxiv.org/abs/2203.12555.

[32] Brandon Smock, Rohith Pesala, and Robin Abraham. "PubTables-1M: Towards Comprehensive Table Extraction From Unstructured Documents". In: *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. June 2022, pp. 4634-4642.

[33] Yuxin Wu et al. *Detectron2*. https://github.com/facebookresearch/detectron2. 2019.

[34] Yiheng Xu et al. *LayoutLM: Pre-training of Text and Layout for Document Image Understanding*. 2019. arXiv: 1912.13318 [cs.CL].

[35] Peng Zhang et al. *VSR: A Unified Framework for Document Layout Analysis combining Vision, Semantics and Relations*. 2021. arXiv: 2105.06220 [cs.CV]. URL: https://arxiv.org/abs/2105.06220.

[36] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. *Image-based table recognition: data, model, and evaluation*. 2020. arXiv: 1911.10683 [cs.CV]. URL: https://arxiv.org/abs/1911.10683.

[37] Xu Zhong, Jianbin Tang, and Antonio Jimeno Yepes. *PubLayNet: largest dataset ever for document layout analysis*. 2019. arXiv: 1908.07836 [cs.CL]. URL: https://arxiv.org/abs/1908.07836.