Table2LaTeX-RL: High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models
Authors: Jun Ling
Yao Qi
Tao Huang
Shibo Zhou
Yanqin Huang
Yang Jiang
Ziqi Song
Ying Zhou
Yang Yang
Heng Tao Shen
Peng Wang
Venue: NeurIPS 2025 poster ()
Keywords: table recognition
latex generation
Primary area: applications

Abstract: In this work, we address the task of table image to LaTeX code generation, with the goal of automating the reconstruction of high-quality, publication-ready tables from visual inputs. A central challenge of this task lies in accurately handling complex tables—those with large sizes, deeply nested structures, and semantically rich or irregular cell content—where existing methods often fail. We begin with a comprehensive analysis, identifying key challenges and highlighting the limitations of current evaluation protocols. To overcome these issues, we propose a reinforced multimodal large language model (MLLM) framework, where a pre-trained MLLM is fine-tuned on a large-scale table-to-LaTeX dataset. To further improve generation quality, we introduce a dual-reward reinforcement learning strategy based on Group Relative Policy Optimization (GRPO). Unlike standard approaches that optimize purely over text outputs, our method incorporates both a structure-level reward on LaTeX code and a visual fidelity reward computed from rendered outputs, enabling direct optimization of the visual output quality.
We adopt a hybrid evaluation protocol combining TEDS-Structure and CW-SSIM, and show that our method achieves state-of-the-art performance, particularly on structurally complex tables, demonstrating the effectiveness and robustness of our approach.

Table2LaTex-RL. High-Fidelity LaTeX Code Generation from Table Images via Reinforced Multimodal Language Models
Jun Ling' cs driinnjn@ggmiil. com
Yao QF
Tao Huang!
Shibo Zhou?
tachuang0313@gmill com
Yanqin Huang²
Jiang Yang²
Ziqi Song²
Ying Zhou²
Yang Yang!
Heng Tao S.en 1.3
Peng Wang
p w.wango@hotmiil.comm
School of Computer Science and Engineering, University of Electronic Science and Technology of China Research Center for Scientific Data Hub, Zhejiang Lab, Hangzhou, China School of Computer Science and Technology, Tongji University
Abstract
In this work, we address the task of table image LaTeX code generation, with the goal of automating the reconstruction of high-quality, publication-ready tables from visual inputs. A central challenge of this task lics accurately handling complex lables-those with large sizes, deeply nested structures, and semantically rich or irregular cell content-wher existing methods often fail. We begin with a comprehensive analysis, identifying key challenges and highlighting the limitations current evaluation protocols. To overcome these issues, we propose a reinforced multimodal large language model (MLLM) framework, where pre-trained MLLM is fine-tuned on a large-scale table-to-ILai LoTeee dataset. To further improve generation quality, we introduce dual-reward reinforcement learning strategy based Group Relative Policy Optimization (GRPO). Unlike standard approaches that optimize purely over text outputs, our method incorporates both structure-level reward on LaTeX code and a visual fidelity reward computed from rendered outputs, enabling direct optimization of the visual output quality. We adopt hybrid evaluation protocol combining achieves state-of state-of-the- of-the-art performance, particularly on structurally complex tables, demonstrating the effectiveness and robustness of our approach. Code and dataset are available at http:://ggibb...oommm.iiiiiiniee
SEDS-Structure and CW-SSIM, and show that our method
1 Introduction
Tables are essential components of scientific and technical documents, providing a structured and concise format for presenting quantitative data, experimental results, and complex relationships. As document digitization becomes increasingly prevalent, the ability to automatically generate table code from images is critical for enabling content reuse and high-quality reproduction. However, most existing methods focus on generating HTML representations (1H55. which lack the structural
Corresponniig author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

expressiveness and typographic precision required for complex tables essecially those with nested headers, merged cells, or mathematical content. contrast, LaTeX the standard in scientific publishing, offering the flexibility and fidelity needed for professional-gaaee tables. Despite its practical importance, the task of directly generating LaTTeX code from table images has received limited attention in prior work [6 7.
In this work, we study the task of table image to LaTeX generation and provide a comprehensive analysis of its challenges. Through empirical observations, we find that the primary difficulty lies in handling complex tables, which are often large, deeply nested, and semantically rich structures naturally suited to LaTeX but difficult for models to predict accurately. These challenges affect both the vision encoder, which must extract fine- ane-grained visual and structural cues, and the language decoder, which must generate long, -ynax-sensitive LaTeX sequences. Errors in either stage often lead to hallucinated, malformed output or even compilation errors. To enable finer-grained evaluation and better understand the current research gaps, we propose splitting the dataset into simple, medium, and complex subsets based on structural complexity.
To tackle these challenges, we leverage pre-trained multimodal large language models (MLLMs), which demonstrate strong capabilities in visual recognition, cross-modal reasoning, and LaTeX fluency. We fine-tune an MLLM on a large-scale image-to-LaTex dataset harvested from scientific documents on arXiv. To further improve performance particulartyy for complex tables-wa we intro- duce dual-reward reinforcement learning strategy built on Group Relative Policy Optimization (GRPO) 8 termed VSGRPO. While standard GRPO methods optimize text generation quality based solely on textual output, we go step further: we render the generated LaTee code into images and directly evaluate visual fidelity using CW-SSIM. This image-based reward complements a structure- level reward computed from the LaTeX source, allowing us to jointly optimize for both structural accuracy and rendered appearance. This novel risual-in-the-loo reinforcement design significantly enhances the model's ability to produce faithful, high-fidelity LaTeX code for structurally rich and visually complex tables.
From evaluation perspective, existing metrics are limited. TEDS [9] a widely used structure-based metric, lacks sensitivity to fine-grat grained errors and suffers from mismatches between HTML and LaTeX. On the other hand, rendered image comparison metrics focus on local visual similarity but ignore global structural correctness. To overcome this, we adopt a hybrid evaluation strategy that combines TEDS-Strrctur [10] for structural fidelity and CW-SSIM for robust visual similarity.
Under this framework, our method achieves state-of-the -rrt performance on the table image to LaTeX generation task, with particularly strong improvements on complex tables. This demonstrates the effectiveness of combining MLLM fine-tuning with targeted reinforcement learning for high-fidelity, publication-ready table generation.
We delve deep into the under-explored task of table image to LaTeX code generation, offering a comprehensive analysis of its core challenges particulartty for structurally complex tables-ane introducing complexity-basee data split for fine- grained evaluation.
We develop reinforced MLLM framework, where pre-trained MLLM is fine-tuned on a large-scale image-to LaTTee dataset harvested from arXiv, effectively bridging visual input and LaTeX code generation.
We propose VSGRPO, a novel dual-reward reinforcement learning strategy based on GRPO, which jointly optimizes structure-level accuracy and visual fidelity by incorporating both LaTeX-baased and rendered-imagg-basee feedback.
We introduce hybrid evaluation strategy combining better reflect both structural and visual correctness. Extensive experiments demonstrate state-of-the-an performance of our approach, especially on complex tables.
and CW-SSIM to
2
Related Work
Table Structure Recognition. Existing table recognition approaches fall into two main categories: detection-based and image-to-text-based methods. Detection-based methods first predict the physical structure- -ucc as grid lines or cell bounding boxes-and -nd then infer logical relationships. Grid-line- based approaches [1H17] segment tables along detected rows and columns and merge regions to

---

reconstruct cells. Cell- boll-bonnding--o box methods [8 21 treat detected cells as graph nodes, using GNNs to infer row/column associations.
Image-to-fext-basee table structure recognition (TSR) decomposes the task into predicting struc- tural layout and transcribing cell content, which are then fused into full table representation. Encoder-decoder models [1-3] generate structure tokens (e.g., HTML tags) and content separately. TableFormer improves this with Transformer-basee decoding and regression for bounding boxes.
VAST 4 frames coordinate prediction as sequence generation and adds a visual alignment loss.
DRCC
adopts hybrid decoding scheme reduce error accumulation.
Most detection and TSR methods target HTML outputs, which are not well-suited for LaTeX due to syntactic and semantic differences. Recently, end-to end-to-end LaTeX generation approaches have emerged. LaTeXNet 7 uses specialized submodules for equations, tables, and text. Nougat [22] bypasses OCR entirely generate LaTTX directly. LATTE [6] introduces iterative refinement via localization and correction models. However, these methods do not explicitly address the combined challenges of large-scale layout and deeply nested LaTeX structures.
Multimodal Large Language Models with Reinforced Fine- Tine-Tuning Pre-trained multimodal large language models (MLLMs) learn joint visual-text representations from large-scale image-text corpora, equipping them with strong capabilities in visual understanding and LaTeX code generation. While recent works such as Nougat [22] and LATTE [6] employ multimodal architectures, they largely underutilize pre-trained priors, relying instead on scom-scratch training.
To further improve performance, especially for complex tables, we apply reinforced fine-tuning using the Group Relative Policy Optimization (GRPO) framework [8]. Compared to earlier reinforcement methods such as RLHF (3) and DPO GRPO eliminates the need for value network and uses correctness-base rewards to guide learning with reduced computational overhead. Unlike prior works that apply reinforcement learning purely in the text domain [25] 26]. our method designs task-specific reward signals: we compile the generated LaTeX into HTML for TEDSSStructure evaluation and into images for CW-SSIM computation, enabling joint optimization of both structural accuracy and rendered visual fidelity. This visual -n- -it--ii--tt-ooooo thop RL approach is particularly effective for high-fidelity LaTeX generation complex table structures.
3 Insight into the Task
We provide key insights into the task of table-to-Lalle generation, focusing on two critical aspects: the challenge of handling complex tables and the limitations of current evaluation protocols.
One of the central challenges in this task lies in accurately processing complex tables, which serve as a meaningful indicator of a model's true capability. Complex tables are prevalent in modern documents, often used to convey large volumes of structured information compactly. However, their intricate layouts. large dimensions, and diverse content introduce significant difficulties for vision encoders-Leading to higher computational costs, reduced performance, and increased inference latency. Despite their importance, complex tables are often underrepresented or overlooked in evaluation. To address this, we propose categorizing tables into three complexity levels levels-simple medium, and complex- to enable a more realistic and fine- mme-grained evaluation of model performance.
In addition to data-level challenges, the evaluation of LaTeX code generation remains underdeveloped. as shown in Appendix A Existing metrics generally fall into two categories: Text-based metries, such as TEDS 9 and BLEU 27 compare the predicted and ground-truth LaTeX code at the token level. However, they fail to account for the inherent syntactic ambiguity of LaTeeX and ignore structural semantics, often penalizing semantically equivalent but syntactically different outputs. Visual-based metrics, such as CW-SSIM (28). evaluate similarity between rendered images of the generated and ground-truth tables. While useful for natural scenes, standard CW-SSIM less effective on binary, high-contrast table images, where sharp edges and sparse textures dominate. Alternative pixel-Ievel metrics, such as Edit (column- -wiee normalized edit distance) and Match (binary pixel-wise agreement), also fall short in capturing higher-Ievel structural or semantic similarity. To address these limitations, we adopt modified version of CW-SSIM, tuned for binary table images with high visual sparsity, to better assess rendering fidelity. However, since CW-SSIM primarily focuses on local visual similarity, we complement with structure and layout alignment.
100 evaluate the global
3

---

4 Method
4.1 Large-Scale Table2LaTeX Collection
Due to the lack of publicly available large-scale datasets containing LaTeX table code, we propose dataset construction pipeline. Specifically, we develop a web crawler to scrape the LaTeX source files of scientific papers from the open-access arXiv repository. We use regular expressions to extract LaTeX code corresponding to table environments. To ensure data quality, we further clean the extracted code by removing references, color settings, and other LaTeX control commands. Through this process, we collect a dataset comprising 1,209,986 table-LaTeX pairs. To classify table complexity, tables with 2 or more Amultirow or \multicol umn commands and 100-160 cells are defined as medium tables, while those with over 160 cells are labeled complex tables. All others considered simple. Within the training set, simple tables account for approximately 94%, while medium and complex tables each represent about 3% of the data.
4.2 Supervised Fine- TunnnT
To enable general multimodal large language models (MLLMs) to acquire preliminary capability for handling the task of table-to-LaTeX generation, we design a second-stage supervised fine-tuning (SFT) process. During this process, perform SFT using data collected in stage 1. The imput consists of a table image and the prompt: "Convert this table to LaTeX", while the ground- truth LaTeX code serves the response. Thus, our dataset is structured as input-response pairs, formally expressed as: DD = {(x(9, yy } where each x is an input and y the corresponding target response. During training optimize 0 to maximize the conditional likelihood of (())) given x(0), Equivalently, we minimize the negative log-likelihood over the dataset:
CSFT (0)
Σ
log (y 1x()).
However, as shown in Table 4 SFT alone is insufficient to fully unlock the model's potential. A key limitation stems from the widespread use of teacher foreing, where the model trained to predict the next token given the prefix. Yet, LaTeX code is inherently ambiguous- control sequences) may yield identical visual outputs. This mismatch between training supervision and evaluation objectives leads to inefficient generalization, particularly for structurally complex tables.
syntactic forms (e.g..
4.3 Reinforeed Fine- Sane-Tunigg via VSGRPO
As analyzed above, the next-token prediction paradigm used in SFT is limited in its ability model the semantic structure and syntactic dependencies embedded in long LaTeX sequences. Moreover, the SFT objective focuses solely on text-level alignment and completely ignores the visual similarity between the rendered LaTeX output and the original table image-despire despite visual appearance being direct and critical indicator of generation quality. However, since LaTeX rendering is non- differentiable operation, it cannot be directly incorporated into gradient-based supervised training.
To address these limitations, we propose a novel reinforced fine-tuning framework that introduces rendered image feedback as an explicit optimization signal. Drawing inspiration from Group Relative Policy Optimization (GRPO) [8], we extend its scope beyond standard textual quality assessment and design a dual-reward mechanism that jointly promotes structural accuracy and visual fidelity. While conventional GRPO-based methods focus solely improving text generation quality, our framework leverages both the LaTeX code structure and its rendered appearance, offering more task- task-aligned supervision signal, as shown in Figure D We select 5,936 complex tables from the training dataset as the training set for VSGRPO, whose ground LaTeX code contains fewer than 3,000 characters to balance complexity and computational feasibility.
Visual Reward. We compile and convert the set of predicted table LaTeX generated by the model from single table image input-by by embedding them into LaTeX fragments with standard macro packages, producing a group of predicted table images. At the same time, we compile the ground-trute -ttut LaTeX code to obtain the ground-truth table image. If the compilation fails, the

---

Table Image
Ground Truth LaTeX
\begin(tabular,
Hellaswng
MMLU
bbegintaabular/clloccc Vtoprule
PGR
\multirow22 SiRightarrow
strong) & &
nalve
0.444
0.387
smulicolumm22/eeHeellasswwg &
Llamaz-7B Liama2-13B
w2s+fllter.f [s.]
0.803
-0.031
0.646
0.398
(multicooumm/2)c(MMMML))) W
w2
0.827
0-485
0.500
0.915
strong cailing
0.855
1,000
0.509
1.000
tendttabulari
Group of generated LaTeX
Reward Function
Rewards
Group Advantage
Convert this table to LaTeX.
LaTeX
Visual Reward
A
LaTeX
Structure
MLLM
LaTeX
Reward
N
(1) Visual Reward
(2)
Structure Reward
Ground Truth Image
Prediction Table Image
Ground Truth
Prediction
Local Patch
Local Patch
MMLUU
accuracy
PGR
Accuracy
PGR
0,404
0.000
0.404
0.387
0.446
0.387
CW-SSIM
TEDS-Structure
Figure 1: Demonstration of our proposed VSGRPO framework for table image to LaTeX code generation. The top section shows an example table image alongside its corresponding LaTeX code, representing the input-output pair used in training. The middle section illustrates the workflow of the VSGRPO framework. The bottom section highlights the dual-reward mechanism: a visual fidelity reward computed via CW-SSIM between the model-generated and ground-truth rendered images, and structure-level reward based on TEDS-Structur computed from the table's structural elements.
corresponding reward is set O. We then compute the CW-SSIM between the ground-truth table image and each predicted table image. the CW-SSIM of predicted image exceeds the predefined threshold, it receives reward of I; otherwise, the reward is O.
To accommodate black-and-wil white table images, we adopt the following CW-SSIM calculation process: the CW-SSIM algorithm preprocesses two table images by converting them to grayscale, resizing them to uniform dimensions, and aligning their rows and columns. It then divides the images into 2x2 pixel blocks and applies simplified Haar wavelet transform [29 decompose each block into four sub-bands: cA wow-freeuency approximation), cH (horizontal), cV (vertical), and cD (diagonal high-free frgh-freeuency details). For each sub-band, the algorithm calculates SSIM metrics optimized for monochrome tables, incorporating pixel-level means, variances, covariance, and stabilizing constants C_ and C_ Finally, averages the SSIM scores from all four sub-bands to generate the comprehensive CW-SSIM metric.
Structure Reward. We convert both the predicted table LaTeX code generated by the model and the cround-truth LaTeX into HTML in order to compute their TEDS-Structure the HTML conversion of predicted table LaTeX fails, the reward is set O. For successfully converted predictions, if the TEDS- SEDDS-Structure similarity with the ground -trut exceeds a predefined threshold, the reward is set to otherwise, it is O.
TEDS-Strueee computes the Minimum Tree Edit Distance between the two trees by applying unit-cost insertions, deletions, and structural-nods substitutions to transform the predicted tree into ground-truth tree. then normalizes this distance by the larger the two tree sizes and converts it into similarity score.
5

---

During the RFT training process, we samples a group of generated output set {or, 02, ON- for each table image q from policy model Tam.. Then RFT maximizes the following objective and optimizes the model no. The specific formula is shown below:
(O) Pqpp((Q) (o.) (Toe (Olg)
min (o 11) Te (or ())
As. clip To(o; g) To (or g)
- - =)) (TL
Tret)
where and B denote the clipping threshold in Proximal Policy Optimization (PPO) and the coefficient controlling the Kullback-Leibler (KL) divergence penalty term, respectively 30 31. set & 0.2 and B 0.02 during training.
The advantage for the i-th sample computed as
r; meean (ri, (..,,,,.....N .N..). , ,N)))
A;
where r2... ........ .. denotes the set of group rewards. The KL divergence between the current policy and the reference policy Tref for the observation-action pair (g, oi) is defined as
log DKL (To Tret) = Teet (0; (9) Tref (o, 19) 11
(4)
To(o; 4)) (o(0; ④)
Experiments
In this section, we present our experimental results. Specifically, Section 5.1 details the datasets,
implementation settings, and evaluation metries used our study. Section 5.2 reports quantitative comparisons against state-of-the- art baselines. To further assess whether the generated table images align with human perception, we conduct human evaluation, presented in Section 5.3 Finally, Section 5.4 provides ablation analysis to highlight the key components and contributions of our proposed method.
5.1 Experimental Setup
In this section, we first describe the detailed construction process and composition of the training and testing datasets. We then present the implementation details for both the SFT and reinforced fine-tuning (RFT) phases.
Training Dataset. We collect LaTeX source code from arXiv papers published between October 2017 and April 2023, extracting total of 1,209,986 table entries using regular expression matches between (begin(tabular) and \end(tabular), After filtering out references, color commands, and other non-structural LaTeX elements, we classify the tables into three categories. This full dataset used for SFT.
Testing Dataset, We construct the testing dataset using the same processing pipeline as the training Specifically, we crawl 101,469 LaTeXX table entries from arXiv papers published between January and November 2024, covering a diverse range of scientific domains. From this pool, we randomly sample 496 simple, 354 medium, and 361 complex tables form the final testing
Implementation Details. We adopt full-parameter fine-tuning for all training phases. During SFT, all models are trained for epoch with a maximum output length of 4096 tokens. For Nougat 22] training is conducted on nodes (each with 8xA100 GPUs) using a batch size of Intern is trained 4 nodes with batch size of 4 and gradient accumulation steps set to 2. Qwen2.5- VL-3B 32] is trained on 2 nodes, also with a batch size of and gradient accumulation steps of 2.
For reinforced fine-tuning (RFT), Intern VI.2-B adopts the VLM-RI framework 33 and trained on nodes with 8 sampled generations per input (num__ees gens 8), batch size of 8, and gradient

---

accumulation steps of 2. Qwen2.5-VL-3B uses the mw-ftift infrastructure [34], trained on 2 nodes with 4 generations per input (num mnum_ 4), batch size of and the same gradient step setting. The reward thresholds are set to 0.6 for CW-SSIM and 0.9 for StDDS-Structue.
During testing, we maximum output length of 8192 tokens, batch size of 1, and temperature 0f All testing is conducted within mexlive-full Docker environment to ensure LaTeX rendering and TEDS for performance evaluation. The scores range from 0 to I, and the exact formulas are shown in Appendix B
fidelity. For metrics, we use Python-based implementations of CW-SSIM,
5.2 Main Results
We compare VSGRPO with various solutions across different categories. In the commercial and paid domain, we evaluate it against the most powerful system to date, Mathpix [35. To compare with current general-purge pprpose multimodal large models, we include the closed-source GPT-4o [36,, as well as the open-source Qwen2.5- V-L---LL- (37] and -eeem2.5-VL-88- (8). For specialized expert models, we compare against Nougat 22 state-of-the-ard open-source LaTeX generation system.
To more accurately evaluate the correctness of LaTeX generation, we
model performance from two complementary perspectives: rendered image quality and LaTeX source fidelity. First, we evaluate the visual accuracy of the generated LaTeX by compiling it into table images. Two metrics are used: the compile ratio, which reflects the proportion of LaTeX outputs that can be successfully compiled using standard LaTeX packages, and CW-SSIM, which quantifies the visual similarity between the rendered output and the ground-truti image. These results are reported in Table [ Second, we assess the semantic and structural correctness of the LaTeX source code itself. To this end, we compute TEDS-Structure Structure, which measures cell-level structural alignment, and TEDS, which additionally considers the tabular content. These metries provide a deeper view into how well the generated code captures the underlying table semantics, and are summarized in Table To further evaluate the generalization ability of our method, we additionally test it on an external benchmark dataset introduced in (6), with the results shown in Table 3
Table T shows that the CW-SSIM values of all models exhibit decreasing trend as table complexity increases (from simple to complex). However, the proposed VSGRPO method achieves comprehen- sive improvements across two model families. Specifically, Intern2- -L--- VSGRPO sets new record on the simple tables with CW-SSIM of 0.8201, surpassing the previous best model by 0.049, Meanwhile, Qwen2.5-VL-3B. W...5-VLL-BB-GGGRO significantly outperforms baselines on the medium tables and complex tables, achieving CW-SSIM scores of 0.7236 (+0.1113 1113 and 0.6145 (+0.0903), respectively. Furthermore, it attains compile success rate of 0.9917 on the complex tables, exceeding Mathpix's 0.9889. These results demonstrate that the proposed VSGRPO strategy effectively enhances the robustness complex tables reconstruction while maintaining high LaTeX compilability through visual- and structure- -uiiuee optimization.
Table 1: Model performance on CW-SSIM and compile ratio across three table complexity levels.
Simple
Medium
Complex
Models
CW-SSIM
Compile ratio
CW-SSIM
Compile ratio
CW-SSIM
Compilc ratio
Commercial Tools
Mathpix 35
0.6884
1.0000
0.5647
0.9943
0,4862
0.9889
General VLMs
GPT4o 36
0.6792
0.9918
0.5612
0,9972
0.4747
0.9917
Qwen2.5-VL-72.
37
0.7077
0,9858
0.6009
0,9887
0,5112
0.9335
Intern2.5-VL. 38
0.7814
0.9959
0.6123
0.9773
0.5242
0.4515
Expert VLMs
Nougat 22]
0.7401
0.7617
0,5505
0.1813
0.4699
0.3352
Our Resuls
Intern2- -L--IB-SG--POO
0.8201
0.9939
0.7185
0.9830
0.5899
0.9640
Qwen2.5-VL-3B. VSGRRP
0.8186
0,9980
0.7236
0.9943
0.6145
0.9917
Table 2 presents results for TEDS and TEDS-Sttucttne metrics. The trend of TEDS scores largely mirrors that of TEDSSStructure although the absolute values are consistently lower due to TEDS additionally accounting for cell content alignment. The commercial tool Mathpix demonstrates relatively stable performance across table types, achieving its highest TEDS-Structes score
7

---

medium medium-complexity tables (0.8965). In the general-purpor purpose VLM category, Qwen2.5-VL-72B shows consistently strong structural performance, with the highest TEDS Structure score on simple tables (0.9400). However, it exhibits gradual performance decline as complexity mncrasessTTEDSS drops from 0.8720 (simple) to 0.8090 (medium) and 0.7448 (complex). By contrast, other large-scale experience a sharp drop complex tables (TEDS: 0.3379), and
models such as Intern2.5-VL- the expert model Nougat collapses almost entirely (TEDS: 0.0424), revealing severe limitations in proposed Qwen2.5-VL-3BD --VRRPO
both structural and content-level generalization. In contrast, achieves consistently superior results across all levels of table complexity. Despite its compact size (3B parameters), it outperforms significantly larger models, reaching TEDS score of 0.8673 on complex tables 0.12 higher than the next-best model- and achieving a TEDS-Structure score of 0.9218, the first surpass the 0.9 threshold complex tables. These results underscore the effectiveness of our dual-reward optimization strategy, which integrates structural and visual supervision to enable robust, high-fidelity LaTex code generation, especially for structurally rich and visually complex tables.
Table 2: Performance of different models on TEDS and TEDS-Structurr across three table complexity levels.
Simple
Medium
Complex
Models
TEDS
TEDS SIDSS-Stucturr
TEDS
TEDS- STDSS-Structure
TEDS
TEDSS-Stucture
Commercial Tools
Mathpix
0.7804
0.8701
0.8044
0.8965
0.7176
0,8100
General VLMs
GPT4o 36
0.8259
0.9117
0.6986
0,8451
0.5865
0.7745
Swen2.5-VL-722
0.8720
0,9400
0.8090
0,8920
0.7448
0.8334
Intern2.5-VI--785
0.8368
0.8795
0.7123
0,7652
0.3379
0.3735
Expert VLMSs
Nougat 22
0,3856
0.4308
0.1193
0.1357
0.0424
0.0527
Our Results
DIB----VLIB--SSKRPO
VSGRPO
0.8959
0.9358
0.8604
0.8988
0.8054
0.8625
Qwen2.5-VL-3B- VSGRPO
0.8997
0.9405
0,9004
0.9427
0.8673
0.9218
To evaluate the generalization capability of our method, we conduct additional experiments on an external benchmark dataset introduced in (6). The results are presented in Table Manual inspection reveals that this dataset is primarily composed of simple tables with limited structural complexity. Consequently, the performance trends largely mirror those observed in the simple-table subsets reported in Table Jnd Table
Once again, our method, Qwen2.5-Ve Dwwen2.5-VL-3B--1 "L-33--SSGPPG achieves superior performance, outperforming both task task-specific baselines for table image LaTeX generation and large language models. These results underscore the model's 's strong generalization capability.
multimodal
Table 3: Experimental comparison on external dataset 66 on CW-SSIM and TEDS-Structun
Models
CW-SSIM TEDS- SrDS-Structure
LATTE
0
0.7615
0.9445
GPT4e
36
0.6897
0.8568
Qwen2.5-VL-722
0.7176
0.8915
Intern2.5-VI-78B
3%
0.7696
0.9009
Owen2.5-VI-333. VLGRROO
0,8225
0.9461
5.3 Human Evaluation
To complement automated metrics and better capture perceived visual quality, we conduct a human preference study on 200 randomly selected tables (50 simple, 50 medium, 100 complex), as shown in Appendix C For each case, rendered outputs from four models are displayed anonymously alongside the ground truth image. Multiple human assessors independently vote on the most visually similar result, and the final decision is determined by majority voting. As shown in Table Qwen2.5-VIL 1B-VSGRPO receives the highest number of votes across all difficulty levels, clearly outperforming other models in terms of visual and structural fidelity.
"The LATTE model proposed in 6 is not publicly available. We compute results based on the authors' released outputs and apply our own metric calculations.

---

Table 4: Results of human evaluation.
Models
Simple
Medium
Complex
GPT4o
5
2
2
Mathpix
19
2
10
Qwen2.5-VI-3B-SFT
29
28
56
Wwen255-LL3B-VSSRRPP
42
37
70
5.4 Ablation Study
To validate the effectiveness and robustness of our proposed method, we conduct a series of ablation studies focusing on three aspects: the impact of data selection strategies, the contribution of individual reward components, and the necessity of staged training. All experiments are evaluated on the complex table subset.
Evaluation the Dataset Selection Strategy for VSGRPO. As shown in Table compare different strategies for constructing the reinforcement learning (RL) training set. Specifically. we evaluate three variants of Qwen2.5-VL-3B fine-tuned with VSGRPO: (1) using only simple tables (-Simple), (2) using balanced mixture of simple, medium, and complex tables (-Mixed-Datal and (3) using only complex tables (-VSGRPO). The results clearly demonstrate that restricting the RL fine-tuning data to complex tables leads to the best overall performance across all metrics. This validates our design choice of focusing on structurally difficult examples during reinforcement leamning to better generalize across complexity levels.
Table 5: Ablation experiments the dataset selection for VSGRPO.
Models
CW-SSIM
Compile ratio
TEDS
TEDS-Simcture
Qwen2.5.VV. VL--BB-VSRRPO Simple
0.5993
0.8614
0.9113
Qwen2.5-VLL B.-----SSGPO-MME
0.6107
0.9861
0.8614
0.9136
Qwen2.5.VL.n VL-3B VSGRPO
0.6145
0.9917
0.8673
0.9218
Evaluation on the Reward Design in VSGRPO. Table Table presents the effectiveness of the two reward components used in our RL Simnewrk-TEDDS TEDS--ttrctuurrr Structure and CW-SSIM. Adding either reward individually the base model leads noticeable performance gains over the SFT-only baseline, demonstrating that both structure-level accuracy and visual similarity are important for improving LaTeX generation quality. The best performance is achieved when both reward signals are combined, suggesting they are complementary in guiding the model toward faithful and well-aligned outputs.
Table 6: Ablation experiments on the reward design.
Models
CW-SSIM
Compile ratio
TEDS
TEDS-Structure
0.5806
0.9889
0.8481
(1,9147
Qwen2.5-VV
W-W.VLL--GPPO-TTTT Structure
TEDS-Structure
0.5925
0.9889
0.8608
0.9155
0.6064
0.9889
0.8607
0.9133
0.6145
0.9917
0.8673
0.9218
Evaluation on the Necessicity of SFT. To verify the necessity of SFT before reinforcement fine- tuning, we perform one epoch of reinforcement learning directly on the pre-trained Qwen2.5-VL. model (VSGRPO without SFT). As shown in Table the performance of the model without SFT initialization is significantly lower across all metrics, indicating that SFT is essential to provide reasonable starting point for subsequent RL-based optimization.
Conclusion and Limitations
Our work tackled the challenge of converting table images into syntactically correct, publication- quality LaTeXX code by integrating vision-language modeling with targeted reinforcement learning. We leveraged pre-trained multimodal large language model (MLLM)), fine-tuned it on diverse corpus of scientific table images, and further enhanced it through a dual-reward scheme: one reward evaluated structural integrity using
while the other measured visual fidelity via refined CW-SSIM on rendered outputs. By jointly optimizing these objectives, the model was able to accurately capture complex table layouts- including nested headers, merged cells, and mathematical expressions-ait produce outputs that closely matched the original visual appearance.

---

Table 7: Ablation experiments the effectiveness of SFT.
Models
CW-SSIM
Compile ratio
TEDS
TEDS-Structure
Qwen2.5- 1--------------EEEE w/o SPT
0.4695
0.9668 0.9917
0.6884
0.8167
Qwen2.5-VL.3B--VSSle
0.6145
0.8673
0.9218
Limitations. Although VSGRPO effectively improved MLLM performance on complex tables, it introduced notable computational overhead during training. Specifically, each LaTeX output had to be rendered into a PDF and then converted to a PNG image for CW-SSIM computation a time-consuming process that created training bottleneck, even with overhead and limited GPU resources, we trained VSGRPO on only 5,936 complex tables.
Due this
References
[1] Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. Image-based table recognition: data, model, and evaluation. In ECCV. Springer, 2020.
[2] Jiaquan Ye, Xianbiao Qi, Yelin He, Yihao Chen, Dengyi Gu, Peng Gao, and Rong Xiao. Pingan- vegroup's solution for icdar 2021 competition on scientific literature parsing task b: table recognition to html. arXiv, 2021.
[3] Ahmed Nassar, Nikolaos Livathinos, Maksym Lysak, and Peter Staar. Tableformer: Table structure understanding with transformers. In CVPR, 2022.
[4] Yongshuai Huang, Ning Lu, Dapeng Chen, Yibo Li, Zecheng Xie, Shenggao Zhu, Liangeai Gao, and Wei Peng. Improving table structure recognition with visual-al aliinmmmigmment sequential coordinate modeling. In CVPR, 2023.
[5] Huawen Shen, Xiang Gao, Jin Wei, Liang Qiao, Yu Zhou, Qiang Li, and Zhanzhan Cheng. Divide rows and conquer cells: Towards structure recognition for large tables. In ICCAI, 2023.
[6] Nan Jiang, Shanchao Liang, Chengxiao Wang, Jiannan Wang, and Lin Tan. Latte: Improving latex recognition for tables and formulae with iterative refinement. In AAAI, 2024.
[7] Renqiu Xia, Hongbin Zhou, Ziming Feng, Huanxi Liu, Boan Chen, Bo Zhang, and Junchi Yan. Latexnet: specialized model for converting visual tables and equations to latex code. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2025.
[8] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Jun-Mei Song. Mingchuan Zhang, Y. K. Li, Yu Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. ArXiv, 2024.
[9] Xu Zhong, Elahch Shafieibavani, and Antonio Jimeno- Yepes. Image-based table recognition: data, model, and evaluation. In ECCV, 2019.
[10] Yongshuai Huang, Ning Lu, Dapeng Chen, Yibo Li, Zecheng Xie, Shenggao Zhu, Liangcai Gao, and Wei Peng. Improving table structure recognition with visual-alignment sequential coordinate modeling. CVPR, 2023.
[11] Sebastian Schreiber, Stefan Agne, Ivo Wolf, Andreas Dengel, and Sheraz Ahmed. Deepdesrt: Deep learning for detection and structure recognition of tables in document images. In ICDAR, 2017.
[12] Shubham Singh Paliwal, Vishwanath, Rohit Rahul, Monika Sharma, and Lovekesh Vig. Tablenet: Deep learning model for end-to-end table detection and tabular data extraction from scanned document images. ICDAR, 2019.
[13] Chris Tensmeyer, Vlad I Morariu, Brian Price, Scott Cohen, and Tony Martinez. Deep splitting and merging for table structure decomposition. In ICDAR, 2019.
[14] Zengyuan Guo, Yuechen Yu, Pengyuan Lv, Chengquan Zhang, Haojie Li, Zhihui Wang, Kun Yao, Jingtuo Liu, and Jingdong Wang. Trust: An accurate and end-to-end table structure recognizer using splitting-based transformers. arXiv, 2022.
10

---

[15] Zhenrong Zhang, Jianshu Zhang, Jun Du, and Fengren Wang. Split, embed and merge: An accurate table structure recognizer. PR, 2022.
[16] Chixiang Ma, Weihong Lin, Lei Sun, and Qiang Huo. Robust table detection and structure recognition from heterogeneous document images. PR, 2023.
[17] Pengyuan Lyu, Weihong Ma, Hongyi Wang, Yuechen Yu, Chengquan Zhang, Kun Yao, Yang Xue, and Jingdong Wang. Gridformer: Towards accurate table structure recognition via grid prediction. In ACM MM, 2023.
[18] Sachin Raja, Ajoy Mondal, and CV Jawahar. Table structure recognition using top-down and bottom-up cues. ECCV, 2020.
[19] Hao Liu, Xin Li, Bing Liu, Degiang Jiang, Yinsong Liu, Bo Ren, and Rongrong Ji. Show, read and reason: Table structure recognition with flexible context aggregator. In ACM MM, 2021.
[20] Jianqiang Wan, Sibo Song, Wenwen Yu, Yuliang Liu, Wenqing Cheng, Fei Huang, Xiang Bai, Cong Yao, and Zhibo Yang. Omniparser: A unified framework for text spotting key information extraction and table recognition. In CVPR, 2024.
[21] Zewen Chi, Heyan Huang, Heng-Da Xu, Houjin Yu, Wanxuan Yin, and Xian-Ling Mao. Complicated table structure recognition. arXiv, 2019.
[22] Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. Nougat: Neural optical understanding for academic documents. arXiv, 2023.
[23] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf. ArXiv, 2023.
[24] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, and Tat-Seng Chua. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. CVPR, 2023.
[25] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaoshen Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. ArXiv, 2025.
[26] Haozhan Shen, Peng Liu, Jingeheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-rl: A stable and generalizable rl-style large vision-language model. ArXiv, 2025.
[27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics, 2002.
[28] Yang Gao, Abdul Rehman, and Zhou Wang. Cw-ssim based image classification. In 2011 18th IEEE International Conference on Image Processing, 2011.
[29] Erhard Schmidt. Zur theorie der linearen und nichtlinearen integralgleichungen. Mathematische Annalen, 1907.
[30] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024.
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063:: 2017.
[32] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangehi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2024.

---

[33] Haozhan Shen, Peng Liu, Jingcheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, Ruochen Xu, and Tiancheng Zhao. Vlm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint M.Xi::5544.06111 2025.
[34] Yuze Zhao, Jintao Huang, Jinghan Hu, Xingjun Wang, Yunlin Mao, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang. Wenmeng Zhou, and Yingda Chen. Swiftra scalable lightweight infrastructure for fine-tuning, 2024.
[35] Mathpix, Inc. Mathpix - po-pee text recognition. https::/mattpiix.cmm 2025.
[36] OpenAI. Gpt-4o: Gpt-4 with vision capabilities. https://ppemai.co//reeeaaall getpchhgpttt444 40 2025.
[37] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[38] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-sourg multimodal models with model, data, and test-time scaling. arXiv preprint arXiv: 242.0221,, 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the Limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This work does not include a theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have demonstrated all technical details reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer:
[NA]
Justification:
The data and the code would be released after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeee //nips.com / casssicnnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Please refer to the experimental setup.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due resource limitations, we do not repeat one of the experiments. Running a complete experiment takes one week.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the device information in the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://ppg.....o.o...oooooo////a
Answer: [Yes]
Justification: We have reviewed the NeurolPS Code of Ethics and checked the paper in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work is not related any private or personal data, and there's explicit negative social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: No such models or datasets are involved.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original papers that offer the original idea or technical details.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We invite graduate students majoring in computer science participate in the volunteer-ber based evaluation. The instructions below are provided for each evaluator: the top of the page, a reference table image is presented, followed by four model- generated table images. You are asked to anonymously select the image that best matches the reference. Preferably, choose only one. If multiple candidates appear equally similar, multiple selections are allowed. In cases where no image is clearly similar, prioritize those with the most similar structural layout.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
18

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
We use the pretrained MLLM in the task of table image LaTeX code generation.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hutpe://hearpss.cco/ooorrrrrrr/p for what should or should not be described.
19

---

A Case Study
In this section, we demonstrate the limitations of relevant metrics through result visualizations, and the capability of our method VSGRPO to generate such complex table and analyze the effectiveness of our approach.
The Limitation of Metric. We illustrate LaTeX-level ambiguity with two visually identical table renderings whose TEDS scores nonetheless differ. In Figure 2 (TEDS 0.8047), the ground -tutt code wraps every cell's contents in an empty group }. whereas the model output omits these no-op braces. Although neither variation alters the final rendering. they change the underlying token sequence and thus reduce the TEDS score. In both figures, the relevant LaTeXX code differences are highlighted in yellow. In Figure 3 (TEDS 0.8983), the sole divergence lies in the use of different bold commands.
Train
Dev
Test
LaRX
? 2
2,745
213
248
Wegginttabular) (looc) Vhline
[Train] [Dev] & [Test] Vhline
ú ;==5
1,477
138
144
[> (2,745) & [213) & (248RI
[>55] & (1,477) & [138) & (144])
& 1==00
793
64
66
1== & (793) & [64] & [66]])
? 20
459
0
0
20= & [459) & [0] {ОJN
(5=50 & [163} & [(] & {ОA) Vhline
? 50
163
0
0
(a) Groun Truth
Train
Dev
Test
LaleX
(taginttabular) (tcoo))bbaa)))ooo)))))))) Vhline
1.=2
2,745
213
248
& Train & Dev & Test \\ thline
':=5
1,477
138
144
x & 2,745 & 213 & 248
> & 1,477 & 138 & 144
ù ;=10
793
64
66
>=0 & 793 64 661
>=20 & 459 & 0 0
ú 20
459
0
0
=50 & 163 & 0 & OW thline
(enu(tay
=0
50
163
0
0
(b) Qwen225VV-38 Metric: CW-SSIM-09816, TEDS SEDDS-tructureel,, TEDS=0.807
Figure Example 1: LaTeX code ambiguity.
Visualisation of Complex Tables. As shown in Figure 4 and Figure 5 they the complex table image from the ground truth and the table image rendered from the LaTeX generated by our method VSGRPO, respectively.
Visualisation effectiveness. As shown in Figure our method VSGRPO improves the quality of the LaTeX generated by SFT, and the CW-SSIM score also reflects the visual similarity between the images.
B Metric
CW-SSIM.
The specific formula as follows:
CW-SSIM(X,Y)
Σ
SSIM dr),
(5)
(E(A.H.V..)
where for each sub-band i:
(2 My, + C1) (20x,y: + +C2)
SSIM(ex, (u&x + 12 +c (0x +oy +C2)
(6)
20

---

Laiex
Problem
UMAD 0.1
Bandi:
GESMR
SAMR
\begin(tabular) Holpiceecccoppeeeeee
Nguyeal
50
50
47
47
Problem UMAD 0.1 & Bandit & GESMR SAMR
Nguyen2
50
48
34
Wmiddule
Nguyen3
Nguyen1& & (bb 50 & (lo 50) &47&4711
Nguyend
34
14
2
Nguyen2& (bb 50 & (lbf 50) (00&&44488 48& (34H)
Nguyenã
50
42
24
23
Nguyen6
50
30
6
23
Nguyen? & Wbf 8) & 2 & 0& Abf 3JM
Nguyea7
8
0
3
Nguyen8 & & 0 & 0 & OWbootoomrule
Nguyea8
(enuulayllr))
(a) Ground Truth
LaTeX
\begin (awbuaaarcclllllleeeeeeoooooooe Hoopplle
Probiem
UMAD 0.1
Bandit
GESMR
SAMR
Problem & UMAD 0.1 Bandit & GESMR & SAMRW
Nguyenl
50
47
\midrule
Nguyen2
50
50
48
34
Nguyenl (50) (50) & 47 47
Nguyen3
48
43
12
g
Nguyen2 & (50) & (textbf(50) & 48 &
Nguyen4
34
43
14
2
lunderline
Nguyen5
50
42
24
23
Nguyen6
50
30
6
23
Nguyen7 (textbf & 2 & & (bexthf W
Nguyen7
8
2
0
3
Nguyen8 & 0 & 0 & 0
Nguyen8
0
0
1bottomrule
lend (tabular)
(b) Qwen2.5-VV-3B-VSGGRPP Metric CW-SSIM=1, TEDS-Smuccure==1, TEDS=0.8983
Figure 3: Example 2: LaTeX code ambiguity.
Let X and Y be two aligned grayscale table images trimmed even dimensions. Apply a one-level Haar wavelet each, yielding four sub-bands cx and cy for i= A (approximation), H (horizontal detail), V (vertical detail), and (diagonal detail). For each sub-band i, let µxx, and μy; be the pixel-wise means, xx and of the variances, and xx,y; the covariance. Constants Ci = (K L) and C2 (.)) stabilize the denominator (T 255.0, Ki is 0.01 and K₂ is 0.03).
TEDS-Structure. The specific formula as follows:
TEDstructure
SEDS-Structure
max (TT, [T
proeal denotes the total number of nodes in the structural tree parsed from the predicted table, and (Tgl denotes the total number of nodes in the structural tree parsed from the ground-truth table.
TEDS. The Tree Edit Distance based Similarity (TEDS) computation extends SEDD--Strcttre by first calculating the total edit distance: TED = TEDstructre + TED and finally normalizing by the larger of the two tree sizes to yield the similarity score. The specific formula as follows:
TED
TEDS
max (TTrred (Tgll)
C Human Evaluation
To evaluate whether the table images rendered from the model-generated LaTeX better align with human preferences, we place the ground truth table image at the top and display the model's predicted table image below it, side by side. The order is randomly shuffled, and the names are hidden. We place the ground truth table image the top, allowing humans select one or more images based on subjective similarity. As shown in Figure
21

---

w/o SACP \ w/ SACP
Model
Score
0.05
0.1
Coverage
Size (V)
SSCV
Coverage
Size (v)
SSCV (4)
APS
0.95
0.95
3.68
2.28
0.41 0.28
0.90 0.90
2.52 \ 1.75
0.51 0.45
1D-CNN
2.29
0.90 \0.00
1.74
SAPS
0.95
6.68 4.31
2.48 1.83
0.90 0.90
5.56 3.02
4.84 1.95
APS
0.95
5.73
3.27
0.47 0.38
0.90 0.90
2.06
0.35 0.34
Indian Pines
3D CNN
RAPS
0.95
4.12
3.92 6,10
0.10 0.21
0.90 0..00
2.338
0.11 0.25
SAPS
5.14
0.98
0.91 0.90
1.97 3..72
1.61 1.51
APS
0.95 \ 0.95
5.56
4.83
0.20 0.16
0.90 0.90
3.28 22.74
0.92 0.15
Hyorid5N
RAPS
0.95
7.12 7.34
0.40 \ 0.96
0.90 0,90
3.995
0.75
SAPS
\ 0.41
3.440
0.61
APS
0.95 \ 0.95
2.81 1.73
0.42 0.18
0.90 0.90
2.00 1.38
0.41 0.47
SSTN
RAPS
0.95 0.95
2.52 1.62
0.29 0.30
0.90 0.90
187 1.36
0.29
SAPS
6.98 4.16
4.38
0.90 0.90
3.25
6.35 3.09
APS
0.95 \ 0.95
2.26
1.92
0.39 0.37
0.90 0.90
1.65 1.57
0.40 0.27
1D-CNN
RAPS
0,95 \ 0.95
2.00
1.83
0.23 \ 0.31
0.90 0.90
1.54
0.40 0.29
SAPS
0,95 0.95
3.92
3.99
1.77 \ 2.21
0.90 0.90
3.44 3.04
3.62 2.68
Pavia University
APS 0.94 \ 0.95
2.77
2.34
1.04 0.69
0.89 0.89
2.14
0.94 0.66
3D-CNN
RAPS
0.94 \ 0.94
2.57
2.28
0.41 \ 0.46
0.89 0.89
2.04 1.76
0.64 0.56
SAPS
4.31
0.89
3.33
6.56 4.23
APS
0.95 \ C.95
4.79
4.59
4.59 3.58
0.90 0.90
3.38 3.00
2.39
1.33
HybridsN
RAPS
0.94 \ 0.95
5.50
5.36
0.47 0.36
0.89 0.900
3.70 3.74
0.81 0.07
SAPS
0,95 0.95
5.44 5.57
1.98 \ 2.35
0.90 0.90
3.99 3.71
3.33 3.47
APS
0.95 \ 0.95
1.75 \ 1.24
0,22 \ 0.26
0.90 0.90
1.39 1.11
0.23 0.29
SSTN
RAPS
0.95 \ 0.95
1.60
1.22
0.20 \ 0.20
0.90
1.3 1.10
0.13 0.23
SAPS
0,95 0.95
3.26
2.24
2.07 0.92
0.90 0.90
2.75 1.99
2.96 1.64
APS
0.95 \ 0.95
1.40 \ 1.20
0.15 \ 0.15
0.90 0.90
1.20
1.09 1.09
0.13 0,25
1D-CNN
RAPS
0.95 \ 0.95
1.37
1.20
0.06 \ 0.20
0.90 0.90
1.9 1.07
0.23 0.25
SAPS
0.95 \ 0.95
3.63
A 1.71
1.20 0.16
0.90 0.90
2.97 1.28
2.04 0.16
APS
0.95 \ 0.95
1.48 1.25
0.20 0.19
0.90 0.90
1.17 X 1.08
0.12 0.16
3D CNN
RAPS
0.95 \ 0.95
1.47
\ 1.24
0.13 \ 0.18
0.90
0.900
1.15 1.07
0.15 0.17
Sa Inas
SAPS
0.95 \ 0.95
2.02 2.91
0.58 0.28
0.90 0.90
2.39 1.29
1.07 0.13
APS
0.95 \ C.95
1.37
1.09
0.18 0.07
0.90 0.90
1.10 X 1.03
0.18 \ 0.23
HybriddS
RAPS
0.95 \ 0.95
1.07 1.90
1.20
0.12 \ 0.10
0.90 0.90
106 1.00
0.12 0,31
SAPS
0.95 \ 0.95
1.37
0.42 0.33
0.90 0.900
1.6 1.18
0.69 0.31
APS
0.95
0.95
1.70
1.19
0.21 0.16
0.90 0.90
1.37 \ 1.08
0.23 0.10
SSTN
RAPS
0.95 \ 0.95
1.50
1.18
0.12 \ 0.14
0.90 0.90
1.99 1.05
0.10 0.11
SAPS
0.95 \ 0.95
5.42
2.55
3.58 0.86
0.90 0.90
3.91 2.07
3.64 1.26
Ground Truth Table Image
Figure 4: Ground truth example of complex table.
22

---

w/o SACP w/ SACP
Model
Score
0.05
Coverage
Size
SSCV (V)
Coverage
Size
SSCV (4)
APS
0.95 \ 0.95
3.58
2.28
0.41 \ 0.28
0.90 \ 0.90
2.52
1.75
0.51 \ 0.45
ID-CNN
RAPS
0.95 \ U.9
4,99 \ 2.29
0.54 0.57
0.90 \ 0.90
2.54
1./4
0.85 \ 0.30
SAPS
\
6.68
1.23
3.02 5.56
1.95
Indian Pines
APS
0.94 \ 0.95
5.73
3.27
0.47 \ 0.38
0.90
0.90
2.85
2.06
0.35 \ 0.34
3D-CNN
RAPS
0.95 \ 0.95
4.12
3.92
0,10
0.21
0.90
0.90
3.21
2.38
0.11 0.25
SAPS
0.95 \ 0.95
6.40
5.44
0.98 0.95
0.91
0.90
4.97
3./2
1.61 \ 1.51
APS
0.95 \ 0.95
5.56 \ 4.83
0,20 \ 0.16
0.90
0,90
3.28
2.74
0.92 \ 0:15
HybridSN
RAPS
0.95 \ 0.95
7.24
7.12
0.40
0.96
0.90 \ 0.90
4.29 \ 3.95
0.98 \ 0.75
SAPS
0.95 \ 0.95
6.79
6.07
0.19 0.41
0.90
0.90
4.03 \ 3.40
0.64 \ 0.61
APS
0.95 \ 0.95
2,81 \ 1.73
0.42 \ 0.18
0.90
0,90
2.00
1.38
0.41 0.47
SSTN
RAPS
0.95 \ 0.95
2.52
1.62
0.29
0.30
0.90
0.90
187
1.36
0.29 0.34
SAPS
0.95 \ 0.95
6.98 \ 4.16
4.38 1.74
0.90
0.90
5.33
3.25
6.35 3.09
APS
0.95 \ 0.95
2.26
1.92
0.39 \ 0.37
0.50
0.90
1.55
1.57
0,10 \ 0.27
1D-CNN
RAPS
0.95 \ 0.95
2,00
1.83
0.23 0.31
0.90
0.90
1.59
1.54
0.43 \ 0.29
SAPS
0.95 \ 0.95
3.92
3,99
1.77 2.21
0.90
0.90
3.44
3.04
3.62 \ 2.68
Pavia Univers'sy
APS
0.94 \ 0.95
2.77
2.34
1.04 \ 0.69
0.89 \ 0.89
2.14 \ 1.79
0.94 \ 0.66
3D-CNN
RAPS
0.94 \ 0.94
2.57
2.25
0.41 \ 0.46
0.89
0.89
2.04 \ 1.76
0.64 \ 0.56
SAPS
0.94 \ 0.94
4.80
4.31
4,56 3.85
0.89
0.39
4.04
3.32
6.56 \ 4.23
APS
0.95 \ 0.95
4.79
4.59
4,59 \ 3.58
0.90 \ 0.90
3.38
3.01
2.39 \ 1.33
HybridSN
RAPS
0.94 \ 0.95
5.50
5.36
0.47 Ş 0.36
0.89
\ 0.90
3.70 \ 0.74
0.51 \ 0.07
SAPS
0.95 \ 0.95
5.57
\ 5.44
1.98 \ 2.35
0.90 \ 0.90
3.99
3.71
3.33 3.47
APS
0.95 \ 0.95
1.75
1.24
0.22 \ 0.26
0.90 \ 0.90
1.39
1.11
0.23 \ 0.29
SSTN
RAPS
0.95 \ 0.95
1.50 \ 1.22
0.20
0.20
0.90 \ 0.90
1.3 \ 1.10
0.13 \ 0.23
SAPS
0.95 \ 0.95
3.26
2.24
2.07
0.92
0.90 \ 0.90
2.75
1.91
2.96 \ 1.64
APS
0.95 \ 0.95
1.40
1.20
0.15
0.15
0.90 \ 0.90
1,20
1.09
0.18 0.25
1D-CNN
RAPS
0.95 \ 0.95
1.37
1.20
0.06 \ 0.20
0.90 \ 0.90
1.19
1.07
0.23 \ 0.25
SAPS
0.95 \ 0.95
3.53
1.71
1.20 \ 0.16
0.90 \ 0.90
2.97
1.28
2.04 \ 0.16
APS
0.95 \ 0.95
1.48
1.25
0.20 \ 0.19
0.90 \ 0.90
1.17
1.08
0.12 \ 0.16
Salinas
3D-CNN
RAPS
0.95 \ 0.95
1.47
1.24
0.13 0.18
0.90 \ 0.90
1.15
1.07
0.15 0.17
SAPS
0.95 \ 0.95
2.94
2.02
0.58 \ 0.28
0.90 \ 0.90
2.39
1.29
1.07 \ 0.13
APS
0.95 \ 0.95
1.37
1.09
0.18 \ 0.07
0.90 \ 0.90
1.10
1.03
0.18 \ 0.23
HybridSN
RAPS
0.95 \ 0.95
1.20
1.07
0.12 Ş 0.10
0.90 \ 0.90
1.06
\ 1.00
0.12 \ 0.31
SAPS
0.95 \ 0.95
1.90
1.37
0.42 \ 0.33
0.90 \ 0.90
1.66
1.18
0.69 \ 0.31
APS
0.95 \ 0.95
1.70
1.19
0.21 \ 0.16
0.50 \ 0.90
1.37
1.08
0.23 \ 0.10
SSTN
RAPS
0.95 \ 0.95
1.50
1.18
0.12 0.14
0.90 \ 0.90
1.29
1.06
0.10 0.11
SAPS
0.95 \ 0.95
5.42
2.65
3,58 \ 0.86
0.90
0.90
3.91
2.07
3.64 \ 1.26
Qwan2.5-VL-3B-VGRRR
Figure 5: Prediction example of complex table.
23

---

Data
Model
Xmin
0
ABTC
30.19
65.49 10087
0.2902 0.2902
ACDM
32.42
69.93 T'ns
0.2658
OHU + SNe la
LDEM
1778.5/
85.82 =: 0.72
0.2840
-19.4b I 0.02
20.42
ACDM
1799.09
67.16 0.81
0.3163
0.013
-19.44
0.02
OHD SNe la BAO
LDEM
1780.53
66.04 1 0.51
0,2800
0.01
199.46 0.02
20.50
ACDM
1803.72
58.08 -1,53
0.2990
10.0000
R.0.3
-19.42 0.002
OHD SNe
LDEM
1778.67
65.83 0.72
0.2800
0.01
-19.46
1.000
-4.81
ACDM
1800.04
67.75 d:
0.3052
-1.02
OHD SNe BAO
CMB
LDEM
1778.67
65.83 4: 0.72
0.01 0.2800
19.46 10.02
2.70
ACDM
1803.74
68.02 +0.32
0.3000 6,109
000000
1..02
(a) Ground Truth
Xmin
OHD
LDEM
30.19
65.49 P03R7
0,2902 Hoanns
-2.23
ACDM
32.42
69.93
0.2558 0.160
OHD + SNe
LDEM
1778.67
65.82 + 0.72
0.2830"N:Ti6
137
-99.66 + 0.02
-00.12
ACDM
1799.09
67.165581
0.3163"
199.44 0.00
OHD BAO
LDEM
1780.53
66.04 + 0.51
0,2800 +
-19.46
0.00
200.0
68.08 10004
0.2990
0.0088
119.42
OHD SNe CMB
LDEM
1778.67
65.83 1: 0.72
0,2800
0.01
199.46
1.0022
-4.81
ACDM
1800.04
67.75 #: 0.55
0,3052
10:0074
0.0003
19.42
OHD SNe la + BAO CMB
LDEM
1778.67
65.83 + 0.72
0,2800
0.01
19..46
0.02
-2.7
ACDM
1803.74
68.02
10153
0.3000
100066
199.42 0.02
(b) Qwenn2.5
(CW-SSIM00.8866
Data
Model
X
Ho
Q
M
ABTC
OHD
LDEM
30.19
65.49 0.50
0.2902
0.075
-2.23
ACDM
32.42
69.93 -2.0
0.2658
0.0.c
OHD + SN la
1778.67
ACDM
1799:09
0.72
0.2830
17.16.10
0,0053
1000
-19.46 0.02
0.3163
Fooms
10.0000
LDEM
1780.53
65.04 0.51
0.2800
0,01
-19 46
OHD
SNe BAO
ACDM
1803.71
68.00 08.08.22 10.4
0.2990
000000
0.0098
-19
29.8
LDEM
1778.67
65.83 0.72
0.2800 2: 0.01
-99.46 0.02
OHD SNe CMB
ACDM
1800,04
67.75 1: 0.56
0.3052 2013
20:00
-19.42 1.002
LDEM
1778.67
65.83 + 0.72
0.2800 #: 0.01
199.46
0.02
ACDM
1803.74
68.02 0.50
11:35
0.3000
0.005
-19.42 0000
(c)
(CW-SSIM:0.6092)
Figure 6: Visualization of result comparisons. (a) Ground Truth refers to the ground truth table image from the simple testing dataset; (b) Qwwn2.5-VL-3B--V55. ----OO represents the table image rendered from LaTeX generated by the Qwen2.5-VL-33 model trained with our VSGRPO method; (c) (Qwen255-LL-3B-SFF. represents the table image rendered from LaTeX generated by the Qwen2.5. VL-3B model trained with SFT. The corresponding CW-SSIM scores are reported. Blue boxes highlight examples where Qwen2.5-VL-3B- VSGRPO differs from the ground truth, and red boxes highlight examples where Qwen2.5-VL-3B--TH differs from the ground truth.
24

---

Interactive Image Selection
Progress: 2/50 images
Proceued
0.45
0.55
5.1
23.7
17.5
11.1
46.
0.38
0.10
8:11
87
18.
18.3
17.2
31.6
39.8
0:3
25.7
Reference Image (Index: 262)
0.31
7.4
260
38.1
0.90
0.30
8.0
71
13.1
12.
13.9
213
0.20
0.15
8:15
75
31a
26.5
0.30
0.00
1.5
12.7
12.5
21.9
26.5
0:98
0.06
5/5
755
155
178
119
318
301
115
100
0.3
025
01
105 2011
0.8
145
39
74
6.30
3.8
183
85
35
13 10 mk
136
20
089
8.00
134
113
334
133
234
81
28
100
105
115
230
134
03
0
3
Figure 7: The table image selection page for human evaluation.
25


Reviews:
[{"review_id": "tv3riwVhEg", "reviewer": "Reviewer_A7cA", "summary": "This paper studies the task of publication-ready latex code for tables from the visual image of the table. They solve this problem by finetuning a pre-trained MLLM on table-to-LaTeX dataset, with newly proposed dual-reward RL based on GRPO. Instead of directly using the outcome reward, they design structure-level reward particuarly designed for the LaTeX code and also a visual fidelity reward from rendered outputs. They show the effectiveness of their approach on their own collected testing dataset, which is processed from LaTeX table from arXiv papers.", "questions": "1. How about we just use the outcome reward? Does it also brings similar gain? E.g., Intern2-VL-1B-GRPO vs. Intern2-VL-1B-VSGRPO", "limitations": "The main concerns are: 1. lacking comparison with other RL framework 2. only verify on in-domain test set which is unfair.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. Generating publication-ready LaTeX source code given visual table input is of important application value. And how to improve the understanding of structure table image and together generating code is an underexplroed research problem.\n2. The proposed approach is novel that not they design two specific reward for this task: 1. the structure-level reward for LaTeX code 2. the visual fidelity reward from rendered outputs. And this dual-reward provides richer signals than simple outcome reward.\n\nWeaknesses\n1. The experimental results are not fair enough, the training and test dataset are built from the same source, which is the LaTeX tables from the arxiv. It is unfair to finetune the proposed model e.g., Intern2-VL-VSGRPO and Qwen2.5-VL-VSGRPO on this training set and then compare with other baseline on in-domain test sets.\n2. Instead of independ votes in Human Evaluation table, it's better to see using VSGRPO vs. w/o using VSGRPO and see how the winning rate looks like.\n3. The VSGRPO is not compared with exisiting RL approach, e.g., GRPO. It's important to show the effectiveness of the new proposed framework over exisitign simple approach.", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer, thank you for your thorough review and helpful suggestions! We also appreciate you mentioning that our paper has \"important application value\" and our approach is \"novel\". Please find our responses and changes below. \n\n### Weakness 1: The results are biased since both training and test data come from arXiv tables, making it unfair to fine-tune models like Intern2-VL-VSGRPO and compare them with baselines on in-domain tests.\n\nWe appreciate the reviewer’s thoughtful concern regarding the evaluation fairness and potential in-domain bias stemming from training and testing on LaTeX tables sourced from arXiv. In response, we have extended our evaluation to include two additional publicly available out-of-domain table datasets: **These two datasets, primarily derived from financial documents and reports, are markedly out-of-domain relative to arXiv LaTeX tables. They differ in content domain, table styles, layout complexity,  visual noise and even the presence of colored cells, providing a challenging and comprehensive testbed for evaluating model generalization**.  FinTabNet [1], which consists of real-world and complex scientific and financial tables with detailed structure annotations, and SynthTabNet [2], a dataset of Chinese financial announcements. Unlike the arXiv-based LaTeX tables, these datasets are typically provided in HTML or image format without LaTeX source code. Therefore, CW-SSIM scores cannot be computed, as they require image rendering from ground-truth LaTeX for comparison. To address this, we conducted a human evaluation study, comparing the visual fidelity of LaTeX-rendered tables generated by our method against state-of-the-art MLLMs, using the original table images as ground truth references.\n\n**We randomly selected 100 samples from the validation set and display the ground truth table image at the top and the model’s predicted table image below it, side by side, with the order randomly shuffled and labels hidden. Multiple human assessors independently vote on which image appears most visually similar to the target. The final outcome is determined by majority vote**.  As shown in Tables R1 and R2, the results of this human study demonstrate that our method consistently achieves superior visual quality across both datasets, indicating its strong generalization capability to tables from different domains and formats. These findings provide further evidence of the robustness and practical applicability of our approach beyond the arXiv domain.\n\n\n\nTable R1：Results of human evaluation in FinTabNet.\n\n| Model | QwenVL-VSGRPO | QwenVL-SFT | Internvl3-78b | QwenVL2.5-72b | GPT4o |\n| ----- | ------------- | ---------- | ------------- | ------------- | ----- |\n| Vote  | 77            | 50         | 14            | 13            | 2     |\n\n\n\n\n\nTable R2：Results of human evaluation in SynthTabNet.\n\n| Model | QwenVL-VSGRPO | QwenVL-SFT | Internvl3-78b | QwenVL2.5-72b | GPT4o |\n| ----- | ------------- | ---------- | ------------- | ------------- |-----  |\n| Vote  | 72            | 47         | 12            | 8             | 4     |\n\n\n\n### Reference\n\n[1] Zheng, Xinyi et al. “Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020): 697-706.\n\n[2]Hou, Qiyu et al. “Synthesizing Realistic Data for Table Recognition.” IEEE International Conference on Document Analysis and Recognition (2024).\n\n\n\n### Weakness 2: VSGRPO vs. w/o using VSGRPO.\n\nThank you for this insightful suggestion. We agree that a direct comparison between our method with and without VSGRPO provides a clearer understanding of its contribution. To this end, we conducted a focused human evaluation on **100 randomly selected samples**, comparing the full model (with VSGRPO) against its ablated variant (without VSGRPO). The results are as follows:\n\n- VSGRPO was preferred in 52 cases.\n\n- The variant without VSGRPO was preferred in 18 cases.\n\n- 30 cases were judged as indistinguishable in quality.\n\nThese results demonstrate a clear preference for the VSGRPO-enhanced version. In addition, as part of our out-of-domain evaluation (detailed in response to the previous comment), we also performed comparisons between the two variants on external datasets (FinTabNet and SynthTabNet). The outcomes consistently showed that the VSGRPO-enhanced model produces LaTeX-rendered table images with higher visual similarity to the ground truth, further validating its effectiveness.\n\n### Weakness 3: VSGRPO isn’t compared with existing RL methods like GRPO, making it hard to show its advantage.\n\nWe appreciate the reviewer’s suggestion and would like to clarify that we have **included a direct comparison with GRPO** as a baseline in our paper. Specifically, the original GRPO framework utilizes a rule-based verifier to assess the quality of generated text against the ground truth. In the context of table generation, this would involve comparing the generated LaTeX code with the ground-truth LaTeX using a manually designed verifier. However, unlike typical question-answering tasks where the ground truth is relatively short and correctness is easier to verify, **LaTeX code is often long, structurally complex, and challenging to evaluate using exact matching or heuristic rules**.  To adapt GRPO to our task fairly and meaningfully, we replaced the original rule-based verifier with a **TEDS-Structure-based verifier**, which is better suited to assessing structural and visual similarity between table representations. The performance of this adapted GRPO variant is reported in **Table 6** as Qwen2.5-VL-3B-GRPO-TEDS-Structure. As shown in the results, our proposed **VSGRPO significantly outperforms** this GRPO baseline, highlighting the effectiveness of our framework in handling the structured nature and generation complexity of LaTeX tables.\n\n### Question 1: Does using only the outcome reward yield similar gains, e.g., Intern2-VL-1B-GRPO vs. VSGRPO?\n\nSee above, in\"Weakness 3\"."}, {"review_id": "tZkWGAe0Mc", "reviewer": "Reviewer_6PYg", "summary": "The paper proposed Table2LaTeX-RL: a novel algorithm that trains a multimodal LM to generate faithful LaTeX code from images of structured tables. The detailed workflow is:\n\n* Collecting a large-scale Table2LaTeX corpus (≈1.2 M image–code pairs) by extracting LaTeX/table pairs from arXiv papers.\n* Performing supervised fine-tuning on a vision–language LLM with the Table2LaTeX dataset.\n* Using VSGRPO, a novel RL strategy, to further fine-tune the LLM. The RL signal combines two rewards: (1) a visual reward that matches the original table image with the image rendered from the generated LaTeX, calculated with CW-SSIM; (2) a structural-match reward that encourages alignment of LaTeX structure, measured by TEDS-Structure. A group of candidate responses is generated for each input query, and the average advantage of the group is used as the baseline against which RL optimisation is performed.\n\nThe extensive experiments compare the VSGRPO-trained model with both general-purpose multimodal LLMs and commercial/open-source LaTeX “snipping” tools. The results indicate that, over both visual fidelity (CW-SSIM) and structural accuracy (TEDS-Structure) metrics, the VSGRPO model outperforms all baselines. An ablation study further shows that each of the two RL reward signals contributes significant gains, with the combination yielding the best performance.", "questions": "+ Could the authors provide architectural discussion or ablation studies that contrast why dual-reward RL outperforms LATTE’s iterative-refinement strategy, including evidence clarifying VSGRPO’s motivation and novelty?\n+ Could the authors also evaluate standard vision-only table-structure models in the experiment tables, providing results that demonstrate how the proposed method compares to mature computer-vision pipelines? Or authors should explain why such methods does not fit in the context.\n+ I am curious that if the model trained on arxiv tables can generalize well on other domains` table like finance reports? Some case study on real finance table benchmark would provide interesting insights. For example on a random subset of FinTabNet.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength:\n+ The paper addressed an important topic related to processing structured tabular data at scale by converting them to latex, which could have important application in domains such as finance/trading. \n+ The idea of combining visual and structural reward is interesting and addressed a core challenge in modeling tabular data: complex, heterogenous structure. By adding visual reward, the method also handles a case where table structure is not standard, i.e, not a nice data-frame style column-row structure. Although prior works has explore generation of structured data from image[1,2], this paper seems the first to be extend its application to tabular data/latex code.\n+ The dataset on latex code/table pair will be a helpful contribution. \n+ The benchmark metrics/dataset and ablation study is to the point and comprehensive.\n\n## Weakness: \n+ The paper does acknowledge LATTE[1]  and even reports LATTE’s numbers on an external benchmark (Table 3) and a foot-note explaining that they recalculated the scores from the authors’ released outputs . However, it offers no architectural discussion or ablation that contrasts why dual-reward RL outperforms LATTE’s iterative-refinement strategy, and thus limits the insight from the VSGRPO`s motivation and the claim of its novelty from existing visual-RL methods.\n+ Although the Related-Work section cites several vision-only table-structure models such as [2], none of those systems are re-implemented or valuated in the experiment tables, leaving a gap in understanding how the proposed method stacks up against mature CV pipelines. \n+ The training and evaluation corpora are drawn only from arXiv paper PDFs, and 94 % of the training set are simple black-and-white tables. As a result, the model’s robustness to scanned documents, colored backgrounds such as financial data/earning report is unknown. \n\n[1] Jiang, Nan, et al. \"LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.\n\n[2] Nassar, Ahmed, et al. \"Tableformer: Table structure understanding with transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "was_revised": true, "final_justification": "Although the authors have satisfactorily clarified the LATTE comparison and demonstrated cross‐domain generalization, they have not provided any empirical results against established CV‐based baselines. Without at least one representative TSR model (direct LaTeX or HTML→LaTeX) in the evaluation, the relative benefit of the proposed dual-reward RL remains unquantified. Until this gap is addressed, I will maintain my current rating.", "rebuttal": "We thank the reviewer for their comments, especially for highlighting our paper could have \"important application\" and that our idea is \"interesting \". Below, we address the issues raised in the review:\n\n### Weakness 1: The paper cites LATTE[1] and reports its results, but lacks analysis comparing dual-reward RL to LATTE’s approach, limiting insight into VSGRPO’s motivation and novelty.\n\nThank you for raising this important point.\n\nFirst, we would like to clarify that **LATTE has not released its full implementation**, particularly the code for the iterative refinement module. This component depends on a **separately curated dataset**, which is also not publicly available. As a result, we are unable to precisely reimplement LATTE or conduct a direct comparison beyond referencing the limited results provided in their paper.\n\n\nSecond, from a methodological perspective, **our approach and LATTE differ fundamentally in design philosophy and execution**. Our method adopts the LLM/MLLM paradigm by establishing an **end-to-end pipeline** encompassing large-scale supervised fine-tuning (SFT), and reinforcement fine-tuning (RFT) for the table-to-LaTeX task. In contrast, **LATTE centers around an iterative post-refinement strategy**, which is applied after initial LaTeX generation.\n\n\nIn other words, **our goal is to embed high-quality LaTeX generation capabilities directly into the model**, enabling accurate outputs at inference time without needing further correction. LATTE, on the other hand, relies on a **separate, explicitly designed refinement stage** to improve the quality of the initial outputs, which introduces an **additional computational overhead** and complexity during deployment.\n\nThese distinctions reflect different design goals: **our method aims for integrated accuracy and efficiency**, while LATTE emphasizes post-hoc quality improvement through refinement. \n\n### Weakness 2: Though cited, vision-only models like [2] aren’t evaluated, leaving unclear how the proposed method compares to established CV approaches.\n\nThank you for bringing this up. Our method differs fundamentally from traditional computer vision (CV) approaches by leveraging the **text generation capabilities of Large Language Models (LLMs)** rather than relying on rule-based decoding pipelines.\n\nTraditional CV solutions like **TableFormer** primarily target **HTML table** reconstruction, which benefits from a relatively rigid and sequential syntax. These methods typically decompose the problem into two steps: first detecting the table layout, then predicting the cell content, following the HTML structure where each cell is enclosed in ```<td> </td>``` tags. The deterministic nature of HTML allows such systems to adopt **rule-based strategies** for both structure and content prediction.\n\nIn contrast, our task focuses on **LaTeX table generation**, which presents significantly greater complexity. LaTeX syntax is **more flexible and less strictly structured** than HTML, making it **challenging to apply rule-based layout parsing or sequential tagging approaches**. As such, we propose to **treat table-to-LaTeX as a generation task**, enabling LLMs to directly produce high-fidelity LaTeX code in a more **holistic and end-to-end fashion**. This not only aligns better with the open-ended nature of LaTeX syntax but also takes advantage of the strong reasoning and generation capabilities of modern MLLMs.\n\n### Weakness 3: Training on mostly simple arXiv tables (94%) limits insight into the model’s robustness to scanned or colored documents.\n\nAs rightfully requested by the reviewer, we have used two additional publicly available table datasets to further validate the effectiveness of our method: FinTabNet [3], which consists of real-world and complex scientific and financial tables with detailed structure annotations, and SynthTabNet [4], a dataset of Chinese financial announcements. We conducted a human evaluation to compare the visual similarity between the LaTeX-rendered images generated by our method and those from existing state-of-the-art MLLMs, against the ground truth table images.\n\nWe randomly selected 100 samples from the validation set. For each sample, the ground truth table image and the model's predicted table image were displayed side-by-side. The display order was randomly shuffled and labels were hidden to ensure an unbiased assessment. Multiple human assessors independently cast their votes for the image most visually similar to the target, with the majority vote determining the final result. As shown in Tables R1 and R2, our method also demonstrates consistently strong competitiveness in these baselines.\n\nTable R1：Results of human evaluation in FinTabNet[3].\n\n| Model | QwenVL-VSGRPO | QwenVL-SFT | Internvl3-78b | QwenVL2.5-72b | GPT4o |\n| ----- | ------------- | ---------- | ------------- | ------------- | ----- |\n| Vote  | 77            | 50         | 14            | 13            | 2     |\n\nTable R2：Results of human evaluation in SynthTabNet[4].\n\n| Model | QwenVL-VSGRPO | QwenVL-SFT | Internvl3-78b | QwenVL2.5-72b | GPT4o |\n| ----- | ------------- | ---------- | ------------- | ------------- | ----- |\n| Vote  | 72            | 47         | 12            | 8             | 4     |\n\n### Reference\n\n[1] Jiang, Nan, et al. \"LATTE: Improving Latex Recognition for Tables and Formulae with Iterative Refinement.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.\n\n[2] Nassar, Ahmed, et al. \"Tableformer: Table structure understanding with transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3]Zheng, Xinyi et al. “Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020): 697-706.\n\n[4]Hou, Qiyu et al. “Synthesizing Realistic Data for Table Recognition.” IEEE International Conference on Document Analysis and Recognition (2024).\n\n### Question 1: Can the authors provide analysis or ablation to show why dual-reward RL outperforms LATTE and clarify VSGRPO’s motivation and novelty?\n\nSee above, in \"Weakness 1\".\n\n### Question 2: Can the authors evaluate vision-only models or explain why they’re unsuitable to clarify how their method compares to mature CV pipelines?\n\nSee above, in \"Weakness 2\".\n\n### Question 3: Can the model trained on arXiv tables generalize to other domains like finance, e.g., via case studies on FinTabNet?\n\nSee above, in \"Weakness 3\"."}, {"review_id": "wvHruGZ0gU", "reviewer": "Reviewer_Wpaz", "summary": "The authors address the task of table image to LaTeX code generation in this paper. To overcome the challenges of large table sizes, deeply nested structures, and semantically rich content, they propose a framework based on the multi-modal large language models. To be more specific, they first collect large-scale table-LaTeX pairs from the Internet and then fine-tune a pre-trained MLLM on the dataset. Afterwards, they introduce a dual-reward RL strategy for post-training. A structure-level and a visual fidelity reward are used during post-training. In the experiments, the authors show that their method achieves the state-of-the-art performance compared to the baselines.", "questions": "1. In terms of the reward design, the reward is set to 1 if the similarity metric exceeds a pre-defined threshold, and 0 otherwise. The question is: why not directly use the similarity value as the reward? Intuitively, continuous rewards are better than binary rewards in this task. Forcibly binarizing similarity values seems to make the model unable to distinguish which of the two predictions that are both very similar to GT is better.\n2. As shown in Tab. 7, the performance of the \"w/o SFT\" model is significantly worse than that of the full version. However, there is another difference between these two Settings. More specifically, according to line 169, the number of training samples used in the \"w/o SFT\" setting is only 5,936, which is much smaller than the full version. Therefore, the performance discrepancy in Tab. 7 may be caused by the volume of data and does not explain the necessity of SFT. Could the authors confirm this?\n3. The performance improvement brought by the proposed VSGRPO seems marginal according to Tab. 6. It would be great if the authors could have a discussion about this.\n4. In the Abstract and Introduction sections, the authors highlight the challenges of Table2LaTeX: large sizes, deeply nested structures, and semantically rich content. However, in the Method section, there seem no customized techniques for these challenges, leaving readers feeling that they cannot echo the challenges mentioned earlier after reading.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. This paper studies a practical task (table image to LaTeX code) that is rarely explored before.\n2. By scraping the LaTeX source files and data cleaning, the authors collect a large-scale Table2LaTeX dataset, which would be beneficial to the research community.\n3. Through SFT and VSGRPO, the obtained code generation models can be comparable to or exceed commercial tools, general VLMs and expert VLMs in multiple quantitative metrics, setting an economical and efficient solution for the task.\n\nWeaknesses:\n1. One of the biggest concerns is the limited technical contribution. Although the task is under-explored, the proposed solution is the standard LLM fine-tuning and RL process, except for the reward design.\n2. There are very few qualitative figures in this paper. Without sufficient qualitative results, it's difficult to have an intuitive understanding of the model performance.\n3. The effectiveness of the introduced two rewards is not significant as shown in Tab. 6.", "was_revised": true, "final_justification": "The rebuttal contains detailed and thorough clarifications, addressing most of my initial concerns. I wish the authors could resolve the unfair setting for ablation studies and provide more qualitative results. I am therefore willing to raise my score to borderline accept.", "rebuttal": "Dear Reviewer, thank you for your response, your continued engagement, and your commitment to improving our paper! Please find our responses below.\n\n### Weakness 1: The main concern is the limited technical novelty, as the proposed LLM fine-tuning and RL solution is standard, with the only notable contribution being the reward design.\n\nThank you for acknowledging the practical value of our task and the effectiveness of our method. However, we respectfully disagree with the assessment that our work lacks technical contribution. Notably, **all three other reviewers** explicitly recognized the novelty and technical strength of our approach:\n\n- **Reviewer A7cA** stated: “The proposed approach is novel.”\n\n- **Reviewer 6PYg** commented: “The idea of combining visual and structural reward is interesting.”\n\n- **Reviewer e8fZ** noted: “Novel and interesting application to leverage MLLM’s ability to address complex table LaTeX generation.”\n\nThese consistent positive remarks affirm our view that the paper presents meaningful and original contributions.\n\n\n\n1. **Revealing MLLM Capability and Proposing an Effective Solution**\n\nThe task is underexplored and one of our contributions lies in **demonstrating for that MLLMs can effectively handle table image to LaTeX generation**—a task that is **highly structured, syntactically complex, and not traditionally studied in the vision-language domain**. We establish a **complete and scalable MLLM-based training pipeline**, incorporating **large-scale supervised fine-tuning (SFT)**, and **reinforcement fine-tuning (RFT)**. This pipeline shows that **MLLMs, when properly guided, can directly generate high-quality LaTeX code**, revealing their potential in structured code generation tasks previously considered beyond their typical scope.\n\n2. **Technically Novel Reward Design for RL Fine-tuning**\n\nFrom a technical perspective, our proposed **Visual-Structural Guided RL (VSGRPO) framework** offers a novel and practical advancement. Unlike existing RL strategies that rely on **rule-based textual matching**, which are often brittle or ineffective for long, complex outputs like LaTeX, we propose a **dual-reward system**:\n- A **visual reward**, which evaluates the fidelity of the generated LaTeX by rendering and comparing it to the ground-truth image—providing a more direct and perceptually aligned quality measure.\n- A **structural reward**, which ensures syntactic and layout consistency in the generated code.\nThis dual-reward mechanism not only improves performance on our target task but also has the potential to **generalize to other vision-to-code generation problems**, such as web UI generation or document formatting.\n\n\n\n### Weakness 2: There are very few qualitative figures in this paper.\n\nThank you for pointing this out. We agree that qualitative results are important for providing intuitive insights into model behavior.\n\n\nIn our paper, we **divided the evaluation into simple, medium, and complex tables**, and provided **four representative sets of qualitative visualizations** in the Appendix, covering all three levels of table complexity. These examples were selected to **illustrate typical error patterns** among the compared methods, helping readers better understand the nature of improvements introduced by our approach.\n\n\nWe acknowledge that **four examples are not sufficient** to comprehensively showcase all aspects of model performance. However, due to strict **page limitations**, including a larger number of qualitative examples in the main paper was not feasible.\n\nThat said, we would like to emphasize that our evaluation is not solely reliant on qualitative examples. We have also\nincluded:\n\n- **Extensive quantitative comparisons** across multiple benchmarks.\n\n- A **scientifically designed and rigorously conducted human evaluation**.\n\n- Together, these three forms of evaluation—quantitative, qualitative, and human—**consistently demonstrate the advantages** of our proposed method over existing baselines.\n  \n  \n\n### Weakness 3: The effectiveness of the introduced two rewards is not significant as shown in Tab. 6.\n\nThank you for your feedback. We emphasize that the visual and structural rewards are crucial for improving model performance, both quantitatively and perceptually.\n\nFirst, as shown in Table 6, our dual-reward mechanism significantly improves CW-SSIM during reinforcement learning, boosting scores by 3.5 and 2 points over no RL and text-only rewards respectively. It is important to note that even a 2-point increase in CW-SSIM is non-trivial and translates into **visibly better table rendering quality**, as confirmed by human visual inspection.\n\nSecond, to validate these gains from a human perspective, we conducted a blind human evaluation on 100 randomly selected complex table samples. Annotators compared the generated images to ground-truth images in randomized order, without knowing the source model. As summarized in Table R1, the dual-reward VSGRPO model was preferred 83 times, significantly outperforming the single-reward variants (Visual-only: 59, Structure-only: 44).\n\nThese results confirm that combining visual and structural rewards leads to more faithful and visually accurate LaTeX table generation.\n\nTable R1: Results of human evaluation in the two rewards.\n\n| Model | QwenVL-VSGRPO | QwenVL-Visual | QwenVL-Structure |\n| ----- | ------------- | ------------- | ---------------- |\n| Vote  | 83            | 59            | 44               |\n\n### Question 1: Directly using the continuous similarity value as a reward would be more effective.\n\nThank you for this insightful comment. We indeed explored this direction by conducting an ablation study comparing continuous rewards (based on CW-SSIM values) with binary rewards (based on thresholding) using the Qwen2.5-VL-3B model. The results are summarized in TableR2:\n\n\n\nTableR2: Ablation experiments on the effectiveness of continuous reward value.\n\n| Complex-Dataset                     | CW-SSIM | Compile ratio | TEDS   | TEDS-Structure |\n| ----------------------------------- | ------- | ------------- | ------ | -------------- |\n| QwenVL2.5-VSGRPO-Continuous         | 0.6056  | 0.9861        | 0.9253 | 0.8726         |\n| QwenVL2.5-VSGRPO-Binary             | 0.6145  | 0.9917        | 0.9270 | 0.8721         |\n\n \n\nAs shown, the continuous reward leads to slightly inferior performance. One possible explanation, supported by [1], is that **higher reward variance can lead to stronger learning signals in RL**. Binary rewards, especially when carefully thresholded, may create **sharper gradients and more decisive feedback**, which can help the model escape suboptimal regions in the reward landscape. In contrast, continuous rewards, being bounded and smoother, might lack this reinforcement strength. This remains an **open research question**, and we believe further investigation into the design of reward functions could be a promising direction for future work.\n\n### Reference\n\n[1]Razin, Noam et al. “What Makes a Reward Model a Good Teacher? An Optimization Perspective.” arXiv abs/2503.15477 (2025): n. page.\n\n\n\n### Question 2: The \"w/o SFT\" model's worse performance in Tab. 7 might be due to a smaller training sample size (5,936 vs. full version), rather than solely indicating SFT's necessity.\n\nWe would like to clarify that the same number of training samples was used in both the \"with SFT\" and \"without SFT\" settings. As stated in **Line 169**, “we select **5,936 complex tables** from the training dataset as the training set for VSGRPO.” This exact subset was used consistently across both settings during the reinforcement learning (RL) stage.\n\nTherefore, the difference between the two methods in **Table 7** reflects only the presence or absence of the SFT stage. The performance gap thus **directly highlights the importance of SFT** in preparing the model for effective RL fine-tuning.\n\nWe also note that fewer samples are used in RL than in SFT due to the higher computational complexity of the RL process.\n\n\n### Question 3: The performance gains from VSGRPO in Tab. 6 appear marginal; a discussion from the authors on this would be beneficial.\n\nSee above, in \"Weakness 3\".\n\n\n\n### Question 4: The Abstract and Introduction emphasize Table2LaTeX's challenges—large, nested, and semantically rich tables—yet the Method section lacks specific techniques to address them, creating a disconnect for the reader.\n\nThank you for pointing out this concern. We understand the reviewer’s impression and would like to clarify that the **methodological components in our work are in fact carefully designed to address the core challenges** highlighted in the Abstract and Introduction—namely, **large table sizes, deeply nested structures, and semantically rich content.**\n\n\n\nAs the reviewer noted, this is a **practically valuable but underexplored task**, and part of our contribution lies in **articulating its unique difficulties**. To effectively tackle these, we built a training pipeline that combines the strengths of **Multimodal Large Language Models (MLLMs) with domain-specific enhancements:**\n\n- **Large-scale supervised fine-tuning** (SFT) on diverse and richly annotated table–LaTeX pairs, enabling the model to **generalize across various table structures** and adapt to the specifics of the Table2LaTeX task. This step is critical for equipping the model with robustness in handling a wide range of real-world table layouts.\n\n- **Reinforcement Learning with Complex Examples:**\n  During RL, we **intentionally select complex tables**—those with large size, deep nesting, and rich semantics—as training samples. This design ensures the model is directly optimized for the most challenging aspects of the task. This design achieves superior performance than using."}, {"review_id": "Pte4nMo7zG", "reviewer": "Reviewer_e8fZ", "summary": "This paper addresses the task of table image to LaTeX code generation by a reinforced MLLM framework VSGRPO: fine-tuning an MLLM on a collected large-scale table-to-LaTeX dataset, and introducing a dual-reward reinforcement learning strategy based on GRPO with visual reward (CW-SSIM) and structure reward (TEDS-Structure). Experiments show that the method achieves SOTA performances, especially on complex tables.", "questions": "1.\tHow is gpt-4o used for evaluation? Can a careful prompting strategy achieve better results in Table 1~3? E.g., specify the possible complexities in Table (multirow, multicolumn) in the prompt, give examples or iteratively use gpt-4o for correcting the generated LaTeX answers? \n\n2.\tAre tables with annotations (e.g. bold, red mark, color highlight) considered in the dataset and evaluation? How do they perform with the proposed method?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n\n1.\tThe writing is clear and easy to follow.\n\n2.\tThe paper introduces a reinforced MLLM framework VSGRPO to table2LaTeX, with a dual-reward reinforcement strategy, which is a novel and interesting application to leverage MLLM’s ability to address complex table LaTeX generation.\n\n3.\tThe experimental section is well-designed, validating the effectiveness compared to existing methods, especially in tables with complex level. The ablation study validates the data selection strategies, the reward components, and the necessity of SFT. \n\nWeakness:\n\n1.\tNeed more experiments to support the effectiveness of this method. E.g. closed-source models (gpt-4o) with careful prompting strategy may achieve better results than the proposed method. \n\n2.\tNeed clear explanation of why visual reward and structure reward are chosen. (Is it sufficient for evaluating the quality of generated tables? are there any other judgement way? )", "was_revised": true, "final_justification": "My concerns are addressed, including the effectiveness of this method by comparing with gpt-4o current abilities.\nHowever, the authors have not provided clear explanation of the design (why visual reward and structure reward are chosen).\nGiven this, I keep my score.", "rebuttal": "We appreciate your thorough review and useful comments, particularly highlighting that our method is \"novel and interesting application\", and that our writing is \"clear and easy to follow\". Please find our responses below.\n\n\n\n### Weakness 1: More experiments are needed to demonstrate the method's effectiveness, potentially comparing it against closed-source models like GPT-4o with optimized prompting strategies.\n\nThank you for pointing this interesting question out. In the original evaluation of GPT-4o, we used the following prompt:\n\n\n\n```\n\"<image>\\nConvert this table to LaTeX. \nPlease output only the LaTeX code between \\\\begin{tabular} and \\\\end{tabular}, without any additional text or explanations.\"\n```\n\n\nAs suggested We then provided a more **detailed prompt** tailored to the table image-to-LaTeX task, explicitly indicating the potential complexity of the table structure and including an example LaTeX output. The specific prompt is as follows:\n\n```\nexample_latex = r\"\"\"\nHere is an example of a valid LaTeX table:\n\\begin{tabular}{|c|c|c|}\\hline\nItem & Quantity & Price \\\\\n\\hline\nApple & 2 & \\$1.00 \\\\\nBanana & 3 & \\$0.75 \\\\\n\\hline\n\\end{tabular}\n\"\"\".strip()\nprompt_text = f\"\"\"{question}\nPlease extract only the LaTeX table between \\\\begin{{tabular}} and \\\\end{{tabular}} from the image.\nThe table may contain multi-row or multi-column cells, such as \\\\multirow or \\\\multicolumn.\nEnsure the output has correct alignment and consistent row/column structure.\nDo not include explanations or any extra text — only return valid LaTeX code for the table.\n{example_latex}\"\"\"\n```\n\nHowever, this did not lead to improved performance. We believe this is because the task itself is clearly defined, and the initial prompt was already sufficient for GPT-4o to understand the objective.\n\n\nFurthermore, we explored an **iterative refinement strategy**. Specifically, we fed the model the target table image, the initial LaTeX output, and the corresponding rendered image. The model was then asked to compare the rendered image with the target image—if no errors were detected, it would return the original LaTeX; otherwise, it would correct the LaTeX and return the updated version.\n\nThe specific prompt is as follows:\n\n```\nprompt_text = f\"\"\"The first image is the target table that should be recognized.\nThe second image shows the rendered LaTeX table from the previous prediction.\n\nHere is the previous LaTeX prediction:\n\\\\begin{{tabular}}{{...}}\n{prediction_latex.split(\"\\\\begin{tabular}\")[-1].split(\"\\\\end{tabular}\")[0].strip()}\n\\\\end{{tabular}}\n\nPlease compare the rendered prediction with the target image.\nIf the LaTeX code is already accurate, return it unchanged.\nOtherwise, correct the LaTeX code.\n\nPlease output only the LaTeX code between \\\\begin{{tabular}} and \\\\end{{tabular}}, without any additional text or explanations.\"\"\"\n```\n\n\n\nTable R1 presents the performance of different prompting strategies. The iterative refinement approach yields a marginal improvement in CW-SSIM and achieves a perfect Compile Ratio of 1.0. However, slight declines are observed in TEDS and TEDS-Structure metrics. We attribute this to the model focusing on visual similarity during the correction step, while lacking the ability to explicitly compare LaTeX structural syntax, which may have affected textual structure alignment.\n\nTableR1: Ablation experiments on the effectiveness of different prompt for GPT4o.\n\n| Complex-Dataset            | CW-SSIM | Compile ratio | TEDS   | TEDS-Structure |\n| -------------------------- | ------- | ------------- | ------ | -------------- |\n| GPT4o-original prompt      | 0.4747  | 0.9917        | 0.7745 | 0.5865         |\n| GPT4o-detailed prompt      | 0.4691  | 0.9860        | 0.7479 | 0.5565         |\n| GPT4o-iterative refinement | 0.4799  | 1             | 0.7673 | 0.5857         |\n\n\n\n### Weakness 2: Are tables with annotations (e.g. bold, red mark, color highlight) considered in the dataset and evaluation? How do they perform with the proposed method?\n\nTo further validate the effectiveness of our method as suggested by the reviewer, we additionally evaluated it on two publicly available table datasets: FinTabNet [1], which comprises real-world, complex scientific and financial tables with detailed structural annotations, and SynthTabNet [2], which includes Chinese financial announcement tables. A human evaluation was conducted to assess the visual similarity between the LaTeX-rendered images generated by our method and those produced by existing state-of-the-art MLLMs, using the original table images as ground truth. \n\nWe randomly sampled 100 images from the validation set. For each sample, the ground truth table image is displayed alongside the model's predicted table image, with their positions randomly shuffled and labels hidden. Multiple human assessors independently voted on which image was more visually similar to the target, with the final outcome determined by majority vote. As shown in Tables R1 and R2, our method consistently demonstrates competitive or superior performance across both datasets.\n\nTable R1：Results of human evaluation in FinTabNet[1].\n\n| Model | QwenVL-VSGRPO | QwenVL-SFT | Internvl3-78b | QwenVL2.5-72b | GPT4o |\n| ----- | ------------- | ---------- | ------------- | ------------- | ----- |\n| Vote  | 77            | 50         | 14            | 13            | 2     |\n\n\n\nTable R2：Results of human evaluation in SynthTabNet[2].\n\n| Model | QwenVL-VSGRPO | QwenVL-SFT | Internvl3-78b | QwenVL2.5-72b | GPT4o |\n| ----- | ------------- | ---------- | ------------- | ------------- | ----- |\n| Vote  | 72            | 47         | 12            | 8             | 4     |\n\n### Reference\n\n[1]Zheng, Xinyi et al. “Global Table Extractor (GTE): A Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context.” 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) (2020): 697-706.\n\n[2]Hou, Qiyu et al. “Synthesizing Realistic Data for Table Recognition.” IEEE International Conference on Document Analysis and Recognition (2024).\n\n\n\n### Question 1: For Tables 1-3, how is GPT-4o evaluated? Could a refined prompting strategy, such as specifying table complexities (e.g., multirow, multicolumn), providing examples, or using iterative correction, yield better LaTeX generation results?\n\nSee above, in \"Weakness 1\".\n\n\n\n### Question 2: Are tables with annotations (e.g. bold, red mark, color highlight) considered in the dataset and evaluation? How do they perform with the proposed method?\n\nSee above, in \"Weakness 2\"."}]

Author rebuttal:

Meta-review: This paper tackles the underexplored problem of converting table images to publication-ready LaTeX code. They do this with a MLLM with a novel dual-reward RL strategy (visual fidelity + structural alignment). 

The application, dataset, and the idea of combining the two reward signals works well. There are some concerns about limited novelty beyond reward design, reliance on in-domain (mostly arXiv tables) evaluation. Also, there are concerns about insufficient comparisons to existing RL and CV-based baselines and limited qualitative analysis. As such, the paper remains a borderline case.
Decision: Accept (poster)