OneChart: Purify the Chart Structural Extraction via One Auxiliary Token

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 04:38:42

Abstract: Chart parsing poses a significant challenge due to the diversity of styles, values, texts, and so forth. Even advanced large vision-language models (LVLMs) with billions of parameters struggle to handle such tasks satisfactorily. To address this, we propose OneChart: a reliable agent specifically devised for the structural extraction of chart information. Similar to popular LVLMs, OneChart incorporates an autoregressive main body. Uniquely, to enhance the reliability of the numerical parts of the output, we introduce an auxiliary token placed at the beginning of the total tokens along with an additional decoder. The numerically optimized (auxiliary) token allows subsequent tokens for chart parsing to capture enhanced numerical features through causal attention. Furthermore, with the aid of the auxiliary token, we have devised a self-evaluation mechanism that enables the model to gauge the reliability of its chart parsing results by providing confidence scores for the generated content. Compared to current state-of-the-art (SOTA) chart parsing models, e.g., DePlot, ChartVLM, ChartAst, OneChart significantly outperforms in Average Precision (AP) for chart structural extraction across multiple public benchmarks, despite enjoying only 0.2 billion parameters. Moreover, as a chart parsing agent, it also brings 10%+ accuracy gains for the popular LVLM (LLaVA-1.6) in the downstream ChartQA benchmark.

Main paper: 
 related work 2.1 chart structural extraction: 
 Chart structural extraction aims to extract the main textual and visual elements (such as title, axis names, legends, values, etc.) from chart images through certain methods or models, and organize them in an appropriate way. In the early stage, some non-endto-end methods used keypoint/region detection or segmentation methods, combined with OCR and other methods for information extraction [3, 4, 6, [25] [26] [27] . While these methods have advanced the extraction and analysis of chart structures, their implementation is complex and heavily reliant on the generalization capabilities of traditional techniques. These methods are commonly used for specific types of tables and have lower generalization performance for real-world charts. Currently, several studies [19, 20, 23, [28] [29] [30] tend to use the vision-language models (VLMs) to extract the information contained in visual charts end-to-end and store it in a table format. This approach effectively translates visual data into linguistic formats. Beyond just transforming chart data into tables, ChartVLM [21] also decouples the task of parsing chart titles.Chart reasoning aims to provide relevant descriptions, summaries, QA, or comparative analysis of visual charts. At present, researches are mainly divided into two-stage and end-to-end methods, which treat chart reasoning as a downstream task after extracting key information from charts. PlotQA [4] and ChartQA [3] extract the key information and send it to TableQA models [8, 9, 31] for reasoning and answering. StructChart [23] and DePlot [28] utilize the inference ability of pre-trained large language model [32, 33] with a small number of shots, and use the output as the prompt for reasoning. End-to-end approaches like ChartAssistant [20] , ChartLlama [30] , and ChartVLM [21] start by aligning visual charts with their textual information through pre-training from charts to tables. They then fine-tune various tasks including information extraction, open question answering, and summarization, enabling simultaneous implementation of information extraction and downstream tasks. Clearly, whether using end-to-end or two-stage methods, the structural extraction of information from charts remains fundamental.At present, there is not a lot of open source benchmarking work.ChartQA [3] and PlotQA [4] are mainly suitable for tasks such as chart-to-table and QA summary. Chart-to-Text [34] is mainly suitable for chart-to-table and summary tasks, but the truth quality of the table is poor. The current benchmark works such as StructChart [23] , MMC [35] , and ChartVLM [21] cover more tasks, such as code redrawing, analysis, and type judgment. These works have to some extent promoted the development of chart parsing work. However, the data used to evaluate multimodal large-scale models for charts is still relatively limited in terms of style, type, and language diversity. 
method: 
 In this section, we outline the methodology behind OneChart, structured into five key areas: Data Engine, Architecture, The Auxiliary Token, Training Process, and Inference. Each part plays a critical role, from providing training data and defining structural design to detailing our approach, optimization strategies, and inference results.Chart data generation. Except for chart data from online platforms, such as ChartQA, most chart data is generated using tools such as Matplotlib and Pyecharts. Consequently, we utilize both tools to generate chart images. The charts generated by Matplotlib all contain four fields: "title", "x_axis", "y_axis" and "chart body". Due to the limited functionality of Matplotlib and Pyecharts, we specifically introduce the "chart source" to better fit the real-world chart data style. In addition to taking general rendering methods, we add an additional two-stage rendering method, which first creates the main part except for the title and source, and then adds the title and source to the chart stochastically through graphic stitching. To enhance the visual diversity of charts, we employ randomly generated 16-bit color codes to alter the colors of both text and graphics, beyond the commonly used color schemes. We also offer hundreds of distinct text fonts. Additionally, we introduce considerable variability in the size, direction, and quantity of visual elements.For the generation of pre-training data, the content of the charts is produced randomly. Specifically, for textual information such as title and source, we utilize the Natural Language Processing (NLP) corpus [36] , extracting entries randomly by setting predetermined lengths. The numerical content is generated under controlled distribution to ensure variability. In total, the process yields about 10M chart images alongside their corresponding truth labels. Figure 2 shows the process of our data generation. Data details. The data we generate predominantly fall into two principal categories: barline and pie charts. (1) Barline Charts: These are categorized into five distinct types: Single Column Chart, Multi Column Chart, Single Line Chart, Multi Line Chart, and Combo Chart (Mixed Chart). Each type is evenly split between visualizations that feature numerical labels and those that do not. Currently, our Barline charts can accommodate up to three legends.(2) Pie Charts: In this category, Labeled Pie Charts and Pie Charts with Legends are distributed in equal proportions. Furthermore, in the process of generating content with logical and practical significance using GPT-3.5, we employ varied prompts to facilitate the creation of thematically diverse data across several domains, such as finance, education, technology, among others.OneChart is an end-to-end chart information extraction tool based popular VLM architecture, as shown in Figure 3 . Regarding the selection of VLMs, we choose for the recently released Vary-tiny model [15] , which consists of a vision encoder from SAM-base and a tiny auto-regressive OPT-125M [37] decoder, linked by a linear layer to synchronize their channel dimensions.For the chart image input, we simply resize the image to a fixed resolution 1024×1024 without any extra data augmentation. The model learn to extract Python-dict information with respect to the input image using causal masked language modeling, which can be written as:EQUATIONwhere 𝑤 denotes the target text sequence, 𝑣 denotes the vision features from the vision backbone, 𝑚 denotes the current index of the output target token and 𝐷 denotes the dataset.To enhance the reliability of number values in outputs and mitigate the risk of significant errors, we introduce a special token denoted as "<Chart>" by prefixing its output. This special token will trigger an extra chart's numerical values prediction. As shown in Figure 3 , the corresponding hidden state embedding 𝑡 ∈ R 768 of auxiliary token "<Chart>" is fed into a auxiliary decoder F comprised of 3 layer MLPs and 2 ReLU activation function. The auxiliary output denoted as F (𝑡), where F (𝑡) ∈ R 256 represents the normalized numerical values prediction within a chart. To supervise the numerical output, we incorporate the L1 loss for the number loss L 𝑛𝑢𝑚 during training:EQUATIONwhere 𝑢 represents the min-max normalized ground truth values within a chart image. Each vector of ground truth values is extended to a fixed length of 256 elements through padding with "nan" values to facilitate parallelized training across batches. In the loss function L 𝑛𝑢𝑚 , 𝑚𝑎𝑠𝑘𝑒𝑑 is the non-padded (non-nan) elements, ensuring that the padding does not influence the loss computation. Since OPT-125M in OneChart is a transformer-based model incorporating causal attention, it can attend to the hidden state of the first <Chart> entries when processing the text output in the form of a Python-dict. The auxiliary decoder is an optional component rather than a primary tool during inference. This design maintains the model's versatility and ease of use, akin to the traditional visionlanguage model (VLM). Additionally, the auxiliary number decoder can participate in the computation of confidence scores to help filter its predictions, thereby enhancing the output's reliability. A detailed exploration of this process is presented in Section 3.5. Along all the training stage, we use the template of Vicuna v1 [38] to construct ground truth in a conversation format as "USER: <img> [image] </img> Covert the key information of the chart to a Python-dict. ASSITANT: <Chart> [texts output] 
". We add the "<img>", "<Chart>" and "</img>" as special tokens of the text tokenizer of OPT-125M and we find that it can adapt very well to the Vicuna template. EQUATIONwhere L 𝑡𝑒𝑥𝑡 is cross entropy loss. The Stage1 training uses 32 A100 (80G) GPUs for around 12 hours. Stage2: Warmup auxiliary number decoder. In the second stage, we use about 2.7 millions SFT data as shown in Table 1 to warmup auxiliary number decoder. In this stage, we frozen the vision encoder and only train language model and auxiliary decoder.The training loss is defined as:EQUATIONIn Stage2, we use batch size of 16 and a learning rate of 5e-5 for 1 epoch, this training uses 16 A100(80G) GPUs for around 3 hours.Stage3: Supervised Fine-tuning (SFT). In this stage, we finetuning total model parameters utilizing above SFT data. The training loss L 𝑠𝑡𝑎𝑔𝑒3 is same as L 𝑠𝑡𝑎𝑔𝑒2 . We use batch size of 16 and a learning rate of 5e-5 for 1 epoch, this training uses 24 A100(80G) GPUs for around 4 hours. Subsequently, we employed this finetuned model to evaluate its performance across all benchmarks in Section 4, recording the scores achieved.During inference, the provided chart image is first resized to 1024 × 1024 pixels, with the pixel values scaled to the range of 0 to 1. Subsequently, through a vision encoder, a vision embedding 𝑣 ∈ R 256×768 is integrated into the text embedding of the Vicuna v1 conversation template, as described in Section 3.4. The text in Python-dict format is then serialized for output. The appearance of the "
" special token signifies the end of the output, which we refer to as the raw output. The performance of this raw output is presented in the SE benchmark in Table 2 and the OCR benchmark in Table 2 .Moreover, and critically, we introduce an option to incorporate the output from the auxiliary number decoder to assess the reliability of the raw output. As illustrated in Figure 4 , the raw predict can be easily parsed into a dictionary in python by json.loads() EQUATIONwhere 𝑁 represents the number of numeric values contained within the "value" field of the raw output. The range of S is between [0, 1]. The smaller the value, the closer the self-consistency distance, which can also be explained as the model being more confident in its own output accuracy. Additionally, by setting a threshold, we can quantify the "quality" of the output results, guiding users to selectively trust the outputs. 
experiments 4.1 evaluation metrics: 
 Model should extract the key elements of chart in the defined Python-dict structure. Predicted dictionary should at least include these elements: "title", "source", "x_axis", "y_axis", and "value". We comprehensively evaluate the model's output from two aspects: textual OCR accuracy and structural extraction precision. We also report the accuracy in the popular QA benchmark.Textual OCR. For the chart's textual elements such as "title", "source", "x_axis", "y_axis" in dictionary, we employ an accuracy evaluation based on normalized edit distance [39, 40] , which allows us to measure the closeness of the model-generated text to the ground truth with precision. In order to unify the evaluation indicators as larger is better, we report the value of 1 minus the normalized editing distance as the OCR accuracy, denoted as Reverse Edit distance (RE).Structural Extraction. For the "values" field, which is itself also a Python-dict, represents the entity name and numerical data presented in the chart. To evaluate the accuracy of this crucial component, we concatenate the key and item pairs into tuples and assess them using the mean Average Precision (AP) from the SCRM (Structuring Chart-oriented Representation Metric) [23] . According to the definition of SCRM, three different levels of tolerance (𝑡𝑜𝑙 := {𝑠𝑡𝑟𝑖𝑐𝑡, 𝑠𝑙𝑖𝑔ℎ𝑡, ℎ𝑖𝑔ℎ}) are set for fine-grained evaluation of SE task:EQUATIONwhere 𝐽 𝑡ℎ𝑟 | 𝑡𝑜𝑙 indicates the edit distance threshold between prediction and GT string, 𝑒 𝑡ℎ𝑟 | 𝑡𝑜𝑙 refers to the relative error threshold between prediction numeric value and GT value. QA. With OneChart's support, we evaluate some LLM and VLMs on chart-related QA tasks like ChartQA [3] . In this downstream task, we use their vanilla metrics for a fair comparison with other methods.Traditional QA benchmarks, such as ChartQA and PlotQA, often limit their scope to querying small, isolated segments of information from charts, such as individual values, which may not effectively gauge a model's ability to extract and understand the full spectrum of data presented in a chart. In contrast, we aim to establish a benchmark centered around the Structural Extraction (SE) task, which directly assesses the model's accuracy in converting chart images into structured Python-dict representations.Our benchmark including several parts, which are meticulously composed of images from the following sources, each contributing uniquely to the breadth of the dataset:• ChartQA-SE and PlotQA-SE: The images for these components are derived from the test sets of ChartQA and PlotQA, respectively. Both datasets originate from real-world online platforms, encompassing a wide range of topics such as economy, finance, society, politics, and industry. Most images in ChartQA have specific numerical annotations on them. PlotQA features charts rendered by software based on real-world data, including horizontal bar plots, vertical bar plots, line plots, and scatter plots. There are no specific numerical annotations on the images. This part of the benchmark aims to assess the model's effectiveness in recognizing and extracting information from charts that mirror real-life complexities. • ChartX-SE: This benchmark is derived from the ChartX test set and includes bar, line, and pie charts rendered using Matplotlib. Some charts contain numerical annotations ("barnum", "linenum" and "pie"), while others do not ("bar" and "line"). The selection of ChartX-SE is instrumental in evaluating the model's ability to process and understand conventional chart formats that are prevalent in academic settings. • ChartY-en and ChartY-zh: Recognizing the diversity limitation in existing datasets, we have augmented benchmark with additional pyecharts rendered images, including both English and Chinese languages. Only some images contain numerical annotations. ChartY-en and ChartY-zh are crucial for evaluating the model's adaptability and robustness across different languages, rendering technologies and aesthetics.By redefining the ground truth formats in Python-dict and incorporating a wider variety of chart images, our benchmark aims to provide a more comprehensive and rigorous evaluation of models' capabilities in extracting structured information. In the textual OCR task, OneChart achieves an average OCR score exceeding 90 across all datasets, indicating its capability to deliver clear chart meanings, which provides a solid foundation for downstream QA tasks.We initially conduct two ablation studies on the proposed auxiliary token. The specific experimental results are shown in Table 3 and Table 4 .The presence or absence of the auxiliary token and the impact of its placement on model performance are recorded in Table 4 . It can be seen from it that the placement at the beginning of the sequence is beneficial, yielding higher AP scores across all evaluation tolerances. This can be attributed to the model's ability to leverage causal attention mechanisms to immediately attend to the initial embeddings, directly influencing the text output which dictates the prediction results. Conversely, when the token is placed at the end, the text output cannot effectively utilize these embeddings due to the causal attention's unidirectional nature. Moreover, the presence of a number loss at the end might introduce noise, further impeding the textual learning process. Therefore, we believe that the introduction of the auxiliary token effectively improves the performance of the model, provided that it is placed in a reasonable position.In addition to enhancing the model's parsing ability for chart images, as discussed in Section 3.5, the introduction of the auxiliary token and the design of the corresponding decoder also enable the model to perform the reliability check on its own output. To demonstrate the effectiveness of our designed reliability check method during the inference process, we filter model's original outputs by setting a self-consistency distance threshold of 𝛿 = 0.1 (as detailed in Section 3.5) to obtain purified outputs, and calculate their AP scores alongside original outputs. The results are displayed in Table 3. The "Raw" and "Purified" categories represent the original and filtered outputs, respectively. Notably, after removing "unreliable" results identified through reliability checks, the purified outputs in the ChartQA-SE benchmark show an impressive 9.95% increase in AP@strict compared to the original results. In the other four benchmarks, the purified results show an increase in AP@strict ranging from 2.02% to 8.77%. This highlights that the introduced auxiliary token endows our model with the inherent ability to effectively evaluate its output accuracy, which is quite remarkable.Our model undergoes a three-stage training process. In Stage2, training is confined to the language model and the auxiliary decoder, while in Stage3, the entire model undergoes training. We conduct a thorough analysis of various training methodologies, with the In order to further analyze and demonstrate the effectiveness of the proposed OneChart, it is combined with large language models (LLMs) and vision-language models (VLMs), and compared with existing methods on downstream tasks.Table 6 provides an overview of the comparative analysis of QA task accuracy on the ChartQA dataset. A prominent aspect of our approach is the use of one shot methods to enable GPT-3.5, which lacks visual functionality, to answer chart related questions. Compared with the results obtained from CSV generated using GPT-3.5 and DePlot, our model exhibits significantly better performance (36.9%→57.5%).In addition, the QA capability of VLMs with inherent visual understanding, such as LLaVA [11] , has been enhanced solely through zero shot using the dictionary we have analyzed. When these VLMs are equipped with both Chart images and our structured parsing dict, their overall QA has improved (17.5%→50.1% for LLaVA1.5, 56.0%→67.2% for LLaVA1.6). The results highlight the versatility and effectiveness of our method in promoting more accurate chart understanding and information retrieval between different artificial intelligence systems. 
conclusion: 
 In this study, we introduce OneChart, an innovative framework designed to revolutionize the process of interpreting and extracting information from charts and plots. By leveraging an end-to-end autoregressive method, this approach transforms chart images into Python-dict formatted text, enhancing both efficiency and accuracy. OneChart is reinforced by the introduction of a auxiliary special token and the integration of a custom L1 loss alongside the language cross-entropy loss. This methodology not only minimizes ambiguity in language supervision and improves the accuracy of structural extraction, but also introduces a reliable scoring system to purify the output during inference.Additionally, by establishing the ChartY benchmark, we provide a comprehensive evaluation tool for chart comprehension, addressing the inadequacies of existing QA-type benchmarks. Overall, OneChart represents a substantial advancement in the field, demonstrating an average 20% improvement in information extraction from a variety of chart types, while maintaining minimal model size. Additionally, the positive outcomes of this study highlight the importance of specialized loss functions in model training for specific tasks. In the future, we will focus on expanding OneChart's capabilities to include more diverse and complex chart types and exploring its application in real-world scenarios.  
: 
 Previous methods [3] [4] [5] [6] rely on traditional techniques like detection [7] and Optical Character Recognition (OCR) to transform images into tables, then fine-tuned specialized TableQA models [8, 9] for inference. It is reasonable that comprehensive and accurate perception can effectively assist in information extraction and downstream reasoning tasks. In recent years, with the evolution of visionlanguage models (VLMs) [10] [11] [12] [13] [14] [15] [16] [17] [18] , end-to-end chart understanding models such as MatChart [19] , ChartAst [20] , and ChartVLM [21] started to surface. These models meld vision encoders and autoregressive decoders, aiming for pre-training on image-to-table tasks and fine-tuning for Question and Answer (QA) applications. Despite their advances, accoring to our experiments in Section 4.3, these models with billions of parameters still face limitations in extracting structured information and processing various chart styles, especially in the scenario of parsing charts lacking numerical annotations.We think the performance issue seen in the above VLMs is primarily due to two factors. Firstly, the vision encoder may exhibit the issue of "CLIP bias". Most of the models mentioned employ a CLIP-based [22] ViT as the vision encoder. However, since CLIP-ViT is primarily trained on short, global descriptions of natural image-caption pairs, using it as a vision encoder may lead to the omission of crucial local details necessary for chart parsing. This discrepancy could result in a gap between CLIP-ViT's functionality and tasks that require dense perception (such as chart parsing). Additionally, the training mainly conducted with English captions also affects the effectiveness of the CLIP-ViT in encoding charts embedded in other languages. Secondly, the use of cross-entropy loss in autoregressive decoders presents limitations in accurately capturing or predicting numerical values. For instance, the crossentropy losses for the numbers "7008" and "70.8" shown in Figure 1 can be deceptively similar. This proximity in loss values complicates the model's convergence process and reduces its accuracy in capturing numerical values in charts. Moreover, there are limited diversified public benchmarks in chart parsing filed. ChartQA [3] and PlotQA [4] primarily consist of bar and line charts from online platforms, with very few pie charts included. Similarly, datasets like SimChart9K [23] and ChartX [21] , created using Matplotlib, show limited stylistic variety. The lack of diverse benchmarks for chart analysis hinders the development of related research areas.To tackle these challenges, we introduce a chart converter named OneChart in this work. It captures essential components like chart titles, sources, and aligned numerical data and maps them to a Python-dict format, which can effectively facilitate downstream chart reasoning tasks. To overcome the "CLIP bias" mentioned above and enhance the model's ability to compress chart information, we train a specialized chart encoder from scratch using a large amount of synthetic chart data in both English and Chinese. In parallel, to elevate the model's capability to interpret numerical values in charts and increase the reliability of its numerical output, an additional auxiliary token is introduced. We also develop a decoder specifically for this token and optimize it using a customized L1 loss. Moreover, we present a large-scale chart-to-dict benchmark named ChartY, which comprises approximately 6K charts. These charts span a broad spectrum of topics and types and include content in both English and Chinese languages. In sum, our primary contributions are as follows: Additionally, the integration with popular VLMs enhances accuracy by 32.6% with LLaVA1.5 [11] and 11.2% with LLaVA1.6 [24] on the ChartQA benchmark. 
 
 Appendix:  


Decision: This paper presents a good approach to improving numerical value parsing in charts through the introduction of an auxiliary token and a custom decoder. Results have shown that it has enhanced the performance of recent VLMs. The authors provide a comprehensive evaluation using the newly introduced ChartY benchmark, which covers a diverse range of chart types and languages. Extensive experiments and ablation studies validate OneChart's effectiveness, showing improvements over state-of-the-art methods, particularly for charts without numerical annotations. Despite some limitations, such as the focus on a limited range of chart types (primarily bar, line, and pie charts), the absence of qualitative analysis, and the need for evaluation on more diverse chart-related tasks, the paper still can be of good interest to ACM MM community. The reviewers generally leaned towards acceptance, acknowledging the potential impact and significance of the work. Considering the strong arguments presented in the rebuttal and the updates from the reviewers, particularly the change from weak accept to accept by Reviewer 5YU6, it makes paper a strong candidate for the conference. Therefore, I recommend that this submission be accepted and presented at ACM Multimedia 2024, with the expectation that the authors will address the noted limitations in future research.

Reviews: review: paper_summary: paper_summary: The authors propose OneChart, which is an end-to-end vision-language model that incorporates an auxiliary token and a custom decoder to enhance the accuracy of numerical value parsing in charts. The authors introduce a new benchmark, ChartY, comprising diverse chart types and languages to evaluate the model's performance.
strength_weakness: strength_weakness: 1. The introduction of an auxiliary token and a specialized decoder for numerical value parsing is an innovative idea that addresses a critical challenge in chart understanding.
2. The ChartY benchmark covers a wide range of chart types, topics, and languages, providing a more robust evaluation platform for chart parsing models.
3. The authors conduct extensive experiments and ablation studies to validate the effectiveness of their approach, showcasing significant improvements over state-of-the-art methods, especially for charts without numerical annotations.
4. The authors demonstrate the utility of OneChart in improving the performance of large language models and vision-language models on downstream chart question answering tasks.
limitations: limitations: 1. While the authors claim that OneChart can handle diverse chart types, the experiments primarily focus on bar, line, and pie charts. The model's performance on more complex chart types, such as scatter plots, area charts, or composite charts, is not extensively evaluated.
2. The paper primarily reports quantitative metrics but lacks a qualitative analysis of the model's output, which could provide insights into the strengths and weaknesses of the proposed approach.
paper_summary: paper_summary: This work focuses on Chart parsing. To address a significant challenge due to the diversity of styles, values, and texts, they propose
OneChart: a reliable agent specifically devised for the structural extraction of chart information. Specifically, to enhance the
reliability of the numerical parts of the output, they introduce an auxiliary token placed at the beginning of the total tokens along with
an additional decoder. The numerically optimized (auxiliary) token allows subsequent tokens for chart parsing to capture enhanced
numerical features through causal attention. Experiments show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1) better results with an auxiliary token, i.e., the proposed method
2) offer a comprehensive evaluation tool for chart comprehension
limitations: limitations: 1) the method is effective, yet so simple
paper_summary: paper_summary: This paper proposes an agent specifically devised for the structural extraction of chart information (OneChart), which introduces an auxiliary token and an additional decoder to improve the accuracy of numerical value parsing. Meanwhile, it presents a new benchmark ChartY spanning a wide array of topics, chart types, and languages, offering a robust platform for future research and evaluation. Experiments demonstrate that OneChart achieves SOTA performance in structural extraction and the proposed auxiliary token's effectiveness.
strength_weakness: strength_weakness: 1. The paper proposes a OneChart model designing an auxiliary token to achieve more accurate numerical value parsing.
2. The paper creates a new ChartY benchmark spanning a wide array of topics, chart types, and languages, offering a robust platform for future research and evaluation.
3. The paper conducts comprehensive experiments and analyses to evaluate the proposed OneChart’s chart structural extraction capability and the effectiveness of the auxiliary token.
limitations: limitations: 1. OneChart mainly focuses on Chart Structural Extraction and Factoid Chart Question Answering tasks, it would be beneficial to include more diverse chart-related tasks (e.g., Complex Question Answering on OpenCQA [1], Chart Summarization on Chart-to-Text [2]) to further demonstrate the effectiveness of OneChart. As mentioned in Line 143-144, "This model serves as a foundational framework for other researchers to further develop and enhance.", I wonder whether OneChart have chart reasoning capability to further solve practical issues.
2. Need to add more lastest baselines for evaluation, such as UniChart [3] and MMC [4].
3. OneChart undergoes a three-stage training process with a number of training data and time (mentioned in Section 3.4), can you discuss the efficiency of OneChart during inference, including inference time?
4. It is helpful for us to release your code to further comprehend the proposed OneChart.

**Reference**  
[1] Opencqa: Open-ended question answering with charts, EMNLP, 2022.  
[2] Chart-to-text: A large-scale benchmark for chart summarization, ACL, 2022.  
[3] UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning, EMNLP, 2023.  
[4] MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning, NAACL, 2024.
paper_summary: paper_summary: This paper focus on the task of chart parsing. Specifically, it attempts to convert charts into Python-dict format. To this end, this paper first constructs a large scale datasets based on the gpt3.5. Then, it pretrains the opt-125M using the dataset. In addition, this paper designs an auxiliary number decoder and proposes an auxiliary number decoder to enhance the capacity of the model. Results on several benchmarks show the advantages of the proposed model.
strength_weakness: strength_weakness: 1. This paper attempts an effective data synthesis method.
2. The proposed method outperforms state-of-the-art chart parsing models on several benchmarks, which has proved its effectiveness.
limitations: limitations: 1. The description of the data synthesis method is not enough.
2. This method is not verified on a larger model.
paper_summary: paper_summary: In this paper, the authors propose a char-to-dict model named OneChart. The OneChart model employs an encoder-decoder architecture, using SAM-base as the vision encoder and OPT-125M as the decoder.

The latest chart understanding models are fundamentally based on VLMs, but the authors point out that existing VLMs have two key shortcomings: CLIP bias and insensitivity to numerical values. To address the issue of CLIP bias, the authors re-pretrained OneChart. They utilized large models to generate a substantial amount of JSON data representing charts and leveraged `pyecharts` and `matplotlib` to produce images of the charts. To tackle the insensitivity to numerical values, the authors introduced an auxiliary token and a decoder, requiring the decoder to generate the numerical values in the chart based on the embedding of the auxiliary token.
strength_weakness: strength_weakness: 1. The introduction of the auxiliary token and the decoder is an novel approach that effectively utilizes the characteristics of transformers. 

2. Compared to existing state-of-the-art (SOTA) models, the OneChart model has achieved a significant lead in performance.

3. The paper introduces a new benchmark called ChartY, which can be helpful to the research field.
limitations: limitations: 1. The authors believe that VLMs have a CLIP bias, which affects their performance in chart understanding tasks, but they have not proved the existence of CLIP bias through experiments or case studies.

2. I am concerned whether the OneChart model will overfit to the layout of charts generated by pyecharts and matplotlib.

3. The authors did not compare OneChart with the latest multimodal LLM APIs in their experiments.