DACO: Towards Application-Driven and Comprehensive Data Analysis via Code Generation

Authors: xueqing  wu
rui  zheng
jingzhen  sha
te-lin  wu
hanyu  zhou
mohan  tang
kai-wei  chang
nanyun  peng
haoran  huang
Venue: ICML_cc_2024_Workshop_AI4MATH
Published: 2024-05-29 23:45:44

Abstract: Data analysis is a crucial analytical process to generate in-depth studies and conclusive insights to comprehensively answer a given user query for tabular data. In this work, we aim to propose new resources and benchmarks to inspire future research on this crucial yet challenging and under-explored task. However, collecting data analysis annotations curated by experts can be prohibitively expensive. We propose to automatically generate high-quality answer annotations leveraging the code-generation capabilities of LLMs with a multi-turn prompting technique. We construct the **DACO dataset**, containing (1) 440 databases (of tabular data) collected from real-world scenarios, (2) ~2k query-answer pairs that can serve as weak supervision for model training, and (3) a concentrated but high-quality test set with human refined annotations that serves as our main evaluation benchmark. We train a 6B supervised fine-tuning (SFT) model on DACO dataset, and find that the SFT model learns reasonable data analysis capabilities. To further align the models with human preference, we use reinforcement learning to encourage generating analysis perceived by human as helpful, and design a set of dense rewards to propagate the sparse human preference reward to intermediate code generation steps. Our DACO-RL algorithm is evaluated by human annotators to produce more helpful answers than SFT model in 57.72% cases, validating the effectiveness of our proposed algorithm.

Main paper: 
 introduction: 
 Data analysis is the process of systematically applying statistical and/or logical reasoning to evaluate and comprehend data. Existing literature has investigated answering queries about information given by structural data (e.g., tables) (Chen et al., 2021a; Nan et al., 2022; Lu et al., 2023) . However, they either focus on straightforward factual retrieval or short-form entity/arithmetic resolutions for specifically given entities, while real-world data analysis can involve more complex analytical processes.Take the scenario in Figure 1 as an example: a user is investigating potential age discrimination of a shop. To effectively answer queries such as this one, a chain of mathematical and logical reasoning and interacting with the data is required. For instance, finding 1 is inferred from analyzing age distribution within the membership data ('member' table) , while finding and suggestion 2 are derived by comparing the participants' ages during the happy hours (using both 'member' and 'happy hour member' tables). These rigorous quantitative analyses eventually conclude the opposite to the user's hypothesis. As valuable as the conclusive suggestions such comprehensive analysis can bring, the extensive labor-efforts, hinted by these examples, can hinder the efficiency of gaining intelligence from the data in a competitive business environment. It is thus imperative to devise a system that is able to automate the aforementioned data analysis process.To this end, we introduce a new dataset for this challenging task, DACO, data analysis via code generation. DACO is constructed from a set of diverse real-world databases associated with curated user queries. In light of the previously described labor-intensive challenge, we propose to leverage LLMs with a multi-turn chained prompts to automatically curate the analytical answers for each query. Specifically, our designed framework employs the code generation capabilities of GPT-4 (OpenAI, 2023) for automating the statistical analysis, interleaved with its ability to interpret the obtained quantitative results. The DACO dataset contains 440 databases and 1, 942 associated user queries, which can be used for both model fine-tuning and evaluation. To provide a refined benchmarking resource, we curate a high-Figure 1 : Task overview. Given a user query driven by an application scenario, a data analysis system should produce an answer containing findings and suggestions based on the database. This requires the system to perform mathematical, logical and strategic reasoning, which can be done through invoking external tools such as Python libraries. In this example, finding 1 is inferred from analyzing age distribution within the membership data ('member' table) through mathematical reasoning and analytical reasoning. Finding 2 is inferred by comparing the ages of the happy hours participants (using 'member' and 'happy hour member' tables) through mathematical reasoning, and suggestion 2 is further derived by relating the data to coffee shop business setting through strategic reasoning.quality test set through comprehensive human annotations on a subset of 100 samples. Detailed statistics are in Table 1 .Although LLM exhibit reasonable analytical capabilities (and hence we are able to automatically curate the prerefined data), we empirically find that its generations often fall short of human expectations of what good analyses should be (e.g., relevance to the queries, logical coherency, and higher-level quantitative interpretations). To further improve the generations by aligning the models with corresponding human preferences, we design a reinforcement learning algorithm (DACO-RL) that leverages two newly designed reward models (RM), i.e., contribution RM and regularization RM, to efficiently provide denser feedback. Concretely, the contribution RM heuristically provides better learning signals for the intermediate code generation steps (for more relevant quantitative analysis), while regularization RM helps preventing reward hacking (Skalse et al., 2022) of typical RLHF models (Casper et al., 2023) . We test our algorithm on a fine-tuned CodeGeeX-6B model (Zheng et al., 2023a) , where the win rate of 57.72% in humanannotated pairwise comparison justifies its effectiveness on learning to generate human preferred analyses.In summary, our contributions are three folds: (1) We explore the challenging task of data analysis, where we construct the DACO dataset with our proposed multi-turn prompting technique on a diverse set of real-world databases.(2) We curate a human-refined evaluation set for benchmarking models. (3) We design the DACO-RL algorithm to jointly optimize code generation and answer generation towards human alignment, which demonstrates a significant 57.72% human evaluated win rate on the helpfulness metric. 
the daco task and dataset: 
 As shown in Figure 1 , the input to our task is a database D and a query q, and the output answer a is formatted as two lists of findings and suggestions respectively. In this work, the database D should be a relational database containing multiple named tables.We construct our DACO dataset through four stages: (1) database collection, (2) query collection, (3) automatic annotation collection, and (4) human refinement. The workflow is illustrated in Figure 2 . Our final dataset contains the training, development and test sets with annotations generated by GPT-4, along with a human-refined testing subset. To distinguish the two test sets, we use Test A to represent the automatically annotated set and Test H to represent the human refined one. Statistics are shown in Table 1 .Database collection. We collect databases from two sources: Spider (Yu et al., 2018) and Kaggle (https: //www.kaggle.com/datasets).There are 157 databases collected from Spider, which originally come from university databases, DatabaseAnswers and Wikipedia. We additionally crawl and filter 5,830 databases from Kaggle. From this pool, we manually select a subset of 314 clean and interpretable databases to build our dataset. To maintain the diversity of the resulting database set, 157 of the databases are deliberately chosen near the long tail of its topic distribution. For this, we employ BERTopic (Grootendorst, 2022) to model the topic distribution, which produces in total 160 topics. We take its least frequent 80 topics as the long tail, which covers 26.79% of the total databases. In total, DACO comprises 471 databases, each of which contains on average 2.3 tables. To better visualize the major topic distribution of this selected subset, we again use BERTopic but group these databases into 10 topics. The keywords for top 5 topics are shown in Figure 3 . The leading topic (topic 1) is associated with business setting and consists of 46.52% of the dataset. The remaining nine topics exhibit a relatively even distribution, covering a broad range of domains, including sports (topic 2), healthcare (topic 3), weather (topic 4), and education (topic 5).Query collection. We generate 10 queries for each database by prompting ChatGPT to first assume the role of a database stakeholder and then generate an application-driven query based on the role. To ensure the quality of the query, we perform a manual filtering to the machine generated queries. Specifically, we remove queries that are not driven by realworld applications or cannot be answered by the given reference database. We train a group of 6 annotators to perform such a filtering process. As a result, there are about 42% of the queries removed, where the removal agreement achieves a 0.62 cohen kappa score.After the aforementioned processes, we obtain in total 2,664 queries. We show the top 15 verbs and their top 3 direct noun objectives in Figure 2 . The queries demonstrate a notable level of diversity. The most common type of queries is to request analysis (such as "analyze data" and "identify pattern"), followed by queries aiming to make decisions (such as "determine strategy" and "make decision").As shown in the right half of Figure 4 , we design a pipeline that leverages the code generation capability of LLMs to automate the answer annotation for our DACO dataset. Based on the database and the query, we instruct the LLM to perform data analysis in multiple turns. At each turn, the LLM will produce a python code snippet and take its execution outputs as evidences to reason over and support its follow-up interpretation. After each turn, we prompt the model to decide whether the analysis is sufficiently comprehensive; if deemed sufficient, it terminates the coding turns and produces the final answer.With this pipeline, we instruct GPT-4 to automatically generate all the answer annotations to each query of our dataset, for both the intermediate code and the final analysis answering the queries. To improve the quality of such automatically constructed annotations, we additionally allow GPT-4 to correct its own mistakes when its generated code leads to run-time or syntax error, where only the corrected codes are kept. In total, we obtain 1.9k valid query-answer pairs, each with roughly 3.3 intermediate coding steps.Human refinement. The annotated analyses thus far have been algorithmically generated, where their actual quality are to be further verified. We thus curate a humanrefined subset containing 100 densely human-annotated query-answer pairs. For each query, we sample 3 different analysis candidates using the previously described automated method (with GPT-4). We ask the annotators to evaluate the quality of each machine generated bullet point and categorize each point into one of bad, borderline, or good (associated with scalar scores of 0, 1 and 2 respectively, the higher the better) to each. The bullet points deemed higher quality are then mixed (from the 3 candidate analyses) and refined (with a few manual textual edits) into one final gold-analysis. In the refinement stage, the annotators should first combine all bullet points ranked as "good", remove duplicate points, and reorder the points to maintain a coherent flow. Suppose the number of bullet points are lower than our pre-defined lowest threshold ( 3bullet points per answer), the annotators should select bullet points ranked as "borderline" to augment the answer. we ask a group of 3 internal members to perform refinement. The agreement accuracy of the refinement process (candidate point selection) is 0.83 and the Cohen's Kappa is 0.67.To evaluate the quality of generated data analysis, we use helpfulness as the main metric. Motivated by literature in the data analysis field (Long & Long, 2009) , we define helpfulness as: (1) relevance to the query, (2) effective and insightful data interpretation, and (3) diversity in terms of analysis perspectives. We evaluate helpfulness through pairwise comparison following common approach (Ouyang et al., 2022; Wu et al., 2023; Zheng et al., 2023b) . Given two analyses generated by two different systems, the annotator (either human or simulated by ChatGPT) selects the more helpful one based on our defined criteria. The winning rate of each system is reported as helpfulness score.To obtain a comparable set of numbers for all models, we report the winning rate of each model against Test A and Test H annotations. The upper bound for this score would be 50, as a score of 50 indicates that the model generations are perceived as helpful as annotations. 
daco-rl: 
 While DACO contains mostly algorithmic machine generated analyses, the machine generations without human refinement cannot well align with human preferences (of "good" analyses). Our human refinement process shows that only 47.4% bullet points are "good" points perfectly addressing user queries; the majority of 52.2% are evaluated as "borderline" points that only partially aligns with human expectations; and the remaining 0.4% are considered as "bad".We are therefore interested in investigating whether aligning human preferences via an RLHF fashion could lead to better machine generated analyses. We thus propose the DACO-RL algorithm, which is illustrated in the left half of Figure 4 .Our end goal is to optimize the helpfulness of the analyzed points, which is modelled with an answer RM R a . In addition to this sparse reward signal, we use a heuristically defined contribution RM R c to reward each intermediate step, which is further regularized with a regularization RM R r to prevent reward hacking. In the following sections, we first explain the three reward models sequentially, and eventually explain our whole RLHF pipeline.Notations. We train a language model that interacts with the python interpreter in a conversational manner. Formally, the full dialogue is a list of messages[h, c 1 , o 1 , • • • , c m , o m , a],where h, c i , o i and a stand for human message, code, execution outputs and final answer respectively. The dialogue starts with human message h containing both query q and database meta-data. In the later messages, code c i and final answer a are generated by our model, while execution outputs o i are produced by the python interpreter. To feed the dialogue into the language model, we wrap each message between a begin-of-message phrase (denoted as <BOM>) and the end-of-sentence token, and then concatenate all messages together. We use a different <BOM> for each type of message. Thus, at each turn of the conversation, the language model can decide whether to generate code c or final answer a by generating the corresponding <BOM>.Answer RM. Our end goal is to optimize the helpfulness of answer a, which is modelled by the answer RM R a . Particularly, we model the helpfulness of each single bullet point rather than the full answer. We collect pairwise comparison data of bullet points from ChatGPT to train R a . Given a pair of bullet points where one is preferred over the other, R a is trained to assign a higher score to the preferred bullet point. Given access to the full conversation, R a produces a reward score at the end of each bullet point in the answer.To encourage diversity, we additionally add a repetition penalty. Given a list of findings[f 1 , • • • , f F ],we encourage the i-th finding to be different from all previous findings by imposing a penalty score as i-1j=1 Sim(f i , f j ), where Sim(•, •) computes the similarity between two bullet points. 1 This score is then subtracted from the reward score from R a . The same procedure is applied to the suggestion list.Contribution RM. The answer RM described above is a sparse reward signal that rewards only the answer a but not intermediate coding steps c i . To provide denser reward signal for optimizing c i , we aim to evaluate the helpfulness of each coding step c i . However, annotating helpfulness of intermediate steps is much more difficult. For one, the 1 Sim(•, •) is computed as the cosine similarity of Sentence-BERT (Reimers & Gurevych, 2019) embeddings. We use the ALL-MINILM-L6-V2 model. helpfulness of python code is more vague to define, and the other, evaluating helpfulness requires coding expertise, which makes it more expensive to collect human annotations.To measure the helpfulness of intermediate steps without the huge expense required by human annotations, we heuristically define the helpfulness as how much an intermediate step contributes to the final answer. Concretely, we compute the similarity Sim(a, o i ) between final answer and code outputs to measure the helpfulness of c i . We use Sim(a, o i ) to rank the helpfulness of different steps, and use the comparison pairs between intermediate steps to train the contribution RM R c . Given the conversation, R c predicts the contribution level of each c i as its reward score. R c does not take the execution output o i into consideration when scoring c i , which simplifies model implementation and excludes spurious correlation the model may exploit from the surface form of o i .Regularization RM. The heuristically defined contribution RM R c may not necessarily perfectly align with the true helpfulness. This is due to a known reward misspecification issue termed reward hacking (Skalse et al., 2022) , where the policy model achieves higher scores from the reward model but its true reward decreases. We propose to regularize such behavior with a regularization RM R r . Given the misspecified reward model, R c in our case, we first train an RL model until its generations start to collapse to certain patterns. These patterns typically receive high rewards from R c but do not align well with human expectation, and thus are considered as reward hacking behaviors. We denoted this RL model as π hack . We use π hack to produce generations with typical reward hacking behaviors. These generations are paired with generations without reward hacking behaviors, such as generations from supervised fine-tuning (SFT) model or the pre-human refined answers generated from GPT-4, to further train the regularization RM R r . As an intuition, this means R r will assign lower scores to typical reward hacking behaviors.RLHF. For our whole DACO-RL pipeline, we optimize the language model against the mixture of all three aforementioned rewards, R a , R c and R r .More specifically, we first train a multi-task reward model denoted as R a+c to jointly learn R a and R c . This guarantees that the reward score distribution for R a and R c are relatively close, so the reward signal for a and c i will not overshadow each other. We train a separate model to learn R r , and mix R c+a and R r into 1 2 (R c+a + w r R r ) when rewarding c i . 2 Here, w r is a hyper-parameter to balance the variance of R c+a and R r tuned to maximize reward model accuracy on development set. This guarantees the mixed reward can both encourage high contribution and penalize reward hacking.We use proximal policy optimization (PPO) (Schulman et al., 2017) as our learning algorithm. During training, PPO jointly optimizes a value model V (s) and a policy model π(s). The objective of the policy model is to optimize the generalized advantage estimation (Schulman et al., 2016 ) Ât = ∞ l=0 (γλ) l δ t+l , where δ t = r t + γV (s t+1 ) -V (s t ) for each time step t. When applied to text generation, the generative language model is the policy model π and each generated token is an action. In our multi-turn conversational setting, however, only part of the tokens in the dialogue are generated by language model (concretely, c i and a). In other words, although the language model still takes the full conversation as input, we only compute GAE and gradients over the model generated subsequence, i.e.,[c 1 , • • • , c m , a]. 
experiments: 
 The goal of our experiments is to verify that (1) augmenting language model with code generation can benefit data analysis, and (2) DACO-RL can further boost the answer helpfulness. To this purpose, we perform the following experiments:Evaluated systems. We evaluate the code generation pipeline with ChatGPT and GPT-4. With the answer annota-tion generated by GPT-4, we further train a 6B CodeGeeX2-6B (Zheng et al., 2023a) model through both SFT and DACO-RL. For each of these models, we experiment with a baseline counterpart that does not include code generation and instead directly takes raw table content as input. We additionally experiment with two models specifically pretrained on tabular data, TAPAS (Herzig et al., 2020) and TAPEX (Liu et al., 2022) . TAPAS is a BERT-style model pre-trained to select relevant information from a table based on user query. For our dataset, we first use TAPAS to select relevant information and then use ChatGPT to interpret the selected information. TAPEX is a pre-trained encoder-todecoder model. We fine-tune TAPEX with GPT-4-generated annotations.Evaluation. The main metric we use is pairwise comparison of helpfulness as in Section 2. We use both ChatGPT and trained human annotators for the evaluation. We additionally report BLEU score, entailment score, and helpfulness evaluation for each individual bullet point. These metrics cannot holistically measure the analysis helpfulness, but can provide complementary insights for analyzing model performance. For entailment, we use an off-the-shelf NLI model to compute the probability that the model generation is entailed by the annotation. For point-wise evaluation, we ask the annotator to assign a score chosen from 0, 1 and 2 to each bullet point using the same standard as in human refinement of test set. Our human annotation achieves high agreement of 0.62 Cohen's kappa for pairwise comparison of helpfulness, and 0.65 Cohen's kappa for point-wise helpfulness evaluation.The main results are in Table 3 . We have the following observations:Code generation significantly helps data analysis, especially for zero-shot LLMs. ChatGPT and GPT-4 both enjoy a significant gain in most metrics, especially helpfulness, when equipped with code generation. As in Table 5 , human evaluation further shows that GPT-4 with code generation has a significant 66.41 win rate over GPT-4 w/o code generation. After SFT, code generation brings less significant improvements, because SFT w/o code generation can simulate the behavior in GPT annotations and achieve competitive helpfulness. However, its mathematical and logical reasoning are not supported by code generation, so it produces more hallucination as reflected by the low entailment score on Test H .Our SFT model learns reasonable data analysis capabilities. By simulating GPT-4 behaviors, SFT with code generation achieves a reasonable helpfulness score and outperforms the TAPEX baseline, but still falls short compared to ChatGPT. We also evaluate the error rate (%) of generated Table 5 : Human evaluation. We report human-rated and ChatGPT-rated helpfulness pairwise comparison of two pairs of models: GPT-4 with v.s. without code generation, and DACO-RL v.s. SFT. We also report point-wise evaluation scores scaled into 0 ∼ 2 rated by human annotators. code per step. Our SFT model has an error rate of 3.08%, which is reasonable low, but still much higher than ChatGPT (0.495%) and GPT-4 (0.491%).DACO-RL significantly improves over SFT. As shown in Table 3 and 4 , DACO-RL significantly boosts the performance. Despite the difference in model size, DACO-RL outperforms ChatGPT w/o code generation on helpfulness and entailment metrics. When matched in size, DACO-RL significantly outperforms SFT by 7 points on helpfulness, further demonstrating its benefits. Human evaluation demonstrates a 57.72 win rate of DACO-RL over SFT as in Table 5 .Our qualitative analysis shows that DACO-RL better focuses on user query, while SFT tends to display generic statistics that are less relevant to user query. An example is shown in Figure 5 .We further perform ablation study and report the results in Table 4 . To directly compare the ablation models against SFT, we report the win rate of each model over SFT model. To evaluate the quality of code generation, we use ChatGPT to compare the "informativeness" of generated code against SFT, where informativeness refers to producing informative and insightful code execution outputs while staying relevant to the user query. We report a few additional statistics including the number of code lines and the number of different API calls. We observe that without our proposed two reward models (DACO-RL w/o R c , R r ), using only answer RM significantly hurts the model generation, leading to short and less diverse code generation and thus less helpful final answers. Contribution RM and regularization RM encourage more diverse code generation and more helpful final answer production.Contribution RM favors API calls that extracts important information from tabular data but is also vulnerable to reward hacking. We report the Pearson correlation between API occurrence and contribution RM scores in Table 6 . The functions rewarded most are related to extracting significant features (nlargest, sort values), aggregating results (mean), and displaying specific information (print). In contrast, the least rewarded functions involve displaying generic statistics (describe) and wrangling data (merge, to datetime, is null) since they cannot directly contribute to the user query. Examples of generated code and their contribution RM scores are shown in Figure 6a and 6b. However, we notice the concerningly high correlation between print function and contribution RM scores, which indicates the RL policy may exploit the correlation to hack reward. Figure 6c shows a typical reward hacking case, where the model achieves a high contribution RM score by printing. Our regularization RM learns to discourage such behavior and helps fix the gap.Evaluation on external test sets. (Pasupat & Liang, 2015; Zhong et al., 2017; Iyyer et al., 2017; Yu et al., 2018; Nan et al., 2022) . 
related work: 
 Later benchmarks further require free-form answer generation (Nan et al., 2022) , multi-hop reasoning (Chen et al., 2021a; 2020) and mathematical reasoning (Zhu et al., 2021; Chen et al., 2021b; Lu et al., 2023) . Despite the similar formulation between our task and existing table QA work, their focus are different: most existing table QA datasets focus on obtaining specific information, our data analysis queries can be complex and requires query decomposition and reasoning. Some concurrent work further targets comprehensive table analysis such as correlation analysis and causal reasoning (Nan et al., 2023; Hu et al., 2024; Liu et al., 2024) . The main difference between this work to the concurrent work is our focus on addressing application-driven user queries.Code Generation. Code generation benchmarks have been proposed for general-purpose programming (Austin et al., 2021; Hendrycks et al., 2021) , math problems (Austin et al., 2021) , and data science scenario (Lai et al., 2022; Huang et al., 2022) . Similar to our work, some recent work allows the language model to interact with a code execution environment and receive execution outputs as feedback (Yang et al., 2023; Wang et al., 2023) . The most relevant work is Cheng et al. (2023) that also addresses data analysis via code generation. Given a data analysis query, they use GPT-4 to first generate code and then provide an interpretation of the execution results. While their analysis queries are still relatively simple, this is an early exploration aiming at automating data analysis.RLHF. Reinforcement learning from human feedback (RLHF) aims to optimize a language model against human preference (Ouyang et al., 2022; Touvron et al., 2023; Bai et al., 2022a; b; Ziegler et al., 2019; Wu et al., 2023) . While traditionally RLHF uses a holistic reward score for the entire generation (Ziegler et al., 2019; Ouyang et al., 2022) , recent work shows that dense reward scores for the intermediate reasoning steps are better learning signals (Lightman et al., 2023; Wu et al., 2023) . These work uses expensive human annotation to collect annotations for the dense reward data. Compared to human preference, heuristic rewards are more accessible but may not align well with true reward. This gap can lead to reward hacking (Skalse et al., 2022; Pan et al., 2022) . A common remedy is to use manually designed heuristics to penalize behaviors that potentially harm the true reward (Ouyang et al., 2022; Laidlaw et al., 2023) .In this work, we train the regularizatoin RM to discourage reward hacking. 
conclusion: 
 In this work, we propose a novel and challenging data analysis task, which involves decomposing user query into multiple perspectives, grounding each perspective to the input data and performing logical and mathematical reasoning.To support this task, we build the DACO dataset containing large-scale annotations automatically generated by GPT-4 and a small but high-quality test set with human curated annotations. We employ LLM enhanced with code generation to this task and evaluate three models on our dataset: zero-shot ChatGPT, zero-shot GPT-4 and a 6B SFT model. While GPT-4 consistently performs the best, SFT achieves reasonably good helpfulness with much less computation.On top of the SFT model, we further proposed our DACO-RL algorithm that significantly boosts the human evaluated helpfulness.% Example Large difference (similarity<0.5) 46 As a weather forecaster, I want to study the correlation between weather conditions and bike rentals. v.s.As a tourist attraction planner, I want to evaluate the bike-sharing program's impact on tourism and visitor satisfaction. Similarity = 0.42 Medium difference (0.5<similarity<0.8) 52 As a farmer, I want to determine the suitable fruit varieties to grow on my farm. v.s.  
 
 Appendix: In practice, we find Rr has a large non-zero mean value, so we subtract it before weighted average. 


Decision: The paper constructs a new benchmark for data analysis in real-world databases. It also proposes a reinforcement learning based method that can better address user queries. The author could further improve the paper by reformulating helpfulness, testing with more high-quality code generation models, and clarifying the pipeline.

Reviews: review: paper_summary: paper_summary: - The study's contributions include constructing the DACO dataset using a multi-turn prompting technique on diverse real-world databases, curating a human-refined evaluation set for benchmarking models, and designing the DACO-RL algorithm to optimize code and answer generation towards human alignment, achieving a 57.72% win rate in human evaluations of helpfulness.
strength_weakness: strength_weakness: - Helpfulness is defined as similarity; however, similarity only captures how closely the outputs resemble each other, not their practical utility or relevance to the user's query.

- The DACO-RL method is tested on fine-tuned CodeGeeX-6B model. Other state of the art code generation models could be compared.

- Other datasets could be evaluated.

- The pipeline proposed could use further clarification: "To improve the quality of such automatically constructed annotations, we additionally allow GPT-4 to correct its own mistakes when its generated code leads to run-time or syntax error, where only the corrected codes are kept." It is unclear how the GPT corrects it mistakes?; "At each turn, the LLM will produce a python code snippet and take its execution outputs as evidences to reason over and support its follow-up interpretation. After each turn, we prompt the model to decide whether the analysis is sufficiently comprehensive; if deemed sufficient, it terminates the coding turns and produces the final answer." This needs further clarification, e.g. what is sufficient?;  Throughout the paper, is is unclear what is bullet point refering to?

- "execute the code with a code interpreter" typo --> executes

- The study could consider different presentation using using w/ as opposed to w/o in Table 4 with rewards.

- The proposed method, DACO-RL, is evaluated on two dataset InfiAgent-DA benchmark, and the FeTaQA dataset perfoming well on one over other. Is is unclear why the algorithm was not performing well for information lookup tasks.

- The study finds that Rc and Rr encourage more diverse and helpful code generation, and Rc favors APIs that extract important information from data, such as nlargest, sort_values, mean, and print. It is unclear how these rewards were created and the need based on the related work.

- The dataset can be shared to encourage reproducibility, and metrics/rewards to validate results.
questions: questions: see above
paper_summary: paper_summary: This paper focuses on the challenge of data analysis for tabular data, proposing new resources and benchmarks. Due to the high cost of expert-curated annotations, the authors use LLMs and a multi-turn prompting technique to automatically generate high-quality annotations. They introduce the DACO dataset, which includes 440 real-world databases, around 2,000 query-answer pairs, and a high-quality test set with human-refined annotations. A 6B parameter supervised fine-tuning (SFT) model trained on this dataset demonstrated reasonable data analysis capabilities. Further enhancement using reinforcement learning (DACO-RL algorithm) produced more helpful answers than the SFT model in 57.72% of cases, as validated by human annotators.
strength_weakness: strength_weakness: 1. The selection of the fine-tuned CodeGeeX-6B model for testing the algorithm seems unrepresentative, given the availability of many other high-quality code models of similar scale.
2. The paper explains the selection of 157 databases to maintain diversity, but it does not clarify the criteria for manually selecting 314 clean and interpretable databases from the initial 5,830. What criteria were used for selecting the remaining databases?
3. The similarity measure Sim(a,oi) between the final answer and code outputs is not persuasive for measuring helpfulness. Helpfulness should differ from "similarity," and a more appropriate metric should be proposed. Additionally, the paper introduces "informativeness" as another metric for evaluating code generation quality in the ablation study. What is the relationship between these two metrics? Is there a way to combine them, and what is the rationale for defining a separate metric?
questions: questions: In Section 2, under the Evaluation part, you mention that "The winning rate of each system is reported as a helpfulness score. To obtain a comparable set of numbers for all models, we report the winning rate of each model against TestA and TestH annotations. The upper bound for this score would be 50, as a score of 50 indicates that the model generations are perceived as helpful as annotations." I may not thoroughly understand this. If the upper bound for this score is 50, then why are there many values higher than 50 in Table 3 and Table 4? This seems a bit confusing. Could you please clarify?