NBDESCRIB: A Dataset for Text Description Generation from Tables and Code in Jupyter Notebooks with Guidelines
Authors: Xuye Liu, Tengfei Ma, Yimu Wang, Fengjie Wang, Jian Zhao
Accepted at: acl25

NBDESCRIB: A Dataset for Text Description Generation from Tables and Code in Jupyter Notebooks with Guidelines

Xuye Liu1, Tengfei Ma2, Yimu Wang1, Fengjie Wang1, Jian Zhao1
1University of Waterloo, Canada    2Stony Brook University, USA

Abstract
Generating cell-level descriptions for Jupyter Notebooks, which is a major resource consisting of codes, tables, and descriptions, has been attracting increasing research attention. However, existing methods for Jupyter Notebooks mostly focus on generating descriptions from code snippets or table outputs independently. On the other side, descriptions should be personalized as users have different purposes in different scenarios while previous work ignored this situation during description generation. In this work, we formulate a new task, personalized description generation with code, tables, and user-written guidelines in Jupyter Notebooks. To evaluate this new task, we collect and propose a benchmark, namely NBDESCRIB, containing code, tables, and user-written guidelines as inputs and personalized descriptions as targets. Extensive experiments show that while existing models of text generation are able to generate fluent and readable descriptions, they still struggle to produce factually correct descriptions without user-written guidelines. CodeT5 achieved the highest scores in Orientation (1.27) and Correctness (-0.43) among foundation models in human evaluation, while the ground truth scored higher in Orientation (1.45) and Correctness (1.19). Common error patterns involve misalignment with guidelines, incorrect variable values, omission of important code information, and reasoning errors. Moreover, ablation studies show that adding guidelines significantly enhances performance, both qualitatively and quantitatively.

1 Introduction
In recent years, computational notebooks like Jupyter have become popular among data scientists and machine learning researchers for documenting ideas, writing code, and visualizing results in a single document (Wang et al., 2021a). Description in a notebook provides a rich medium for users to record what the code does and the reasoning behind it. Description is found essential for data scientists to share or reuse code (Zhang et al., 2020; Chattopadhyay et al., 2020). However, research has shown that many data scientists still neglect to write appropriate descriptions for their code in notebooks, especially for code output (i.e. table), as they feel writing description will slow down their coding process (Ramasamy et al., 2023). Rule et al. (2018a) reported that among one million computational notebooks on Github, 25% of them have no comment. Besides, unlike other integrated development environments (IDEs) such as Visual Studio, data scientists working in computational notebooks often write concise descriptions, typically less than 100 words(Wang et al., 2022a). This brevity poses challenges for large language models (LLMs), such as GPT and LLaMA, limiting their ability to generate accurate markdown descriptions(Park and Choi, 2024).

Existing literature has explored techniques to generate descriptions for code snippets or table outputs independently (Richardson et al., 2017; Li et al., 2021; Liu et al., 2018; Wang et al., 2021b; Liu et al., 2021; Wang et al., 2024; Parikh et al., 2020a; An et al., 2022; Zhao et al., 2023b,a; Ding and Xu, 2023; Cm et al., 2023), achieving satisfying performance. However, in real-world applications, documentations are always related to the corresponding code and table output, at the same time. For example, ground truth descriptions require summarization from the code and table output simultaneously, as shown in Section I.

Moreover, while the same code-table output pair could serve different purposes under different user requirements, existing work (Zhao et al., 2023b; Chen et al., 2020; Zhao et al., 2023a; Ding and Xu, 2023; Cm et al., 2023; Muller et al., 2021; Richardson et al., 2017; An et al., 2022; Liu et al., 2021; Khan and Uddin, 2022; Koehn and Knowles, 2017; Pasupat and Liang, 2015; Ye et al., 2023; Zhang et al., 2024; Guo et al., 2024) only focuses
Code
from transformers import ModelResults
df_test=pd.read_csv('/kaggle/input/titanic/test.csv')
model_consumer = ModelResults(model=FilePipeline, features=INITIAL_PRED_FEATURES)
predictor = model_consumer.predictor
predictor._initial_, initial_pred_pipeline,
predictor._predict_, model_consumer._predict_
df_pred = prediction_pipeline.transform(df_test)
df_pred.model

<table>
  <tr>
    <th>passengerid</th>
    <th>class</th>
    <th>sex</th>
    <th>age</th>
    <th>fare</th>
    <th>cabin</th>
    <th>embarked</th>
    <th>y_scores</th>
  </tr>
  <tr>
    <td>892</td>
    <td>3</td>
    <td>male</td>
    <td>41.5</td>
    <td>7.8292</td>
    <td>q</td>
    <td>0</td>
    <td>0.108682</td>
  </tr>
  <tr>
    <td>893</td>
    <td>3</td>
    <td>female</td>
    <td>77.0000</td>
    <td>nan</td>
    <td>s</td>
    <td>-1</td>
    <td>0.156669</td>
  </tr>
  <tr>
    <td>894</td>
    <td>2</td>
    <td>male</td>
    <td>9.6875</td>
    <td>nan</td>
    <td>q</td>
    <td>0</td>
    <td>0.139859</td>
  </tr>
  <tr>
    <td>895</td>
    <td>3</td>
    <td>male</td>
    <td>9.6625</td>
    <td>nan</td>
    <td>s</td>
    <td>0</td>
    <td>0.163866</td>
  </tr>
  <tr>
    <td>896</td>
    <td>3</td>
    <td>female</td>
    <td>12.2875</td>
    <td>nan</td>
    <td>s</td>
    <td>1</td>
    <td>0.553491</td>
  </tr>
</table>

<table>
  <tr>
    <th>Guideline</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Value</td>
    <td>The 'y' column in the output table represents the binary prediction (0 or 1) generated by the model consumer for each passenger, such as passenger 893's score.</td>
  </tr>
  <tr>
    <td>Goal</td>
    <td>Builds a prediction pipeline and applies it to a "test" dataset generating predictions and scores for further analysis.</td>
  </tr>
  <tr>
    <td>Association</td>
    <td>It is clear that fare has nothing to do with age in the Titanic dataset.</td>
  </tr>
  <tr>
    <td>Outlier</td>
    <td>Some features have some missing values in the test set.</td>
  </tr>
  <tr>
    <td>Aggregation</td>
    <td>There are two classes for Embarked feature.</td>
  </tr>
  <tr>
    <td>Reason</td>
    <td>Analyze the data like passenger class, gender, age, and fare to generate predictions and scores with prediction pipeline.</td>
  </tr>
</table>

<table>
  <tr>
    <th>Code Only</th>
    <th>Table Only</th>
  </tr>
  <tr>
    <td>It sets up a machine learning prediction pipeline with initial preprocessing and prediction stages and applies it to test data from a CSV file.</td>
    <td>The table lists passenger data with features like class, sex, age, fare, and predictions with scores for a Titanic dataset.</td>
  </tr>
</table>

Figure 1: An example in our proposed NBDESCRIB dataset, which targets generating high-fidelity and personalized descriptions based on the input of codes, tables, and user-written guidelines. Previous methods and benchmarks focus on understanding the codes or tables only, which makes the generated description unfaithful.

on description generation for either code or table without considering various user requirements. As shown in Figure 1, under different user-written guidelines, the focus of different ground-truth descriptions could differ from each other.

To address the aforementioned issues and step forward in faithful and personalized description generation, in this paper, we propose a challenging user-written guideline-based text generation task focusing on the table and code description generation (TCDG) for Jupyter Notebooks. Given codes and output tables, the goal of TCDG is to produce a concise description under different user-written guidelines. The guidelines will be of a given category corresponding to the type of target text as shown in Table 2.

Moreover, we construct the first benchmark (NBDESCRIB) for TCDG that contains 3,924 processed code-table-description pairs extracted from highly-ranked notebooks from Kaggle competitions and identifies 15 guideline categories of the texts (details in Section3). Specifically, the raw Jupyter Notebook data with tables, code, and associated text from popular Kaggle competitions is crawled. However, the raw data cannot be directly used due to the large amount of noise (Mondal et al., 2023; Lin et al., 2022). For example, "I plan to refine the models by using more sophisticated machine learning techniques." is about personal experiences and future plans, which is not useful for text generation tasks. On the other side, the user-written guidelines and ground-truth descriptions in the markdown cell generally contain multiple different purposes or facts on the tables. To reduce the noise in the raw data, we recruit annotators to first break down the markdown cells and make each piece of text only contain one purpose or fact. For each guideline category, we create the label as well as descriptions, and then curate the tables and filter out the noise text. Finally, the text descriptions in our data are natural, faithful, and specifically targeted under different guidelines (Figure 1).

Next, we evaluate the performance of existing pretrained models. The ablation study shows that guidelines significantly affect the final performance at different levels, which demonstrates the validity of our task. Human evaluation further shows that these advanced models (i.e., CodeT5, Llama, and GPT-4o) still struggle to produce faithful enough results, regardless of high-quality training data. Moreover, common error patterns involve misalignment with guidelines, incorrect variable values, omission of important code information, and reasoning errors.

Based on these results, we integrated our approach into a user-facing application to explore Human-AI collaboration in code documentation. In a follow-up study, users found that the generated documentation reminded them of documenting codes they might have overlooked and increased their satisfaction with their notebooks.

The main contributions of our work are:
• We formulate a novel task, TCDG, and collect a high-quality benchmark, namely NBDESCRIB;
• Experiments show that fine-tuned LMs (CodeT5) outperform powerful pretrained models, i.e.,
GPT-4o and Llama, highlighting the vulnerability of LLMs on TCDG. Ablation studies and downstream user application demonstrate that guidelines significantly enhance model performance, helping users create accurately oriented and reasonable descriptions.
• Error analysis shows that LLMs fail to align with guidelines, understand variable values, and reason.

2 Related Work

In this work, we focus on table and code description generation (TCDG) tasks. Our work is closely related to table-to-text generation and code documentation generation (CDG). Most existing datasets for table-to-text generation (Li et al., 2021; Liu et al., 2018; Parikh et al., 2020a; Dhingra et al., 2019; Zhao et al., 2023b; Chen et al., 2020; Zhao et al., 2023a; Ding and Xu, 2023; Cm et al., 2023; Zhang et al., 2024; Guo et al., 2024; Min et al., 2024) or code documentation (Richardson et al., 2017; An et al., 2022; Liu et al., 2021; Khan and Uddin, 2022; Wang et al., 2022b; Divvedi et al., 2024; Luo et al., 2024) generation contain one text per table or code on a specific topic and schema. For instance, (Suadaa et al., 2021) contains 1.3K table-documentation pairs with richer inference from scientific papers and CodeSearchNet (Husain et al., 2019) contains 2M function-documentation pairs across six programming languages (e.g., java, php, python). Differing from previous CDG and table-to-text datasets, a documentation text can correspond to both code and its table output in ours.

Previous work on table-to-text focuses on text generation for standalone table data. Parikh et al. (2020b) proposed an open domain table-to-text dataset. They collected tables from Wikidepia and paired them with single-sentence documentation. They then requested annotators to revise the Wikipedia candidate sentences into target sentences, rather than writing new ones. Several studies focused on a specific topic and schema such as WEATHERGOV (Liang et al., 2009) and ROBOCUP (Chen and Mooney, 2008), Rotowire (Wiseman et al., 2017), Wikibio (Lebret et al., 2016, Biographies), E2E (Novikova et al., 2016, Restaurants). However, they cannot provide different target texts for various data facts in tables, resulting in too singular results during model training.

Another task similar to table-to-text is table question answering (Pasupat and Liang, 2015; Wang et al., 2018; Nan et al., 2022; Cheng et al., 2021; Ye et al., 2023). While they can locate relevant tables and provide answers by tagging relevant cells, they do not provide a meaningful explanation of different kinds of data facts. There are also other sources of information that may be used in data science projects. Without following appropriate guidelines and integrating these sources, we would not be able to produce satisfactory results. Chen et al. (2019); Gupta et al. (2020) attempted to verify whether a provided textual statement is entailed or refuted by the given table. But they only address verification issues and cannot generate descriptive statements for different data fact types.

Since our work focuses on both code and table, it is essential to discuss related work on CDG, which aims to understand the code and generate the code descriptions. Typical datasets include CodeSearchNet (Husain et al., 2019) and some datasets collected from GitHub (Kanade et al., 2019) or BigQuery (Yue Wang, 2021). Recently, LLMs have been applied to the CDG task such as CodeLlama (Roziere et al., 2023), CuBERT (Kanade et al., 2019), CodeBERT (Feng et al., 2020), GraphCodeBERT (Guo et al., 2020) or GPT (Svyatkovskiy et al., 2020; Lu et al., 2021). Some recent works explore encoder-decoder models such as PLBART (Ahmad et al., 2021), CodeT5 (Yue Wang, 2021), and TreeBERT (Jiang et al., 2021). The documentation in this task often does not cover the facts of the output from the code, only focusing on the description of the code.

Different from the aforementioned works that only focus on one text generation for a single standalone code or table, in our new TCDG task for computational notebooks, code and its table output can correspond to one description, and these descriptions may have many categories depending on the needs of the user. We thus propose to construct a notebookTCDG dataset to handle text generation of multiple guideline categories for code and its table output.

3 TCDG - Task Description

In our task, the model is provided with a long text including a code cell and its table output, as well as the corresponding guideline category description. The guideline indicates the direction of the target description generation, and the specific category is described in Section 4.3. The model is asked to read the input and generate reasonable descriptions
based on the given guideline, code, and table.

3.1 Input
The input to a text generation model consists of an input text and a target document:

(1) Codes and tables from the input texts are extracted from notebooks crawled from the Kaggle website. The code provides the necessary context to understand how the table output was generated. By analyzing the code, one can infer the logic and algorithms applied to the input data, which facilitates accurate interpretation of the table’s contents.

(2) A guideline category serves as a guiding principle for generating the target description. Some descriptions focus on interpreting table content while code snippets also provide essential contextual information. At the same time, while describing the purpose of the code, we also need tables to provide essential data-driven explanations. This setup mimics the real scenario.

3.2 Output
A text generation model is employed to predict the specific guideline category of descriptions. Table code associated Markdown cells are the target documents that we collect since these cells are typically used to provide descriptive text for code and tables. Also, some Markdown cells can be used only for headings in the notebook. To exclude such Markdown cells, search for key characters like #, which generally refers to the titles.

4 NBDESCRIB

4.1 Data Collection
As we are focusing on the description generation in Jupyter Notebooks, we need to crawl a sufficient number of code-table-description pairs first. Publicly shared notebooks on GitHub are often ill-documented (Rule et al., 2018b) and do not have many tables, thus are not suitable for this task.

On the other side, Kaggle allows community members to vote up and down on uploaded notebooks, and findings show that the highly-voted notebooks are of good quality and quantity for code documentation (Wang et al., 2021a; Liu et al., 2021). Thus, we decided to utilize the top-voted and well-documented Kaggle notebooks. We crawled notebooks from seven popular competitions, i.e., seven top popular Kaggle competitions - House Price Prediction, Titanic Survival Prediction, Predict Future Sales, Spaceship Titanic, U.S. Patent Phrase to Phrase Matching, JPX Tokyo Stock Exchange Prediction, and Ubiquant Market Prediction, and built around 4,000 pairs of code-table-description pairs. Links for these competitions can be found in Appendix C. To build this dataset, we also filtered out the description which is not in English. We checked the data policy of each of the competitions, and none of them have copyright issues. We also contacted Kaggle to make sure our data collection complies with the platform’s policy.

<table>
  <tr>
    <th></th>
    <th>Overall</th>
    <th>Train</th>
    <th>Dev</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>Code-Table-Description pairs</td>
    <td>3,924</td>
    <td>2,747</td>
    <td>393</td>
    <td>785</td>
  </tr>
  <tr>
    <td>Code vocabulary size</td>
    <td>3,497</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Table vocabulary size</td>
    <td>16,424</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Description vocabulary size</td>
    <td>4,481</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Avg. # token in Description</td>
    <td>12.41</td>
    <td>12.37</td>
    <td>12.51</td>
    <td>12.52</td>
  </tr>
  <tr>
    <td>Max. # token in Description</td>
    <td>66</td>
    <td>57</td>
    <td>46</td>
    <td>66</td>
  </tr>
  <tr>
    <td>Std. # token in Description</td>
    <td>7.45</td>
    <td>7.43</td>
    <td>7.26</td>
    <td>7.66</td>
  </tr>
  <tr>
    <td>Avg. # token in code cell(s)</td>
    <td>10.68</td>
    <td>11.17</td>
    <td>10.46</td>
    <td>10.22</td>
  </tr>
  <tr>
    <td>Max. # token in code cell(s)</td>
    <td>310</td>
    <td>310</td>
    <td>131</td>
    <td>131</td>
  </tr>
  <tr>
    <td>Std. # token in code cell(s)</td>
    <td>19.54</td>
    <td>21.40</td>
    <td>16.42</td>
    <td>15.64</td>
  </tr>
  <tr>
    <td>Avg. # token in table</td>
    <td>13.85</td>
    <td>13.92</td>
    <td>12.84</td>
    <td>14.11</td>
  </tr>
  <tr>
    <td>Max. # token in table</td>
    <td>272</td>
    <td>272</td>
    <td>97</td>
    <td>261</td>
  </tr>
  <tr>
    <td>Std. # token in table</td>
    <td>17.27</td>
    <td>16.47</td>
    <td>11.97</td>
    <td>21.68</td>
  </tr>
</table>

Table 1: NBDESCRIB dataset statistics.

Cell Matching: We search for codes that produce tables in the notebooks and check if there is a corresponding Markdown cell describing the code and table above. The sentences are also split if there are multiple sentences in the corresponding Markdown cell. However, there are instances where the text content is inaccurate. For example, in texts labelled as value guidelines, the extracted value might be incorrect, or in texts describing Feature Engineer guidelines, the variable name extracted from the code might be misspelled. Thus, we label each sentence and let annotators rewrite it accordingly, since each sentence may have a different description angle. Details about how annotators code the sentence and reach an agreement are shown in Section 4.3. The specific guideline details will be described in the following.

Table Processing: Since the table in Jupyter Notebook is in HTML code, to transfer it into a table format, we use HTMLParser1 to get the data value for each row, column, and their relationship based on the tags, such as <th>, <td>. We first drop their parent tags <table> to simplify the document format. Next, we remove the tags <td> and <th>

1https://docs.python.org/3/library/html.parser.html
from cells to extract variables and corresponding values from the HTML code. Then we concatenate variables and values with pipe("|") to generate table description. An example of table processing is shown in Appendix I.

Table Curation: If the description contains variable names in a table, the corresponding rows and columns containing those variables are extracted to create a new table. If no key variables are included, we keep the original tables. This process aims to minimize the inclusion of irrelevant information. Appendix I.3 provides an example of how to implement table curation.

4.3 Guideline Category Description

Three members of the research team conducted an iterative open-coding process to analyze the collected notebooks. Differing from Wang et al. (2020), where their qualitative coding stopped at the tabular data level, and our analysis goes deep to the granularity of the cell, the cell be used to explain beyond the adjacent code cell whose output is the table: we annotate these cells’ purposes and types of content. Each annotator independently analyzed the same five notebooks to develop a codebook. After discussing and refining the codebook, they again went back to recode those five notebooks and achieved pairwise inter-rater reliability ranged 0.81–0.93 (Cohen’s K). To further determine the correctness of inter-annotator agreement, we let these three annotators analyze another undiscussed five notebooks and get pairwise inter-rater reliability ranging from 0.78 to 0.89 (Cohen’s K) which is convincing to demonstrate the reliability of our codebooks. After getting a reliable agreement, the three coders divided and coded the remaining notebooks. In total, we identified fifteen guideline categories for the content of the markdown cells (Table 2, Appendix E provides examples).

As shown in Table 2, eleven guideline categories mainly focus on the data facts of a table. It is worth noting while these guidelines focus more on the description of the table data, the code still provides contextual information to supplement their description, as shown in Figure 1. Our analysis revealed that markdown cells are mostly used to describe the specific attribute values from the table (Value, 7.29%). Second to the Value category, 6.55% markdown cells are used to specify the outliers from the table output (Outliers).

However, these guidelines do not meet the needs of Jupyter Notebook users. This kind of markdown cells can also be used to mainly explain the beyond adjacent code cells. Even though they mainly focus on the code, a clear understanding of table data is also crucial for understanding the code logic.

We found that some of these markdown cells describe the motivation from the code descriptive text(Goal, 19.64%), to explain the results or critical decisions (Reason, 7.03%), or to describe a combination of mathematical transformations from the code (Feature Engineer, 10.02%). We also found that some markdown cells aim to provide additional context to help with code and table understanding (Complementary Details, 22.17%).

<table>
  <tr>
    <th>Guideline</th>
    <th>#</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Value</td>
    <td>286<br>(7.29%)</td>
    <td>Get the exact data attribute values for a set of criteria.</td>
  </tr>
  <tr>
    <td>Difference</td>
    <td>138<br>(4.02%)</td>
    <td>A comparison between at least two distinct attributes within the target object, or a comparison between the target object and previously measured values.</td>
  </tr>
  <tr>
    <td>Trend</td>
    <td>31<br>(0.79%)</td>
    <td>Indicates a general tendency over a period of time.</td>
  </tr>
  <tr>
    <td>Proportion</td>
    <td>120<br>(3.06%)</td>
    <td>Measure the proportion of selected data attribute(s) within a specified set</td>
  </tr>
  <tr>
    <td>Categorization</td>
    <td>74<br>(1.89%)</td>
    <td>Select the data attribute(s) that meet the condition.</td>
  </tr>
  <tr>
    <td>Distribution</td>
    <td>127<br>(3.29%)</td>
    <td>Show the amount of shared value for the selected data attributes or present a breakdown of all data attributes.</td>
  </tr>
  <tr>
    <td>Rank</td>
    <td>73<br>(1.86%)</td>
    <td>Sort data attributes by their values and display a breakdown of selected attributes.</td>
  </tr>
  <tr>
    <td>Association</td>
    <td>165<br>(4.21%)</td>
    <td>Identify the useful relationship between two or more data attributes.</td>
  </tr>
  <tr>
    <td>Extreme</td>
    <td>227<br>(5.78%)</td>
    <td>Identify the data cases that are the most extreme in relation to the data attributes or within a specific range</td>
  </tr>
  <tr>
    <td>Outlier</td>
    <td>257<br>(6.55%)</td>
    <td>Determine whether there are unexpected data attributes or statistically significant outliers.</td>
  </tr>
  <tr>
    <td>Aggregation</td>
    <td>125<br>(3.19%)</td>
    <td>Calculate the descriptive statistical indicators(e.g., average, sum, count, etc.) based on the data attributes.</td>
  </tr>
  <tr>
    <td>Goal</td>
    <td>771<br>(19.64%)</td>
    <td>Express user’s goal. To say what value or function they tend to expect for the final result.</td>
  </tr>
  <tr>
    <td>Reason</td>
    <td>276<br>(7.03%)</td>
    <td>Express reason using the data from the table or explain the reasons why certain functions are used or why a task is performed.</td>
  </tr>
  <tr>
    <td>Feature Engineer</td>
    <td>393<br>(10.02%)</td>
    <td>The process of selecting, transforming, extracting, combining, and manipulating raw data to generate the desired variables for analysis or predictive modeling.</td>
  </tr>
  <tr>
    <td>Complementary Details</td>
    <td>870<br>(22.17%)</td>
    <td>Express additional contextual elements and supporting information to enhance understanding of the primary content</td>
  </tr>
</table>

Table 2: We identify 15 guideline categories based on the types of descriptions in the Markdown cells which are below the code whose output is a table.

4.4 Train / Dev / Test Splits

Overall, the dataset contains 2747 Code-Table-Description pairs in the training set, 393 pairs in the development set, and 785 pairs in the test set (see Table 1 for more statistics).

5 Experiments

Baselines. We utilized three types of models: a fine-tuned encoder-decoder-based CodeT5, the popular decoder-only LLMs (an off-the-shelf GPT-4o, Llama3, CodeLlama3), and a fine-tuned GPT-3.0. Details are shown in Appendix D.
<table>
  <tr>
    <th>Model</th>
    <th>ROUGE-1</th>
    <th>ROUGE-2</th>
    <th>ROUGE-L</th>
    <th>BERTScore</th>
    <th>Pyramid Evaluation</th>
    <th>G-EVAL:Coherence</th>
    <th>G-EVAL:Correctness</th>
    <th>G-EVAL:Consistency</th>
    <th>G-EVAL:Fluency</th>
    <th>G-EVAL:Orientation</th>
    <th>G-EVAL:Readability</th>
  </tr>
  <tr>
    <th colspan="12">Baseline</th>
  </tr>
  <tr>
    <td>CodeT5</td>
    <td>29.61</td>
    <td>14.38</td>
    <td>17.80</td>
    <td>51.51</td>
    <td>19.37</td>
    <td>3.12</td>
    <td>2.43</td>
    <td>2.28</td>
    <td>3.98</td>
    <td>2.46</td>
    <td>3.88</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>26.06</td>
    <td>3.57</td>
    <td>26.18</td>
    <td>51.62</td>
    <td>18.29</td>
    <td>3.23</td>
    <td>2.78</td>
    <td>2.61</td>
    <td>4.13</td>
    <td>2.42</td>
    <td>4.15</td>
  </tr>
  <tr>
    <td>GPT-3</td>
    <td>19.25</td>
    <td>4.42</td>
    <td>20.73</td>
    <td>51.26</td>
    <td>13.57</td>
    <td>2.64</td>
    <td>2.19</td>
    <td>2.16</td>
    <td>2.89</td>
    <td>2.08</td>
    <td>2.86</td>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>19.57</td>
    <td>2.07</td>
    <td>20.91</td>
    <td>52.41</td>
    <td>13.43</td>
    <td>3.06</td>
    <td>2.38</td>
    <td>2.72</td>
    <td>3.88</td>
    <td>2.21</td>
    <td>3.82</td>
  </tr>
  <tr>
    <td>CodeLlama3</td>
    <td>19.63</td>
    <td>2.46</td>
    <td>21.80</td>
    <td>52.85</td>
    <td>15.72</td>
    <td>2.35</td>
    <td>1.98</td>
    <td>2.06</td>
    <td>2.61</td>
    <td>2.03</td>
    <td>2.96</td>
  </tr>
  <tr>
    <th colspan="12">Ablation Study</th>
  </tr>
  <tr>
    <td>CodeT5<br>without table</td>
    <td>22.40</td>
    <td>9.54</td>
    <td>20.49</td>
    <td>55.91</td>
    <td>18.52</td>
    <td>2.55</td>
    <td>2.06</td>
    <td>2.17</td>
    <td>3.74</td>
    <td>2.02</td>
    <td>3.53</td>
  </tr>
  <tr>
    <td>CodeT5<br>without code</td>
    <td>26.35</td>
    <td>11.72</td>
    <td>24.01</td>
    <td>57.80</td>
    <td>19.14</td>
    <td>3.12</td>
    <td>2.31</td>
    <td>2.59</td>
    <td>3.83</td>
    <td>2.26</td>
    <td>3.69</td>
  </tr>
  <tr>
    <td>CodeT5<br>without guideline description</td>
    <td>25.09</td>
    <td>10.78</td>
    <td>22.61</td>
    <td>56.63</td>
    <td>18.75</td>
    <td>3.07</td>
    <td>2.99</td>
    <td>2.56</td>
    <td>3.90</td>
    <td>2.21</td>
    <td>3.72</td>
  </tr>
  <tr>
    <td>GPT-4o<br>with chain of thoughts</td>
    <td>24.91</td>
    <td>3.82</td>
    <td>25.42</td>
    <td>51.25</td>
    <td>16.91</td>
    <td>3.22</td>
    <td>2.87</td>
    <td>2.69</td>
    <td>3.84</td>
    <td>2.48</td>
    <td>3.86</td>
  </tr>
  <tr>
    <td>GPT-4o<br>with in-context learning</td>
    <td>23.64</td>
    <td>3.52</td>
    <td>24.62</td>
    <td>49.10</td>
    <td>15.92</td>
    <td>3.03</td>
    <td>2.81</td>
    <td>2.54</td>
    <td>3.86</td>
    <td>2.42</td>
    <td>3.92</td>
  </tr>
</table>

Table 3: ROUGE scores and BERTScore for the baselines, our model, and the ablation studies. Results show that this task is challenging though we use it in the state-of-art text generation models.

5.1 Evaluation Metrics
We use the ROUGE scores (ROUGE-1, ROUGE-2, and ROUGE-L) (Lin, 2004), BERTScore (Zhang et al., 2019), and G-EVAL (Liu et al., 2023a) to evaluate our model’s performance with regard to the ground-truth description content. Details are shown in Appendix H.

5.2 Results
The numbers in Table 3 show that this guideline-based text generation task is very challenging, while the fine-tuned CodeT5 obtained the best performance. It is interesting to note that CodeLlama3 has bad performance in this task. The possible reason is that CodeLlama3 focuses mainly on code generation and is not suitable for description generation. As shown in Table 3, CodeT5-Large outperforms the GPT-4o, GPT-3, Llama3, and CodeLlama3 in this task. Additionally, we notice that the ROUGE-L of CodeT5-Large is above 25, and the ROUGE-2 is around 15, indicating that our dataset can produce more accurate and fluent text in response to different guidelines in this task. However, these metrics primarily check for semantic similarity. To further evaluate generated descriptions in diverse dimensions and achieve higher human correspondence, we conduct LLM-based evaluators G-EVAL (Liu et al., 2023b) in six dimensions (Coherence, Correctness, Consistency, Fluency, Orientation, and Readability) on a 5-point Likert Scale. Details of G-EVAL are shown in Appendix H. we can find that CodeT5-Large outperforms the GPT-4o, GPT-3, Llama3, and CodeLlama3 in this task. It is interesting to note that existing LLM can generate fluent descriptions but struggle to produce coherent and consistent descriptions.

Ablation Study: To better understand the impact of each component on this new task, we perform ablation studies(Table 3) to evaluate how table, code, and guideline description contribute to the model performance separately. More concretely, we generate ablation models with the following settings: (1) without table, (2) without code, (3) without guideline description, (4) chain of thought prompting on GPT-4o, (5) in-context learning on GPT-4o.

Since CodeT5 performs best in the task, we use it as a backbone to test its performance without code, table, and guideline description. In general, all the elements contribute to the performance, and removing one element will lead to a significant performance drop.

Note that table content has a bigger effect on model performance compared to code. Code also influences performance by providing the necessary context to infer the logic, which aids in interpreting the table’s content accurately. Guideline description can be seen as a synergy of tables that guide the generation system to generate desirable topics, and without it, the performance is slightly higher than one without any table content.

One intuitive method to enhance the reasoning ability of LLMs is Chain-of-Thoughts (CoT). Here we want to further answer this question: using CoT, can a large language model automatically find an optimum guideline and generate summaries better aligned with human interests? CoT is well known to work well for GPT models, so we experimented on GPT-4o with a CoT prompt containing both an example and middle steps of guessing a guideline (prompts shown in Appendix F). For a fair comparison, we also added the performance of in-context-learning for GPT-4o, by removing the provided guideline and directly providing the example
(prompts shown in Appendix G). The result of CoT improved over in-context learning but is still inferior to the performance of the original GPT-4o with ground-truth guidelines (except ROUGE-2).

Then we analyzed the match rate between guidelines generated through the CoT process and the ground truth. Results show that 72% of the guidelines did not match. Thus, although LLM can often generate readable and decent descriptions for code and table(see results in Table 5 and Table 3), most of the generated descriptions are not as the users expected (see result in Orientation dimension in Table 5). This demonstrates the necessity of guidelines. To fairly compare the generation models and standardize evaluation, we need to specify our guidelines for generating descriptions.

Pyramid Evaluation: We employ an automatic evaluation method based on pyramid evaluation (PyrEval) (Gao et al., 2019) to assess the faithfulness of generation. This metric, which correlates well with human evaluation, extracts and filters key phrases to preserve factual information, offering a more reliable metric to ROUGE. The results in Table 3 show consistent trends between PyrEval and ROUGE, validating the impact of table, code, and guideline descriptions in our ablation studies.

5.3 Human Evaluation

We conducted a human evaluation with 10 participants to assess the performance of various LLM models in generating code documentation. Participants rated 50 pairs of code and documentation across three dimensions: Correctness, Orientation, and Readability. The results showed that the Ground Truth outperformed other baselines. Results and analysis are deferred to Appendix A.

6 Error Analysis

In this section, we analyze some common error cases in this guideline-based text generation task. Some examples can be found in the Appendix B.

(1) Variable values were generated and matched incorrectly. As shown in the example in Table 4, even though CodeT5-Large, GPT-4o, GPT-3, Llama3, and CodeLlama3 are capable of generating keywords such as "highest" based on the "Extreme" guideline, it remains difficult to produce accurate text content based on the variables in the table. For example, CodeT5-Large incorrectly predicted the oldest passenger as 80 years old. Table 4 also has this kind of error.

<table>
  <tr>
    <th colspan="6">Guideline category description</th>
  </tr>
  <tr>
    <td colspan="6">Extreme: Identify the data cases that are the most extreme in relation to the data attributes or within a specific range</td>
  </tr>
  <tr>
    <th colspan="6">Code Cells</th>
  </tr>
  <tr>
    <td colspan="6">
      cols2 = X_test.columns.tolist() # List of column names<br>
      X_test = X_test[cols2] # Applying the new order<br>
      X_test
    </td>
  </tr>
  <tr>
    <th colspan="6">Table</th>
  </tr>
  <tr>
    <th>Passenger</th>
    <th>Sex</th>
    <th>Age</th>
    <th>Embarked</th>
    <th>FamilySize</th>
  </tr>
  <tr>
    <td>0</td>
    <td>0</td>
    <td>34.5</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>1</td>
    <td>1</td>
    <td>47.0</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <td>2</td>
    <td>0</td>
    <td>62.0</td>
    <td>1</td>
    <td>1</td>
  </tr>
  <tr>
    <td>3</td>
    <td>0</td>
    <td>27.0</td>
    <td>2</td>
    <td>1</td>
  </tr>
  <tr>
    <td>4</td>
    <td>1</td>
    <td>22.0</td>
    <td>2</td>
    <td>3</td>
  </tr>
  <tr>
    <th colspan="6">Description</th>
  </tr>
  <tr>
    <th>ground truth</th>
    <td colspan="5">The oldest passenger in X_test dataset is 62 years old</td>
  </tr>
  <tr>
    <th>CodeT5-Large</th>
    <td colspan="5">Oldest person in the titanic was 80 years old and youngest person was less than one year</td>
  </tr>
  <tr>
    <th>GPT-4o</th>
    <td colspan="5">It shows the initial rows of the table providing insights into the age distribution of passengers, with the label drawing attention to the oldest passenger</td>
  </tr>
  <tr>
    <th>GPT-3</th>
    <td colspan="5">The oldest passenger is a man in his fifties</td>
  </tr>
  <tr>
    <th>Llama3</th>
    <td colspan="5">The ages of passengers ranging from 27 to 62 years old with the youngest passenger being 27 years old</td>
  </tr>
  <tr>
    <th>CodeLlama</th>
    <td colspan="5">The average age of the passengers is 34.5 years old.</td>
  </tr>
</table>

Table 4: An example of code-table cells and different models. Note that the table above is the original table, which will be processed by the table curation in Section 4.2 when generating the description. In this case, the curated table only keeps the passenger and age columns.

(2) The generated text focuses solely on the table and ignores important information in the code. In the example from Table 10, ground truth is in the “Extreme” guideline and tends to convey that the first red wine has the highest pH value. However, the table does not have a related keyword “red wine.” And CodeT5 failed to extract this information and also extracted the wrong value. GPT-4o cannot generate correct description either. Example from Table 8 also has this kind of error.

(3) Generating incorrectly oriented text based on guidelines. For example, GPT-3, Llama3, and CodeLlama3 produce text related to “Difference”, “Extreme”, or “Distribution”, but not “Trend” in the example from Table 7. Another example in Table 8, requires models generating text related to “Goal”, but GPT-3 generates text related to “Associ-
ation”, describing the relationship between SibSP and Parch. Llama3 also generates text in “Feature Engineer” but not “Aggregation” in the example from Table 6.

(4) Reasoning error. If operating under Aggregation guidelines, CodeT5, GPT-4o, and GPT-3 may generate incorrect Aggregation data (count, mean, sum). In this example (Table 6), GPT-3 can generate text such as this feature has many null values, but cannot obtain the count of null values.

We manually check 50 examples of CodeT5, GPT-4o, GPT-3, Llama3, and CodeLlama3 models used in our user study and label the type of errors made. The most errors are made when they generate incorrectly oriented text (3rd type) (54.1%). This is because the model tends to generate documents related to the best-trained guideline type in the dataset, such as “Association” or “Value”. There are also two common errors made by generating documents with wrong values (1st type) and wrong reason (4th type) (27% respectively). Such errors are commonly made by generating “Value” or “Aggregation” type documents. There are also 13.5% errors made by generating documents without considering the code. The dataset has numerous examples with insufficient code, leading the model to overlook some code instances.

These errors show that our task and dataset provide some challenges for existing foundation models. We firmly believe that researchers can enhance the existing foundation models in the future when they address the challenges. By building on our work and leveraging the valuable insights gained from it, they can push the boundaries even further, contributing to the continuous evolution of foundation models.

7 Downstream User Application

To demonstrate the application of our benchmark NBDESCRIB, we designed a Jupyter Notebook plugin to assist document writing in data science programming (as shown in Figure 2).

The plugin is triggered when detecting users focusing on a code cell (Figure 2.A). The plugin then reads the contents from the focused cell and its curated table output based on the selected columns and rows and sends the content to the backend. The backend server first generates documentation under different guideline categories using fine-tuned CodeT5-Large(Figure 2.B). In addition, we used prompts to nudge users to explain an output (Figure 2.C). If the user selects one of his preferred candidates, the chosen documentation will be inserted below the code cell.

Our plugin went through several rounds of pilot testing and iterative design. Participants found that it reminded them to document code they would have ignored, reducing the time spent developing documentation while actively exploring the data science task.

![Screenshot of a Jupyter Notebook plugin interface showing options for selecting columns, rows, and guideline categories, with a sample table and code cell.](page_180_120_1087_393.png)

Figure 2: We implement a downstream application as a Jupyter Notebook plugin (A) to assist users in documentation writing for selected rows and columns from different guideline categories (B) and a user-prompt approach (C).
8 Conclusion

In this paper, we formulated a new task, TCDG, that aimed to automatically generate descriptive text for code and table based on the given guidelines for a computational notebook. We collected a large amount of well-documented Jupyter Notebooks from Kaggle, resulting in a new benchmark dataset NBDESCRIB. From our analysis, our task imposed unique challenges to the current generation methods, including CodeT5 and LLMs. This dataset facilitated the task like creation of practical slides (Wang et al., 2023) for Jupyter notebooks and enabled evaluations on faithful, high-fidelity, and factual generation.

Limitations and Potential Risk

As annotations are often performed by multiple individuals, there may be a degree of subjectivity and bias in guidelines and datasets used for text generation. As a result, text can be generated that does not reflect a diverse range of perspectives. Furthermore, although we have automatic evaluation metrics such as ROUGE and BERTScore, the correctness of the generated texts is primarily evaluated through human evaluation, which is accurate but not efficient. Future research should focus on developing methods for automatically evaluating the factual correctness of the generated texts, in order to ensure that the generated text is accurate, unbiased, and representative of a diverse range of perspectives.

One potential risk involves the substantial computational resources needed to run state-of-the-art language models. These resources consume significant amounts of energy, which not only raises the carbon footprint of such research but also leads to environmental degradation.

Acknowledgements

This work is supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant #RGPIN-2020-03966, and Canada foundation for innovation (CFI) John R. Evans Leaders Fund (JELF) #42371. We acknowledge that much of our work takes place on the traditional territory of the Neutral, Anishinaabeg, and Haudenosaunee peoples. Our main campus is situated on the Haldimand Tract, the land granted to the Six Nations that includes six miles on each side of the Grand River.

References

Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2668, Online. Association for Computational Linguistics.

Chenxin An, Jiangtao Feng, Kai Lv, Lingpeng Kong, Xipeng Qiu, and Xuanjing Huang. 2022. Cont: Contrastive neural text generation. Advances in Neural Information Processing Systems, 35:2197–2210.

Souti Chattopadhyay, Ishita Prasad, Austin Z Henley, Anita Sarma, and Titus Barik. 2020. What’s wrong with computational notebooks? pain points, needs, and design opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–12.

David L Chen and Raymond J Mooney. 2008. Learning to sportscast: a test of grounded language acquisition. In Proceedings of the 25th international conference on Machine learning, pages 128–135.

Wenhui Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020. Logical natural language generation from open-domain tables. arXiv preprint arXiv:2004.10404.

Wenhui Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2019. Tabfact: A large-scale dataset for table-based fact verification. arXiv preprint arXiv:1909.02164.

Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2021. Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv preprint arXiv:2108.06712.

Harika Cm, Debasmita Das, Rajesh Kumar Ranjan, Siddhartha Asthana, et al. 2023. Loki: Money laundering report generation via logical table-to-text using meta learning. In Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial Forecasting, pages 104–110.

Hoa Trang Dang. 2005. Overview of duc 2005. In Proceedings of the document understanding conference, volume 2005, pages 1–12. Citeseer.

Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. arXiv preprint arXiv:1906.01081.

Haijie Ding and Xiaolong Xu. 2023. San-t2t: An automated table-to-text generator based on selective attention network. Natural Language Engineering, pages 1–25.
Shubhang Shekhar Dvivedi, Vyshnav Vijay, Sai Leela Rahul Pujari, Shoumik Lodh, and Dhruv Kumar. 2024. A comparative analysis of large language models for code documentation generation. In Proceedings of the 1st ACM International Conference on AI-Powered Software, pages 65–73.

Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547, Online. Association for Computational Linguistics.

Yanjun Gao, Chen Sun, and Rebecca J. Passonneau. 2019. Automated pyramid summarization evaluation. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 404–418, Hong Kong, China. Association for Computational Linguistics.

Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, et al. 2020. Graphcodebert: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366.

Zhixin Guo, Jianping Zhou, Jieying Qi, Mingxuan Yan, Ziwei He, Guanjie Zheng, Zhouhan Lin, Xinbing Wang, and Chenghu Zhou. 2024. Towards controlled table-to-text generation with scientific reasoning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9951–9955. IEEE.

Vivek Gupta, Maitrey Mehta, Pegah Nokhz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324, Online. Association for Computational Linguistics.

Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436.

Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. 2021. Treebert: A tree-based pre-trained model for programming language. In Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161 of Proceedings of Machine Learning Research, pages 54–63. PMLR.

Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2019. Pre-trained contextual embedding of source code. ArXiv, abs/2001.00059.

Junaed Younus Khan and Gias Uddin. 2022. Automatic code documentation generation using gpt-3. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, pages 1–6.

Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics.

Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1203–1213, Austin, Texas. Association for Computational Linguistics.

Tongliang Li, Lei Fang, Jian-Guang Lou, and Zhoujun Li. 2021. Twt: Table with written text for controlled data-to-text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1244–1254.

Percy Liang, Michael Jordan, and Dan Klein. 2009. Learning semantic correspondences with less supervision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 91–99, Suntec, Singapore. Association for Computational Linguistics.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Sherry Lin, Winthrop F Gillis, Caleb Weinreb, Ayman Zeine, Samuel C Jones, Emma M Robinson, Jeffrey Markowitz, and Sandeep Robert Datta. 2022. Characterizing the structure of mouse behavior using motion sequencing. arXiv preprint arXiv:2211.08497.

Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018. Table-to-text generation by structure-aware seq2seq learning. In Proceedings of the AAAI conference on artificial intelligence, volume 32.

Xuye Liu, Dakuo Wang, April Wang, Yufang Hou, and Lingfei Wu. 2021. HAConvGNN: Hierarchical attention based convolutional graph neural network for code documentation generation in Jupyter notebooks. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4473–4485, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a. G-eval: Nlg evaluation using gpt-4 with better human alignment. Preprint, arXiv:2303.16634.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023b. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Linguistics.
Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lindong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. CoRR, abs/2102.04664.

Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. 2024. Repoagent: An llm-powered open-source framework for repository-level code documentation generation. arXiv preprint arXiv:2402.16667.

Dehai Min, Nan Hu, Rihui Jin, Nuo Lin, Jiaoyan Chen, Yongrui Chen, Yu Li, Guilin Qi, Yun Li, Nijun Li, et al. 2024. Exploring the impact of table-to-text methods on augmenting llm-based question answering with domain hybrid data. arXiv preprint arXiv:2402.12869.

Tamal Mondal, Scott Barnett, Akash Lal, and Jyothi Vedurada. 2023. Cell2doc: MI pipeline for generating documentation in computational notebooks. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 384–396. IEEE.

Michael J Muller, April Yi Wang, Steven I Ross, Justin D Weisz, Mayank Agarwal, Kartik Talamadupula, Stephanie Houde, Fernando Martinez, John T Richards, Jamie Drozdal, et al. 2021. How data scientists improve generated code documentation in jupyter notebooks. In IUI Workshops.

Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, et al. 2022. Fetaqaq: Free-form table question answering. Transactions of the Association for Computational Linguistics, 10:35–49.

Jekaterina Novikova, Oliver Lemon, and Verena Rieser. 2016. Crowd-sourcing NLG data: Pictures elicit better data. In Proceedings of the 9th International Natural Language Generation conference, pages 265–273, Edinburgh, UK. Association for Computational Linguistics.

Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020a. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373.

Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020b. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373.

Bumjin Park and Jaesik Choi. 2024. Identifying the source of generation for large language models. arXiv preprint arXiv:2407.12846.

Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, Beijing, China. Association for Computational Linguistics.

Dhivyabharathi Ramasamy, Cristina Sarasua, Alberto Bacchelli, and Abraham Bernstein. 2023. Visualising data science workflows to support third-party notebook comprehension: an empirical study. Empirical Software Engineering, 28(3):58.

Kyle Richardson, Sina Zarriß, and Jonas Kuhn. 2017. The code2text challenge: Text generation in source code libraries. arXiv preprint arXiv:1708.00098.

Baptiste Rozière, Jonas Gehrting, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.

Adam Rule, Aurélien Tabard, and James D Hollan. 2018a. Exploration and explanation in computational notebooks. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–12.

Adam Rule, Aurélien Tabard, and James D. Hollan. 2018b. Exploration and explanation in computational notebooks. CHI ’18, page 1–12, New York, NY, USA. Association for Computing Machinery.

Lya Hulliyyatus Suada, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. 2021. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1451–1465, Online. Association for Computational Linguistics.

Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020. Intellicode compose: Code generation using transformer. ESEC/FSE 2020, page 1433–1443, New York, NY, USA. Association for Computing Machinery.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

April Yi Wang, Dakuo Wang, Jamie Drozdal, Xuye Liu, Soya Park, Steve Oney, and Christopher Brooks. 2021a. What makes a well-documented notebook? a case study of data scientists’ documentation practices in kaggle. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7.
April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael Muller, Soya Park, Justin D. Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. 2022a. Documentation matters: Human-centered ai system to assist data science code documentation in computational notebooks. ACM Trans. Comput.-Hum. Interact., 29(2).

April Yi Wang, Dakuo Wang, Xuye Liu, Lingfei Wu, et al. 2021b. Graph-augmented code summarization in computational notebooks. In IJCAI, pages 5020–5023.

Dakuo Wang, Lingfei Wu, Xuye Liu, Yi Wang, Chuang Gan, Jing Xu, Xue Ying Zhang, Jun Wang, and Jing James Xu. 2024. Learning-based automated machine learning code annotation with graph neural network. US Patent 11,928,156.

Dakuo Wang, Lingfei Wu, Yi Wang, Xuye Liu, Chuang Gan, Si Er Han, Bei Chen, and Ji Hui Yang. 2022b. Learning-based automation machine learning code annotation in computational notebooks. US Patent 11,360,763.

Fengjie Wang, Xuye Liu, Oujing Liu, Ali Neshati, Tengfei Ma, Min Zhu, and Jian Zhao. 2023. Slide4n: Creating presentation slides from computational notebooks with human-ai collaboration. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–18.

Hao Wang, Xiaodong Zhang, Shuming Ma, Xu Sun, Houfeng Wang, and Mengxiang Wang. 2018. A neural question answering model based on semi-structured tables. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1941–1951, Santa Fe, New Mexico, USA. Association for Computational Linguistics.

Yun Wang, Zhida Sun, Haidong Zhang, Weiwei Cui, Ke Xu, Xiaojuan Ma, and Dongmei Zhang. 2020. Datasht: Automatic generation of fact sheets from tabular data. IEEE Transactions on Visualization and Computer Graphics, 26(1):895–905.

Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.

Robert F Woolson. 2007. Wilcoxon signed-rank test. Wiley encyclopedia of clinical trials, pages 1–3.

Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023. Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning. arXiv preprint arXiv:2301.13808.

Shafiq Joty Steven C.H. Hoi Yue Wang, Weishi Wang. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021.

Amy X Zhang, Michael Muller, and Dakuo Wang. 2020. How do data science workers collaborate? roles, workflows, and tools. arXiv preprint arXiv:2001.06684.

Haowei Zhang, Shengyun Si, Yilun Zhao, Lujing Xie, Zhijian Xu, Lyuhao Chen, Linyong Nan, Pengcheng Wang, Xiangru Tang, and Arman Cohan. 2024. Opent2t: An open-source toolkit for table-to-text generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 259–269.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.

Yilun Zhao, Zhenting Qi, Linyong Nan, Lorenzo Jaime Yu Flores, and Dragomir Radev. 2023a. Loft: enhancing faithfulness and diversity for table-to-text generation via logic form control. arXiv preprint arXiv:2302.02962.

Yilun Zhao, Zhenting Qi, Linyong Nan, Boyu Mi, Yixin Liu, Weijin Zou, Simeng Han, Ruijie Chen, Xiangru Tang, Yumo Xu, et al. 2023b. Qtsumm: Query-focused summarization over tabular data. In The 2023 Conference on Empirical Methods in Natural Language Processing.
A Appendix: Human Evaluation

We also conduct a human evaluation to further evaluate whether those models can generate reasonable and oriented text with our dataset.

Participants: Our human evaluation task involves reading the code snippet, its output table, and a guideline description and rating the generated documentation from them. We recruited 10 participants (6 male, 4 female) who are fluent English speakers with around six years of experience in the data science and machine learning field. We conducted a rigorous qualification process, evaluating their knowledge of coding practices and data analysis, to ensure high-quality annotations. We hired them by sending invited emails to graduate students who have experience in data science work. We allocated up to 90 minutes for each participant to complete the study, and for their valuable time and input, each participant received a compensation of $20.

Task: We randomly selected 50 pairs of documentation and code from NBDESCRIB. Note that each pair has only one code, one table, and one guideline description, but may have one descriptive text. Each participant is assigned 50 pairs. Each pair is evaluated by 10 individuals. In each trial, a participant reads 8 candidate descriptions for the same code snippet-table-guideline: one by GPT-4o with chain-of-thought, one by GPT-4o with in-context-learning, one as the ground truth, and another five by five different models. The order of these eight is also randomized, so participants do not know which descriptive text is from which model. The participant is asked to rate the 8 documentation texts along three dimensions using a five-point Likert scale from -2 to 2. For the human study, we adopted a -2 to 2 scale because it provides participants with a clear central point (0) to indicate neutrality, while the negative and positive ranges (-2 and 2) allow for stronger differentiation in agreement and disagreement. This scale helps human evaluators express nuanced opinions about correctness, orientation, and readability, reducing ambiguity and improving clarity during evaluation.

• Correctness: The generated description matches the code and table content.
• Orientation: The generated description is written in the correct guideline category.
• Readability: The generated description is in readable English grammar and words.

Evaluation Results: We conducted Wilcoxon tests (Woolson, 2007) with a significance level of 0.05 to compare the performance of Ground Truth against CodeT5-Large, GPT-3, and GPT-4o in the Correctness, Orientation, and Readability dimensions. The Wilcoxon test is a non-parametric statistical test used to compare two paired groups of data. The obtained p-values indicate the probability of observing the reported differences if there were no true differences between the models. The results indicate significant differences in the Correctness dimension, where Ground Truth outperforms CodeT5-Large (V = 5628, p = 1.74e-30), GPT-3 (V = 5635, p = 5.46e-31), GPT-4o (V = 5647, p = 4.79e-30), Llama3 (V = 5732, p = 3.32e-30), and CodeLlama (V = 5948, p = 1.27e-30). It is also worth noting that CodeT5 performs slightly better than GPT-4o in terms of correctness from Table 5, possibly because it handles code-containing data sets better.

Similarly, in the Orientation dimension, Ground Truth surpasses CodeT5-Large (V = 3567, p = 1.59e-20), GPT-3 (V = 3731, p = 1.77e-20), GPT-4o (V = 3559, p = 1.82e-20), Llama3 (V = 3722, p = 1.89e-20), and CodeLlama (V = 3883, p = 1.33e-20).

For the Readability dimension which considers whether the generated description is a valid English sentence, Ground Truth outperforms all models once again: CodeT5-Large (V = 4363, p = 1.40e-7), GPT-3 (V = 4030, p = 3.81e-14), GPT-4o (V = 4451, p = 3.01e-10), Llama3 (V = 4556, p = 2.38e-13), and CodeLlama (V = 4573, p = 1.62-15). It is also worth noting that GPT-4o with chain-of-thought and in-context-learning have worse performance than GPT-4o which demonstrates that guidelines can better assist the description generation for code and table.

The statistically significant p-values (all below 0.05) in each dimension demonstrate it is difficult to meet the correctness, orientation, and readability requirements of the user due to the difficulty of the task. Future work can be accomplished by designing an innovative model to address this challenge.
<table>
  <tr>
    <th>Model</th>
    <th>Correctness</th>
    <th>Orientation</th>
    <th>Readability</th>
  </tr>
  <tr>
    <td>Groundtruth</td>
    <td>\( \overline{x} = 1.19, \sigma=1.32 \)</td>
    <td>\( \overline{x} = 1.45, \sigma=1.02 \)</td>
    <td>\( \overline{x} = 1.61, \sigma=0.78 \)</td>
  </tr>
  <tr>
    <td>CodeT5-Large</td>
    <td>\( \overline{x} = -0.43, \sigma=1.55 \)</td>
    <td>\( \overline{x} = 1.27, \sigma=1.11 \)</td>
    <td>\( \overline{x} = 0.55, \sigma=1.60 \)</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>\( \overline{x} = -0.42, \sigma=1.67 \)</td>
    <td>\( \overline{x} = 1.18, \sigma=1.44 \)</td>
    <td>\( \overline{x} = 0.53, \sigma=1.51 \)</td>
  </tr>
  <tr>
    <td>GPT-3</td>
    <td>\( \overline{x} = -0.41, \sigma=1.58 \)</td>
    <td>\( \overline{x} = 0.98, \sigma=1.39 \)</td>
    <td>\( \overline{x} = 0.51, \sigma=1.61 \)</td>
  </tr>
  <tr>
    <td>Llama3</td>
    <td>\( \overline{x} = -0.44, \sigma=1.41 \)</td>
    <td>\( \overline{x} = 1.03, \sigma=1.42 \)</td>
    <td>\( \overline{x} = 0.51, \sigma=1.82 \)</td>
  </tr>
  <tr>
    <td>CodeLlama</td>
    <td>\( \overline{x} = -0.61, \sigma=1.39 \)</td>
    <td>\( \overline{x} = 0.88, \sigma=1.36 \)</td>
    <td>\( \overline{x} = 0.52, \sigma=1.69 \)</td>
  </tr>
  <tr>
    <td>GPT-4o with chain-of-thought</td>
    <td>\( \overline{x} = -0.39, \sigma=1.54 \)</td>
    <td>\( \overline{x} = 0.94, \sigma=1.35 \)</td>
    <td>\( \overline{x} = 0.48, \sigma=1.66 \)</td>
  </tr>
  <tr>
    <td>GPT-4o with in-context-learning</td>
    <td>\( \overline{x} = -0.35, \sigma=1.60 \)</td>
    <td>\( \overline{x} = 0.91, \sigma=1.28 \)</td>
    <td>\( \overline{x} = 0.46, \sigma=1.83 \)</td>
  </tr>
</table>

Table 5: Human Evaluation Result.

B Appendix: Guideline-Code snippets-Table-Documentation Pair Examples

Guideline description
Aggregation: Calculate the descriptive statistical indicators (e.g., average, sum, count, etc.) based on the data attributes

Code Cells
for dataset in [titanic_train,titanic_test]:
    dataset['IsAlone'] = 0
    dataset.loc[dataset['FamilySize'] == 1, 'IsAlone'] = 1
titanic_train.head(3)

<table>
  <tr>
    <th></th>
    <th>PassengerId</th>
    <th>Survived</th>
    <th>Sex</th>
    <th>Ticket</th>
    <th>Cabin</th>
  </tr>
  <tr>
    <td>0</td>
    <td>1</td>
    <td>0</td>
    <td>male</td>
    <td>A/5 21171</td>
    <td>NaN</td>
  </tr>
  <tr>
    <td>1</td>
    <td>2</td>
    <td>1</td>
    <td>female</td>
    <td>PC 17599</td>
    <td>C85</td>
  </tr>
  <tr>
    <td>2</td>
    <td>3</td>
    <td>1</td>
    <td>female</td>
    <td>STON/O2.3101282</td>
    <td>NaN</td>
  </tr>
</table>

Documentation
ground truth
Cabin feature has 2 missing values
CodeT5-Large
These five passengers are in the same cabin
GPT-4o
It adds an IsAlone column to the Titanic datasets and displays the first three rows data
GPT-3
The cabin column has many null values
Llama3
Creates a new column to indicate if passengers are traveling alone or with family.
CodeLlama
Average fare paid by the passengers

Table 6: An example of code and its table output pair(cabin is the Cabin number). In this case, the curated table only keeps the cabin columns.

Guideline description
Trend: Indicates a general tendency over a period of time

Code Cells
train[['Pclass', 'Survived']].groupby(['Pclass'], as_index=False).mean().sort_values(by='Survived', ascending=False)

<table>
  <tr>
    <th>Pclass</th>
    <th>Survived</th>
  </tr>
  <tr>
    <td>0</td>
    <td>0.629630</td>
  </tr>
  <tr>
    <td>1</td>
    <td>0.472826</td>
  </tr>
  <tr>
    <td>2</td>
    <td>0.242363</td>
  </tr>
</table>

Documentation
ground truth
Higher class survived more
CodeT5-Large
The survived rate for this new feature varies whether it is a tend to be a higher class passengers or a very high class passengers
GPT-4o
it presents the survival rates for each passenger class, demonstrating a clear pattern where lower class numbers correspond to higher survival rates
GPT-3
the survived rate for the pclass 1 is higher than the pclass 2
Llama3
First-class passengers having the highest survival rate and third-class passengers having the lowest.
CodeLlama
The most common class is 3rd class with 24% of the survived passengers

Table 7: An example of code and its table output pair (Pclass is the ticket level. The smaller the value, the higher the class). In this case, the table is not curated because ground truth have class and survived variables.
Guideline description
Goal: Express user’s goal. To say what value or function they tend to use for the later research

Code Cells
for dataset in [titanic_train,titanic_test]:
    dataset['FamilySize'] = dataset['SibSp'] + \
        dataset['Parch'] + 1
titanic_train.head(3)

<table>
  <tr>
    <th>PassengerId</th>
    <th>Survived</th>
    <th>SibSp</th>
    <th>Parch</th>
    <th>IsAlone</th>
    <th>FamilySize</th>
  </tr>
  <tr>
    <td>0</td>
    <td>1</td>
    <td>0</td>
    <td>1</td>
    <td>0</td>
    <td>2</td>
  </tr>
  <tr>
    <td>1</td>
    <td>2</td>
    <td>1</td>
    <td>0</td>
    <td>0</td>
    <td>2</td>
  </tr>
  <tr>
    <td>2</td>
    <td>3</td>
    <td>1</td>
    <td>0</td>
    <td>1</td>
    <td>2</td>
  </tr>
</table>

Documentation
ground truth
Checking if the person is alone or with a family by checking the SibSp and Parch column in Titanic passenger data and add a FamilySize column in titanic_train and titanic_test datasets

CodeT5-Large
We can create another feature called IsAlone

GPT-4o
The goal is to create a new attribute called family size in both the titanic train and titanic test datasets

GPT-3
we can see that sib sp and parch are highly correlated

Llama3
Understand the relationships between different passenger characteristics in the Titanic dataset

CodeLlama
To predict the family size of a passenger in the Titanic disaster

Table 8: An example of code and its table output pair. In this case, the curated table only keeps the passenger, SibSp, Parch, and FamilySize column.

Guideline description
Extreme: Identify the data cases that are the most extreme in relation to the data attributes or within a specific range

Code Cells
Tuned_rf = tune_model(rf)

<table>
  <tr>
    <th>Model</th>
    <th>Accuracy</th>
    <th>AUC</th>
    <th>Recall</th>
    <th>...</th>
  </tr>
  <tr>
    <td>0</td>
    <td>0.7895</td>
    <td>0.8864</td>
    <td>0.6250</td>
    <td>...</td>
  </tr>
  <tr>
    <td>1</td>
    <td>0.9474</td>
    <td>1.000</td>
    <td>0.8750</td>
    <td>...</td>
  </tr>
  <tr>
    <td>2</td>
    <td>0.8947</td>
    <td>0.9318</td>
    <td>0.8750</td>
    <td>...</td>
  </tr>
  <tr>
    <td>3</td>
    <td>0.7368</td>
    <td>0.8523</td>
    <td>1.0000</td>
    <td>...</td>
  </tr>
  <tr>
    <td>4</td>
    <td>0.8947</td>
    <td>0.8667</td>
    <td>0.8889</td>
    <td>...</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.9474</td>
    <td>0.9111</td>
    <td>0.7778</td>
    <td>...</td>
  </tr>
  <tr>
    <td>6</td>
    <td>0.8947</td>
    <td>0.9111</td>
    <td>0.7778</td>
    <td>...</td>
  </tr>
  <tr>
    <td>7</td>
    <td>0.7895</td>
    <td>0.8333</td>
    <td>0.6667</td>
    <td>...</td>
  </tr>
  <tr>
    <td>Mean</td>
    <td>0.8617</td>
    <td>0.9189</td>
    <td>0.8222</td>
    <td>...</td>
  </tr>
  <tr>
    <td>SD</td>
    <td>0.0675</td>
    <td>0.0556</td>
    <td>0.1348</td>
    <td>...</td>
  </tr>
</table>

Documentation
ground truth
Model 1 has the highest accuracy while the code tune with random forest

CodeT5-Large
The highest accuracy is 0.8442

GPT-4o
It shows the accuracy values for different sequence numbers where the mean accuracy is 0.8617 and the standard deviation is 0.0675

GPT-3
the highest accuracy is 0.7895

Llama3
Identify the most extreme data points in relation to their corresponding data attributes

CodeLlama
The Random Forest model is the best model for this dataset.

Table 9: An example of code and its table output pair. In this case, the curated table only keeps the Accuracy and Model columns.
Guideline description

Extreme: Identify the data cases that are the most extreme in relation to the data attributes or within a specific range

Code Cells

df =
→ pd.read_csv("../input/red-wine-quality-cortez-et-al-2009/winequality-red.csv")
df.head()

<table>
  <tr>
    <th>Wine</th>
    <th>fixed acidity</th>
    <th>volatile acidity</th>
    <th>pH</th>
    <th>sulphates</th>
    <th>alcohol</th>
    <th>quality</th>
  </tr>
  <tr>
    <td>0</td>
    <td>7.4</td>
    <td>0.70</td>
    <td>4.01</td>
    <td>0.56</td>
    <td>9.4</td>
    <td>5</td>
  </tr>
  <tr>
    <td>1</td>
    <td>7.8</td>
    <td>0.88</td>
    <td>3.20</td>
    <td>0.68</td>
    <td>9.8</td>
    <td>5</td>
  </tr>
  <tr>
    <td>2</td>
    <td>7.8</td>
    <td>0.76</td>
    <td>3.26</td>
    <td>0.65</td>
    <td>9.8</td>
    <td>5</td>
  </tr>
  <tr>
    <td>3</td>
    <td>11.2</td>
    <td>0.28</td>
    <td>3.16</td>
    <td>0.58</td>
    <td>9.8</td>
    <td>5</td>
  </tr>
  <tr>
    <td>4</td>
    <td>7.4</td>
    <td>0.70</td>
    <td>4.01</td>
    <td>0.56</td>
    <td>9.4</td>
    <td>5</td>
  </tr>
</table>

Documentation

ground truth    The first red wine has the highest pH value
CodeT5-Large    the biggest ph is 3.20
GPT-4o          It determine how these values deviate significantly from the normal range, providing insights into outliers.
GPT-3           data frame sort ph values
Llama3          Identifies the top 4 most extreme data points for each attribute.
CodeLlama       The wine quality is a continuous variable with a range of 3 to 9

Table 10: An example of code and its table output pair. In this case, the curated table only keeps the Wine and pH column.
C Appendix: Kaggle competition link

We crawled highly voted notebooks from seven top popular Kaggle competitions - House Price Prediction2, Titanic Survival Prediction3, Predict Future Sales4, Spaceship Titanic5, U.S. Patent Phrase to Phrase Matching6, JPX Tokyo Stock Exchange Prediction7, Ubiquant Market Prediction8

D Appendix: Detail of Baseline Models

CodeT5 is a large pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Since CodeT5 is a competitive code-related text generation model, when using this model in our task, we converted the relevant table and guideline category description into an inline comment in code and then fine-tuned the model. It has 770 million parameters and the computational budget is around 3 hours.

GPT-3 (Generative Pre-training Transformer 3) is an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model. To use the GPT3 model for our task, we combine guideline description, code, and table as input text. It has 175 billion parameters. The computational budget is around 1 hour. To use the GPT-3 model, we register an account on OpenAI and use the related API (openai api fine_tunes.create9) to fine-tune the GPT-3 model. Also, we built a dataset suitable for GPT-3 training, which can shared with the public.

GPT-4o is an advanced iteration of the GPT-3 model with around 12 billion parameters and a default backend of free ChatGPT. The computational budget is around 1 hour and 15 minutes. Its ability to comprehend context, generate coherent and contextually relevant responses, and perform a wide array of language-related tasks is further refined. It is an easily accessible tool and has been widely used in real life. So we add it as an advanced baseline.

Llama3 (Touvron et al., 2023). We use Llama3 3.1-70B in this task. It is an advanced language model with approximately 70 billion parameters. Its default backend is designed for efficiency and scalability. The computational budget for Llama3 is approximately 1 hour and 30 minutes. Its ability to understand context, generate coherent and contextually relevant responses, and perform a wide range of language-related tasks is significantly enhanced. Llama3 is a powerful and accessible tool, widely used in various applications. Therefore, it is included as an advanced baseline.

CodeLlama3 (Roziere et al., 2023) is an advanced version of the Llama3 model. We use the 13B version in this task. The model aims to handle complex code generation and comprehension tasks efficiently. The computational budget of CodeLlama3 is about 2 hours. CodeLlama3 performs well in understanding intricate programming contexts, generating accurate and contextually appropriate code, and performing various code-related tasks. Its accessibility and versatility have made it a valuable tool for developers and researchers, serving as an advanced baseline for code-centric applications.

E Appendix: Guideline Categories

2https://www.kaggle.com/c/house-prices-advanced-regression-techniques
3https://www.kaggle.com/c/titanic/
4https://www.kaggle.com/competitions/competitive-data-science-predict-future-sales
5https://www.kaggle.com/competitions/spaceship-titanic
6https://www.kaggle.com/competitions/us-patent-phrase-to-phrase-matching
7https://www.kaggle.com/competitions/jpx-tokyo-stock-exchange-prediction
8https://www.kaggle.com/competitions/ubiquant-market-prediction
9https://beta.openai.com/docs/guides/fine-tuning
F Appendix: Prompt for doing chain of thought on GPT-4o

Given the 15 guidelines describing the code cell and its table output in the Jupyter Notebook:
1. Value(Get the exact data attribute values for a set of criteria)
2. Difference(A comparison between at least two distinct attributes within the target object, or a comparison between the target object and previously measured values)
3. Trend(Indicates a general tendency over a period of time)
4. Proportion(Measure the proportion of selected data attribute(s) within a specified set )
5. Categorization(Select the data attribute(s) that meet the condition)
6. Distribution(Show the amount of shared value for the selected data attributes or present a breakdown of all data attributes)
7. Rank(Sort data attributes by their values and display a breakdown of selected attributes)
8. Association (Identify the useful relationship between two or more data attributes)
9. Extreme(Identify the data cases that are the most extreme in relation to the data attributes or within a specific range )
10. Outlier(Determine whether there are unexpected data attributes or statistically significant outliers)
11. Aggregation(Calculate the descriptive statistical indicators (e.g ., average, sum, count, etc. ) based on the data attributes.)
12. Goal(Express user's goal. To say what value or function they tend to use for the later research)
13. Reason(Express reason using the data from the table or explain the reasons why certain functions are used or why a task is performed.)
14. Feature Engineer(The process of selecting, transforming, extracting, combining, and manipulating raw data to generate the desired variables for analysis or predictive modeling)
15. Complementary
Details (Express additional contextual elements and supporting information to enhance understanding of the primary content)

Q: When using Jupyter Notebook, the data scientist wants to write a description in the Markdown cell covering the code cell and its table output. The description should be less than 50 tokens.
Table Sequence:
    | passengerid| survived
mean| 446.000000 | 0.383838
Code: train = pd.read_csv("../input/titanic/train.csv")
    # take a quick look at the training data
    train.describe(include="all")"

A: The data scientist wants to write a description in Extreme guideline, the description he writes is: the mean survived rate is 38.3 denoting most of the passengers have not survived

Q: When using Jupyter Notebook, the data scientist wants to write a description in the Markdown cell covering the code cell and its table output:
<table>
</table>
<code>

G Appendix: Prompt for doing in-context learning on GPT-4o

Q: When using Jupyter Notebook, the data scientist wants to write a description in the Markdown cell covering the code cell and its table output.
Table Sequence:
    | passengerid| survived
    mean| 446.000000 | 0.383838
train = pd.read_csv("../input/titanic/train.csv")
# take a quick look at the training data
train.describe(include="all")"
<table>
  <tr>
    <th>Guideline</th>
    <th>N</th>
    <th>Description</th>
    <th>Example</th>
  </tr>
  <tr>
    <td>Value</td>
    <td>286 (7.29%)</td>
    <td>Get the exact data attribute values for a set of criteria</td>
    <td>The mean survived rate is 38.3 denoting most of the passengers did not survive</td>
  </tr>
  <tr>
    <td>Difference</td>
    <td>138 (3.52%)</td>
    <td>A comparison between at least two distinct attributes within the target object, or a comparison between the target object and previously measured values.</td>
    <td>The difference though narrows down considerably if we were to consider groups of 2 woman travelers</td>
  </tr>
  <tr>
    <td>Trend</td>
    <td>31 (0.79%)</td>
    <td>Indicates a general tendency over a period of time.</td>
    <td>table is displayed in a descending trend in accuracy</td>
  </tr>
  <tr>
    <td>Proportion</td>
    <td>120 (3.06%)</td>
    <td>Measure the proportion of selected data attribute(s) within a specified set</td>
    <td>8 of 10 passengers have parents</td>
  </tr>
  <tr>
    <td>Categorization</td>
    <td>74 (1.89%)</td>
    <td>Select the data attribute(s) that meet the condition.</td>
    <td>1 denotes survived while 0 denote not survived</td>
  </tr>
  <tr>
    <td>Distribution</td>
    <td>127 (3.20%)</td>
    <td>Show the amount of shared value for the selected data attributes or present a breakdown of all data attributes.</td>
    <td>Fare value range from 7 to 13</td>
  </tr>
  <tr>
    <td>Rank</td>
    <td>73 (1.86%)</td>
    <td>Sort data attributes by their values and display a breakdown of selected attributes.</td>
    <td>Selecting the top 3 classifiers for model prediction</td>
  </tr>
  <tr>
    <td>Association</td>
    <td>165 (4.21%)</td>
    <td>Identify the useful relationship between two or more data attributes.</td>
    <td>These two passengers are in the same PClass</td>
  </tr>
  <tr>
    <td>Extreme</td>
    <td>227 (5.78%)</td>
    <td>Identify the data cases that are the most extreme in relation to the data attributes or specific research.</td>
    <td>Model 1 has the highest accuracy</td>
  </tr>
  <tr>
    <td>Outlier</td>
    <td>257 (6.55%)</td>
    <td>Determine whether there are unexpected data attributes or statistically significant outliers.</td>
    <td>Age column has some missing values</td>
  </tr>
  <tr>
    <td>Aggregation</td>
    <td>125 (3.19%)</td>
    <td>Calculate the descriptive statistical indicators (e.g., average, sum, count, etc.) based on the data attributes.</td>
    <td>There are 2 classes in the Deck</td>
  </tr>
  <tr>
    <td>Goal</td>
    <td>771 (19.64%)</td>
    <td>Express user's goal. To say what value or function they tend to use for the later research</td>
    <td>We use the Gaussian Process Classifier to plot the confusion matrix</td>
  </tr>
  <tr>
    <td>Reason</td>
    <td>276 (7.03%)</td>
    <td>Express reason using the data from the table or explains the reasons why certain functions are used or why a task is performed.</td>
    <td>We go through deleting the column for Cabin deleting 2 rows for Embarked and since Age plays some role we can ...</td>
  </tr>
  <tr>
    <td>Feature Engineer</td>
    <td>393 (10.02%)</td>
    <td>The process of selecting, transforming, extracting, combining, and manipulating raw data to generate the desired variables for analysis or predictive modeling.</td>
    <td>Delete Name and Ticket due to it s high cardinality</td>
  </tr>
  <tr>
    <td>Complementary Details</td>
    <td>870 (22.17%)</td>
    <td>Express additional contextual elements and supporting information to enhance understanding of the primary content</td>
    <td>Column details, counts, and data types provide supplementary technical information about the Titanic passenger dataset structure.</td>
  </tr>
</table>

Table 11: We identify 15 guideline categories based on the types of descriptions in the Markdown cells which are below the code whose output is a table.

A: The data scientist wants to write a description in Extreme guideline, the description he writes is: the mean survived rate is 38.3 denoting most of the passengers have not survived

Q: When using Jupyter Notebook, the data scientist wants to write a description in the Markdown cell covering the code cell and its table output
<table>
  <tr>
    <th>Table</th>
    <th>Code</th>
  </tr>
</table>

H Appendix: G-Eval in Coherence, Consistency, Correctness, and Fluency

H.1 Definition

G-EVAL is a prompt-based evaluation system with three main components: 1) a prompt defining the evaluation task and criteria, 2) a chain-of-thoughts (CoT) that includes intermediate instructions generated by the LLM to outline the detailed evaluation steps, and 3) a scoring function that uses the LLM to calculate scores based on the probabilities of the returned tokens. The prompt should also contain customized evaluation criteria for different NLG tasks.

Coherence The overall quality of all sentences working together. This aligns with the DUC (Dang, 2005) quality question on structure and coherence, which states that "the summary should be well-structured and well-organized, building from sentence to sentence to form a coherent body of information about a topic." Followed by (Liu et al., 2023a), we conduct the G-EVAL in the four dimensions below:

Consistency The generated description is written in the correct guideline category. A factually
consistent summary contains only statements that are supported by the source document. Annotators were instructed to penalize summaries containing hallucinated facts.

Fluency The quality of individual sentences. According to the DUC (Dang, 2005) quality guidelines, sentences in the summary "should have no formatting problems, capitalization errors, or obvious grammatical errors (e.g., fragments, missing components) that make the text difficult to read."

Correctness Compare the actual output directly with the expected output to verify factual accuracy. Check if all elements specified in the expected output are present and accurately represented in the actual output. Evaluate whether there are any discrepancies in details, values, or information between the two outputs.

H.2 Prompt Details for obtaining metrics

<table>
  <tr>
    <th>You will be given one description written for a code, its table output, and its guideline category description. Your task is to rate the description on one metric. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.</th>
    <th>key points of the source text, and if it presents them in a clear and logical order.<br>3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.</th>
  </tr>
</table>

Evaluation Criteria:
Coherence (1-5) – the collective quality of all sentences. We align this dimension with the DUC quality question of structure and coherence whereby the description should be well-structured and well-organized. The description should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic.

Evaluation Steps:
1. Read the code, its table output, and its guideline category description carefully and identify the main topic and key points.
2. Read the description and compare it to the source text including the code, its table output, and its guideline category description. Check if the description covers the main topic and key points of the source text, and if it presents them in a clear and logical order.
3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.

Example:
Source Text:
{{Guideline Category Description} {Code} {Table}}
Description:
{{Description}}
Evaluation Form (scores ONLY):
- Coherence:

You will be given one description written for a code, its table output, and its guideline category description. Your task is to rate the description on one metric. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Correctness (1-5) – Determine whether the actual output is factually correct based on the expected output.

Evaluation Steps:
1. Read the code, its table output, and its guideline category description carefully and identify the main topic and key points.
2. Read the description and compare it to the source text including the code, its table output, and its guideline category description. Check if the description covers the main topic and key points of the source text, and if it presents them in a clear and logical order.
3. Assign a score for correctness on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
Example:
Source Text:
{{Guideline Category Description} {Code} {Table}}
Description:
{{Description}}
Evaluation Form (scores ONLY):
- Correctness:

You will be given one description written for a code, its table output, and its guideline category description. Your task is to rate the description on one metric. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Consistency (1-5) – the generated description is written in the correct guideline category. A factually consistent description contains only statements that are entailed by the source document. Annotators were also asked to penalize descriptions that contained hallucinated facts.

Evaluation Steps:
1. Read the code, its table output, and its guideline category description carefully and identify the main topic and key points.
2. Read the description and compare it to the source text including the code, its table output, and its guideline category description. Check if the description covers the main topic and key points of the source text, and if it presents them in a clear and logical order.
3. Assign a score for consistency on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
Example:
Source Text:
{{Guideline Category Description} {Code} {Table}}
Description:
{{Description}}
Evaluation Form (scores ONLY):
- Consistency:

You will be given one description written for a code, its table output, and its guideline category description. Your task is to rate the description on one metric. Please make sure you read and understand these instructions carefully. Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:
Fluency (1-5) – the quality of individual sentences. Drawing again from the DUC quality guidelines, sentences in the summary "should have no formatting problems, capitalization errors or obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult to read."

Evaluation Steps:
1. Read the code, its table output, and its guideline category description carefully and identify the main topic and key points.
2. Read the description and compare it to the source text including the code, its table output, and its guideline category description. Check if the description covers the main topic and key points of the source text, and if it presents them in a clear and logical order.
3. Assign a score for fluency on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
Example:
Source Text:
{{Guideline Category Description} {Code} {Table}}
Description:
{{Description}}
Evaluation Form (scores ONLY):tab:
example4
- Fluency:
1 Appendix: Table preprocess example

1.1 An example of Original Table Crawled from Kaggle Notebooks

<div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {vertical-align: middle;}
        .dataframe tbody tr th {vertical-align: top;}
        .dataframe thead th {text-align: right;}
    </style>
    <table border="1" class=" " data-frame">
        <thead>
            <tr style="text-align: right;">
                <th></th>
                <th>count</th>
                <th>mean</th>
                <th>std</th>
                <th>min</th>
                <th>25%</th>
                <th>50%</th>
                <th>75%</th>
                <th>max</th>
            </tr>
        </thead>
        <tbody>
            <tr>
                <th>PassengerId</th>
                <td>418.0</td>
                <td>1100.50000</td>
                <td>120.810458</td>
                <td>892.00</td>
                <td>996.2500</td>
                <td>1100.5000</td>
                <td>1204.75</td>
                <td>1309.0000</td>
            </tr>
            <tr>
                <th>Pclass</th>
                <td>418.0</td>
                <td>2.265550</td>
                <td>0.841838</td>
                <td>1.00</td>
                <td>1.0000</td>
                <td>3.0000</td>
                <td>3.0000</td>
                <td>3.0000</td>
            </tr>
            <tr>
                <th>Age</th>
                <td>332.0</td>
                <td>30.272590</td>
                <td>14.181209</td>
                <td>0.17</td>
                <td>21.0000</td>
                <td>27.0000</td>
                <td>39.00</td>
                <td>76.0000</td>
            </tr>
            <tr>
                <th>SibSp</th>
                <td>418.0</td>
                <td>0.447368</td>
                <td>0.896760</td>
                <td>0.00</td>
                <td>0.0000</td>
                <td>0.0000</td>
                <td>1.00</td>
                <td>8.0000</td>
            </tr>
            <tr>
                <th>Parch</th>
                <td>418.0</td>
                <td>0.392344</td>
                <td>0.981429</td>
                <td>0.00</td>
                <td>0.0000</td>
                <td>0.0000</td>
                <td>0.00</td>
                <td>9.0000</td>
            </tr>
            <tr>
                <th>Fare</th>
                <td>417.0</td>
                <td>35.627188</td>
                <td>55.907576</td>
                <td>0.00</td>
                <td>7.8958</td>
                <td>14.4542</td>
                <td>31.50</td>
                <td>512.3292</td>
            </tr>
        </tbody>
    </table>
</div>

1.2 Table preprocessing from the the original table in H.1

After table preprocessing, table is

<table>
    <tr>
        <th></th>
        <th>count</th>
        <th>mean</th>
        <th>std</th>
        <th>min</th>
        <th>25%</th>
        <th>50%</th>
        <th>75%</th>
        <th>max</th>
    </tr>
    <tr>
        <td>PassengerId</td>
        <td>418.0</td>
        <td>1100.50000</td>
        <td>120.810458</td>
        <td>892.00</td>
        <td>996.2500</td>
        <td>1100.5000</td>
        <td>1204.75</td>
        <td>1309.0000</td>
    </tr>
</table>
|Pclass |418.0 |2.265550 | 0.841838 |1.00 |
| 1.0000 |3.0000 | 3.00 3.0000|
Age |332.0 |30.272590 14.181209 | 0.17 |
| 21.0000 | 27.0000 | 39.00 76.0000|
SibSp |418.0 |0.447368 | 0.896760 | 0.00 |
| 0.0000 | 0.0000 | 1.00 | 8.0000|
Parch |418.0 |0.392344 | 0.981429 | 0.00 |
| 0.0000 | 0.0000 | 0.00 | 9.0000|
Fare |417.0 |35.627188 | 55.907576 |
| 0.00 | 7.8958 | 14.4542 | 31.50 |
| 512.3292|

1.3 Table Curation from the the table in H.2

In the above example, the ground truth description is: “From the count column, we find that some variables have missing values"". Based on the Table Curation method in Section 4.2, we extract the “count"" keyword from the ground truth description and extract this related column to generate the final table.

<table>
  <tr>
    <th>count</th>
  </tr>
  <tr>
    <td>PassengerId |418.0 |</td>
  </tr>
  <tr>
    <td>Pclass |418.0 |</td>
  </tr>
  <tr>
    <td>Age |332.0 |</td>
  </tr>
  <tr>
    <td>SibSp |418.0 |</td>
  </tr>
  <tr>
    <td>Parch |418.0 |</td>
  </tr>
  <tr>
    <td>Fare |417.0 |</td>
  </tr>
</table>

Reviewer paper summaries: The study introduces NBDESCRIB, a dataset for generating personalized descriptions in Jupyter Notebooks using code, tables, and user-written guidelines. It defines the TCDG task and evaluates models like CodeT5 and GPT-4o. CodeT5 performs best but struggles with alignment, reasoning, and factual accuracy, highlighting challenges in faithful description generation. A proposed Jupyter Notebook plugin demonstrates practical applications, emphasizing the importance of guidelines for improving large language models in structured data tasks.

This paper addresses a novel challenge in the field of natural language processing by proposing a new task of personalized description generation in Jupyter Notebooks. In particular, the task aims to generate descriptions based on code, tables, and user-written guidelines that reflect the user's specific needs. Existing methods typically generate descriptions from either code or tables alone, without considering the user’s unique objectives, making them less adaptable to practical scenarios. To evaluate this new task, the authors present a dataset, NBDESCRIB, which consists of 3,924 processed code-table-description pairs. These pairs are sourced from Kaggle competitions, ensuring both high quality and real-world relevance. The dataset is designed to capture diverse types of descriptions based on 15 guideline categories, such as Value, Goal, and Reason, allowing the model to tailor its outputs according to various user demands. The paper emphasizes the need for generating descriptions that are both factually accurate and contextually appropriate, given the complex nature of Jupyter Notebooks, where code and table outputs must be interpreted together. This work introduces a crucial step forward by formalizing the TCDG task and offering a practical solution for improving the user experience in data science workflows.
Summary of strengths: The task is novel and a new dataset is proposed in this work.

Both open source and close source models have been evaluated.

Human evaluation has been conducted.

Pairwise inter-rater reliability is provided in this paper.

The primary strength of this paper lies in its novel and practically significant task formulation. The authors introduce the TCDG task, which focuses on generating descriptions that align with user-specific needs, using both code and table outputs from Jupyter Notebooks. This task is highly relevant in real-world applications where users often need explanations tailored to their individual objectives. Furthermore, the authors support this new task with the NBDESCRIB dataset, which contains 3,924 processed code-table-description pairs. This dataset is well-curated from high-quality Kaggle notebooks and represents a practical, real-world challenge for description generation, making the proposed task both feasible and impactful.

Additionally, the paper addresses an important gap in the existing literature by incorporating user-written guidelines, which allows for more accurate and tailored descriptions. The integration of such guidelines helps to create personalized outputs that reflect the user's specific focus, something that previous research has largely overlooked. This approach not only enhances the relevance of the generated text but also brings a new dimension to automatic documentation generation in Jupyter Notebooks. By focusing on user needs, the authors contribute to a significant improvement in the quality of auto-generated descriptions, which has the potential to be highly beneficial for data scientists and researchers who work in computational notebooks.
Summary of weaknesses: Does the automatic evaluation metric reflect the actual performance? How do they align with human evaluation in a quantitive manner?

It's desirable to introduce ablation study using different prompt strategies to ask the model avoid the common errors listed in Section 6. Do prompt strategies improve the LLM performance, especially for GPT-4, which seems to be powerful for text generation?

Is it possible to show the generalizability of the fine-tuned CodeT5 in other related datasets?

One key weakness of the current approach lies in its limited consideration of Jupyter Notebook's diverse content types. While the implementation handles code cells and table outputs effectively, it does not currently address other important notebook elements, such as Markdown cells containing additional explanations, mathematical formulas, or user-specific notes. These elements are common in real-world computational notebooks and are often integral to understanding the context behind the code and its output. As such, the authors could benefit from exploring how their model could be adapted to better handle the full spectrum of Jupyter Notebook content, ensuring broader applicability and scalability in diverse notebook environments.
Comments, suggestions and typos: -

Extend Scalability Discussion: The authors should provide more details on how their model might scale to handle other content types in Jupyter Notebooks, such as formulas or free-form text. It would also be useful to explore how the model could be adapted to different types of computational notebooks, not limited to the Kaggle examples used here.

Explore Model Performance and Enhancement: A deeper exploration of how different models perform on the task and the principles behind improving their performance could enrich the paper. Specifically, the authors could discuss strategies for enhancing large language models to handle the complexities of this task more effectively, such as incorporating domain-specific knowledge or improving alignment with user guidelines.
Overall assessment scores: 2.5

3.5
Metareview score: 4
Metareview: An approach for generating personalized descriptions of Jupyter notebook cells. A new dataset of ~4k cells from Kaggle competitions is produced for evaluation. Several LLMs are evaluated on the task, showing readable outputs but struggling with accuracy.
Summary of reasons to publish: * Addresses a novel and interesting task, for which it builds a useful benchmark dataset
* Includes a focus on personalization
* Includes human evaluation, with inter-annotator agreement scores
* The paper sufficiently addressed reviewers' comments from the previous review round
* Very convincing author response on this round