NameGuess: Column Name Expansion for Tabular Data
Authors: Jiani Zhang
Zhengyuan Shen
Balasubramaniam Srinivasan
Shen Wang
Huzefa Rangwala
George Karypis
Venue: EMNLP 2023 Main (2023)
Keywords: Column Name Expansion
Natural Language Generation
Datasets
TL;DR: We introduce a new task that aims to expand column names as a natural language generation problem.

Abstract: Recent advances in large language models have revolutionized many sectors, including the database industry. One common challenge when dealing with large volumes of tabular data is the pervasive use of abbreviated column names, which can negatively impact performance on various data search, access, and understanding tasks. To address this issue, we introduce a new task, called NameGuess, to expand column names (used in database schema) as a natural language generation problem. We create a training dataset of 384K abbreviated-expanded column pairs using a new data fabrication method and a human-annotated evaluation benchmark that includes 9.2K examples from real-world tables. To tackle the complexities associated with polysemy and ambiguity in NameGuess, we enhance auto-regressive language models by conditioning on table content and column header names -- yielding a fine-tuned model (with 2.7B parameters) that matches human performance. Furthermore, we conduct a comprehensive analysis (on multiple LLMs) to validate the effectiveness of table content in NameGuess and identify promising future opportunities. Code has been made available at https://github.com/amazon-science/nameguess.

NameGuess: Column Name Expansion for Tabular Data
Jiani Zhang", Zhengyuan Shen; Balasubramanian Srinivasan, Shen Wang,
Huzefa Rangwala, George Karypis
Amazon Web Services (zhajiani, donshen, srbalasu, shenwa, rhuzefa, (aapypisseammzzoo.ooo com
Abstract
Qutzul Logical
Department
Dapartment
Employee Name
Salary
Commission
Name
Recent advances in large language models have revolutionized many sectors, including the database industry. One common challenge when dealing with large volumes tabular data is the pervasive use of abbreviated column names, which can negatively impact perfor- mance on various data search, access, and un- derstanding tasks. To address this issue, in- troduce new task, called NAMEGUESS, to ex- pand column names (used database schema) natural language generation problem. We create a training dataset of 384K abbreviated- expanded column pairs using new data fab- rication method and a numan-annotated evalu- ation benchmark that includes 9.2K examples from real-world tables. To tackle the com- plexities associated with polysemy and am- biguity NAMEGUESS, we enhance auto- regressive language models by conditioning on table content and column header names yielding fine-tuned model (with 2.7B param- eters) that matches human performance. Fur- thermore, we conduct a comprehensive anal- ysis (on multiple L.MMs) validate the effec- tiveness of table content in NAMEGUESS and identify promising future opportunities. Code has been made available https: /tttt::///tthb com/ aamazoonsscienncc/namggssss
hput
D_DD ID
D NAME
E_ E_NAME
SAL
COMM
Query
Column
10
Accounting
Allen
14000
4
Name
Accounting
Ward
13500
3
Research
Martin
12000
Sales
Turner
11000
3
Research
Smith
10500
22
Table name:
Engloyee Salary 2022
An example of the column name expansion task. The input are query column names with table context and the output are expanded logical names.
Figure I:
Name" as in Figure While abbreviations can be convenient for representation and use code, they can cause confusion, especially for those unfamil- iar with the particular tables or subject matter. Col- umn headers are essential for many table-related tasks (Xie et al., 2022), and using abbreviations makes it challenging for end users to search and retrieve relevant data for their tasks.
Abbreviated column names can negatively im- pact the usefulness of the underlying data. For example, in the text2SQL semantic parsing task, which converts natural language into formal pro- grams queries for retrieval, abbreviations can lead to a mismatch with the terms used in the natu- ral language queries. In fact, in the human-labeled tex12SQL spider dataset (Yu et al., 2018), 6.6% of column names are abbreviations. Figure 2 shows an example containing abbreviated column names like "c_name" and "acc_bal" " in tables, which mis- match the terms "the name of all customers" and "account balance". Simple changes in using abbre- viated column names in the spider dataset result in performance degradation of over ten percentage points, with the exact match score of 66.63% (Xie et al., 2022) dropping 56.09% on the T5-large model (Raffel al., 2020). The effect of the abbre- viated column names on table question answering (QA) (Yin et al., 2020), and column relation dis-
1 Introduction
Tabular data is widely used for storing and organiz- ing information web (Zhang and Balog, 2020) and enterprise applications (Leonard, 201 1). One common practice when creating tables in databases is to use abbreviations for column headers due to character length limits in many standard database systems. For example. the maximum length for column names in SQL database is 256 bytes, leading to the of abbreviations such as "D_ID" for "Department ID" and "E_NAME" for "Employee
These two authors contributed equally. Corresponding author.

---

Table1:
branch_ID
bname
no_of_
city
state
80
GPT-4 tankmoee
bank
70-
Table2:
customer
account
acc_
of
credit score
60-
LLa (556I
customer
JD
name
_typ
bal
loans
Exact Match (%)
50-
GPT-Neo Falton-Instriees
Utterance (input): List the name of all customers sorted by their account balance ascending order.
GPT-Neo
SQL (output): SELECT eustomer FROM cus tomer ORDER BY aeeoent_tye
30
GPT-2 (1.59)
Human Performance
GPT HPPT-21124W) (124M)
Figure 2: An example for Text2SQL semantic parsing. The terms "the name of all customers" and "account balance" do not match the abbreviated column names c_name and acc_bal. Instead, they match with the column names customer_iis and account_type.
Non-fine-tuned
fo
10"
10
10a
Model Size (B)
Figure 3: Exact match results for fine-tuned models (*), non-finee non-finetuned LLMs, and human performance. Solid and hollow symbols denote inclusion and exclusion of sampled table contents.
covery (Koutras et al., 2021) are Table and description is in Appendix A.1. The performance degradation emphasizes the need for descriptive column headers in handling tabular data.
with 9,218 column pairs 895 tables. We then proposed method to produce training data by selectively abbreviating well-curated column names from web tables using abbreviation look-ups and probabilistic rules. Next, we enhanced auto- regressive language models with supervised fine- tuning, conditioned on table content and column headers, and conducted extensive experiments to evaluate sfateoof-hhe-at LLMs. The overall model performance is shown in Figure 3. While GPT-4 ex- hibited promising performance NAMEGUESS, the deployment of such LLMs comes with much larger memory and computation overhcads. Our findings indicate that supervised fine-tuning of smaller 2.7B parameter models achieves close to human performance, and including table contents consistently improved performance. However, all models found the task challenging, with the best only achieving 54.7% accuracy the extra-hard examples, indicating room for improvement in panding abbreviated column names. Our main con- tributions are:
Expanding column names and generating de- scriptive headers also has other beneficial aspects. First, using expanded column names can increase the readability of tables, especially when complex or technical data is present. The expansion also enables data integration by allowing users to eas- ily distinguish between tables with similar column names but different meanings and helping identify relationships between tables with different abbre- viated column names. Finally, expanded column names can also improve the efficacy of keyword- based searches for discovering related tables.
This work addresses the task of expanding ab- breviated column names in tabular data. To the best of our knowledge, this is the first work to in- troduce and tackle this problem. Unlike previous textual abbreviation expansion works that formu- lated the task as classification problem with a predefined set of candidate expansions (Roark and Sproat, 2014; Gorman et al., 2021), we formulate NAMEGUESS as a natural language generation problem. Acquiring extensive candidate expan- sions can be laborious, pairs of abbreviated and expanded column names seldom present in the expansion same table. Conversely, abbreviation Abbreviatiionexpaniioo pairs can be gleaned from textual data through co- occurrence signals, such as parenthetical expres- sions. Moreover, abbreviated headers may exhibit ambiguity and polysemy arising from developer- specific naming conventions and domain-related variations in expansions.
Introduced a new column name expansion task, named NAMEGUESS, as a natural lan- guage generation problem,
2. Developed large-scale training dataset for the NAMEGUESS task using an automatic method that largely reduces human effort,
3. Created a human-annotate evaluation bench- mark with various difficulty levels, which pro- vides standard for comparing results,
4. Performed a comprehensive evaluation of LMs of different sizes and training strategies and compared them to human performance on the NAMEGUESS task.
To tackle NAMEGUESS, we first built a large dataset consisting of 163,474 tables with 384,333 column pairs and a human-annotatee benchmark

---

Original Column Names
Ahbreviated Column Names
Table Collection
|-200K tables)
Text2SQL
66.63
56.09
"199K tables
(Match score %)
"1K tables
Schema-base based
Relation Detection
100.00
59.50
Logical Name Identification
Pilot Study (250 examples)
Quality Audit (>955% agreement)
(Recall %)
Table QA
(163K tables
84,32
80.49
384K examples)
(Accuracy %)
Abbreviated/ Logical Name Annotation (895 tables 9.2K examples]
Quality Audit (>955% agreement)
Table I: The effect of abbreviated column names on three table understanding tasks. The performance drops all the tasks.
Abbreviation Generation tables
Final Review & Processing
384K examples)
The automatic generation process of the training set
The annotation process of the evaluation set
2 Problem Formulation
Figure 4: The processes of creating the training and evaluation datasets.
We formulate the NAMEGUESS task as a natural language generation problem: given query name x from table I, generate a logical name y that de- scribes the column. A table contains table content and various schema data, like table name, col- umn headers, and data types. Let fo be generator with parameters 6, the formulation becomes y fo ta) Note that the query column names within a table may take diffcrent forms, including abbre- viations and full names. The output logical names are expanded column names and should be easily understandable without additional knowledge of the table content. See Figure 1 for example inputs and outputs in the ble, where "SAL" stands for "Salary" and "COMM" stands for "Commission".
3.1 Table Collection
The training and evaluation datasets were obtained from seven public tabular dataset repositories. To ensure the quality of the tables, we filtered out tables with less than five rows columns and re- moved tables with more than half of the entries being NaN more than half of column names being duplicates. Table 2 summarizes the dataset statistics.
City Open Data. We sourced tables from New York (NYC), Chicago (CHI), San Francisco (SF), and Los Angeles (LA), covering categories, such as business, education, environment, health, art, and culture. We downloaded all the tables using Socrata Open Data APIs in June 2022.
Saayee_Salary_2000 ta-
3 Dataset Creation
GitTables was extracted from CSV files open- source GitHub repositories (Hulsebos et al., 2023). GitTables is the largest dataset we can access with relatively larger table size. WikiSQL
We created training dataset comprising 384,333 columns spanning 163,474 tables and human- labeled evaluation benchmark containing 9,218 ex- amples across 895 tables for NAMEGUESS. The main challenge in obtaining abbreviated-exxpadee column name pairs is the rare co-occurrence of these two names within the same table or database. Therefore, we employed the strategies of convert- ing well-curated column names to abbreviated names that align with the naming convention of database developers and annotating the abbreviated column names based the information in the in- put table. Figure illustrates the main steps for creating the training and evaluation datasets. The details of the source tables are discussed Sec- tion 3. The training and evaluation datasets are constructed Section 3.2 and Section 3.3.
released for the text2SQL task (Zhong et al., 2017). We only used the large corpus of Wikipedia tables in this dataset.
Dresden Web Table Corpus (Eberius et al., 2015). We only utilized the relational tables in this dataset and applied strict filtering criteria to keep tables with high-quality column names and contents. This is the largest accessible dataset but with relatively smaller tables than others.
3.2 Training Data Creation
We utilized two steps convert logical names to abbreviated names: (1) identifying the logical names as the ground truth and (2) abbreviating the names as the input column names x.

---

Data Source
Avg.
Avg. #Row
Abbreviation
Examples
#Ex.
#Table
#Col
Schemess
Logical Name
Abbreviated Name
Training Datasets
Word-level
Current Balance
CUR_BAL
NYC
16,697
1,921
23.2
642
Abbreviation
GitTables
163,204
49,259
19.5
93
Acronym
Fiscal Year 2021
FY_2021
WikiSQL.
22,963
9,268
6.4
20
Extraction
DWTC
181,469
103,026
65.6
Word
Zip Code
Zip
Overall
384,333
163,474
47.8
42
Removal
Word Order
Evaluation Datasets
Birth Rate 2018
2818_BR
Change
SF
4,781
388
23.9
643
CHI
3,975
21.1
605
Table 3: Examples of four common abbreviation schemes for logical column names in database tables.
LA
462
65
21.3
Overall
9,218
895
21.9
620
Table 2: Statistics for the training and evaluation datasets. '#Ex.' stands for 'number of examples'. 'Avg. #Col' is 'the average number of columns per table'.
very common, especially when the column header contains fewer words or words that cannot be fur- ther shortened without creating ambiguity. Method 2 (lookup): Replaced with an abbrevia- tion from expansion-abbreviig
3.2.1 Logical Name Identification
look-up table. This is mainly for producing commonly-used abbre- viations with enhanced diversity in naming style that is hard obtain using the above four refor- matting methods. Examples include abbreviations based on pronunciations (e.g.. transaction > txn and conversions (c.g., second - 2nd, number + no. /#, and at + @). The lookup dictionary con- tains common abbreviations for 23,110 10 English words terms. In cases where multiple candi- date abbreviations are available, the abbreviation is chosen randomly.
Identifying high-quality column names from rela- tional tables is essential, as further abbreviating a vague term can lead to even more ambiguity. Algorithm 2 in Appendix A.2 shows the detailed algorithm with vocabulary-basee strategy. We regarded a column name as well-curated only if all the tokens in the column name can be found from a pre-defined vocabulary. Here, we used the Word- Ninja packuge to split the original column headers, which allowed us to check whether individual to- kens are included in the vocabulary.
and symbolic
To construct this pre-defined vocabulary, used WordNet open-class English words (Fellbaum, 1998) followed by set of filtering criteria (such as removing words with digits, punctuation, and short words that are abbreviations or acronyms) so that the classifier achieves high precision for detecting the logical names, which further used as ground truth labels.
Method (rule): Processed by one of the word- level abbreviation rules:
Rule 1: Keep the first k characters, k E [1,5]
(e.g. abbreviation * abbr);
Rule 2: Keep removing non-leading vowels un- til the threshold &s [1, 5] all non-leading vow- els have been removed (e.g. abbreviation A=p abbrvtn, doodle S doodl); Rule 3: Specifically, while the length of the input string is longer than a specified threshold value & E [1,5], the following steps are applied: 1) neighboring duplicate characters are removed, 2) vowels are removed randomly until no vowel remain, and 3) consonants are removed randomly until no consonant remains. (e.g. abbreviation + abrv). This is to emulate the real-world data and varying preferences of database developers.
3.2.2 Abbreviation Generation
After obtaining well-curated names, we used an abbreviation generator to produce the abbreviated names. Table 3 summarizes four abbreviation schemes from logical to abbreviated names. We adopted word-level abbreviation and acronym ex- traction methods and limited word removal and word order change cases, because specifying rules for word removal or order change is an open-ended and difficult-to-scat problem. Our abbreviated name generator employs a probabilistic approach to determine the specific method used for word- level abbreviation. The method is chosen from the following three options, with the selection proba- bility determined by pre-defined weights:
k=4
Once rule is selected from the above choices, it applied to all words within the same column header. is important to note that these rules do not apply to non-alphabetical words. In the case of numerical words representing a four-digit year, we shorten it to a two-digit representation with 50%
Method (keep): Left as-is. This is trivial but

---

Algorithm Abbreviation generation
Madium
Hard
Extra Hard
Duteet Logical
Inputs:
Object
Project URL
Parnit
Attacmment
Nome
A lookup dictionary D and string
Initialize:
Query
obj id__
prj .ur
permitED
Colurn
1943
2/16/2023
Methodx f select_ment select_method))
6617
Nan
2/16/2023
Rulex +- select_rule()
abbr.words. x- 0
7195
2/16/2023
Nan
X + tokenize(I)
> Input: well-curated
for z; in x do
Figure 5: Examples with different difficulty levels. Each example contains query column name, sampled col- umn contents, and ground truth logical name.
if Methody is keep then:
2 <
Methody is lookup then
if σ; C D then
3.3 Evaluation Benchmark Annotation
ã - D(r
Instead of splitting a subset of training examples for evaluation, created a human annotated eval- uation dataset.
else
human-annotated
ã; (x;)
if Methody is rule then
3.3.1 Human Annotations
3. <- (2 abbr.words 8;
The evaluation dataset was confirmed by 15 hu- man annotators using the City Open Data from Los Angeles, San Francisco, and Chicago. Detailed instructions were provided to the annotators to en- sure consistency in annotations. The instructions are outlined follows:
f <- combine abbbbbbbwoooo words abbreviated
D Output:
probability, e.g., 2020 + 20.
Hybrid Method. To simulate the naming con- vention for database tables, probability of 0.5 is assigned for converting patterns subject to common acronyms in the well-curated column headers. The
1. Read the table metadata, including table cate- gory, name, and description.
2. Read and understand the original column name and sampled cell values for that column.
entire or a part of the well-curated form could be replaced by an acronym, such Employee Date of Birth -> EMP_DOB. Furthermore, additional rules are introduced to selectively remove switch the order of words without altering the semantics of the column name. For example, Event Name
Determine if the original column name is ab- breviated or well-curated. If in a well-curated form, provide only an "abbreviated variant". Otherwise, provide "well-curated variant" and an "abbreviated variant".
Evnt and Mailing Address District 2013 + 201 3Mai lAddrDist. Note that when the same word(s) appears in different columns of the same table, we use the same abbreviation form.
When creating abbreviated names, please com- bine abbreviated words as suggested by the combining rules detailed in Table 3.
A Abbreviation Combination. As the last step of the abbreviation algorithm, the combinc() function assigns an equal probability of concatenating the resulting abbreviated words into camel case, Pascal case, snake case, or simple combination which further adds diversity in the naming style.
A pilot study was first implemented produce a small number of examples, followed by a quality audit to ensure the guidelines were well-understood. Note that the column names that were found to be unclear or difficult to interpret even with the provided metadata and column cell values were discarded from the dataset. Finally, the annota- tions underwent another audit process from sep- arate group of annotators. We employed crite- rion where if two of three annotators agreed, the annotation was considered to pass this agree- ment measure. Overall, the annotations achieved an agreement rate of 96.5%.
Overall Algorithm. The overall algorithm is in Al- gorithm 1 with the probability of Method (keep), 2 (lookup) and 3 (rule) set 0.3, 0.6 and 0.1, re- spectively. The probabilities for Rule 1, 2, and 3 rule are set to 0.2, 0.4, and 0.4, respec- tively. These assignments are undertaken to main- tain statistics that resemble woal-world datasets.

---

3.3.2 Difficulty Breakdown
q "As abbreviations of column names from a table, {2131... kax3 stand for (w))I...IIuk).. [uK).".
We divide the data samples into four difficulty lev- els to gain deeper insight into the model perfor- mance. This classification is based on the edit dis- tance between characters in abbreviated names and ground truth labels. Both names will first be tok- enized and have underscores replaced with spaces, and numberrspunnctuations were discarded when calculating the edit distance. Four categories are established: (1) 1,036 (11%) casy examples, (2) 3,623 (39%) medium examples, (3) 3,681 (40%) hard examples, and (4) 878 (10%) extra-hard exam- ples. Figure 5 shows one representative example from each level. The difficulty breakdown we em- ploy is significantly different from dataset difficulty breakdown dataset cartography approaches in literature (Swayamdipta al., 2020; Ethayarajh et al., 2022), since the column name expansion task is formulated as a generation task as opposed to classification tasks considered in prior literature.
Here K represents the number of query names, and N is the number of sampled rows in table. [ægfé = 1,... K) refers the abbreviated column names from the same table. {yili = 1,..., K) represents the ground-trute truuh logical name. And ( 1....,;;; - .,..N denotes the sam- pled cell values of each query name. We set K and N to 10 based on the ablation study results after testing different K and N values in Appendix A.6 Table 7. A table with more than ten columns is split into multiple sequences. The prompt can also be designed in different ways. These prompts were selected by testing various template templates on the GPT-2 XL model and choosing the ones that yielded the best outcomes. We utilized decoder- only models for training, where the entire prompt sequence employed to recover logical names in autoregressive manner.
4 Methods
Prediction. Given K column names T1,..,, in table used LM to predict the corre- sponding logical names "1, WK- To generate predictions, we concatenated the linearized table context t' and the task prompt q as the input se- quence. We used the same query prompt as training during inference, except that we removed the ground truth. The modified query prompt be- comes "As abbreviations of column names from table, {=1)l-.. KÆ stand for". For the non-finetuned LLMs, provided single demonstration before the query prompt to ensure that the model can generate answers a desired for- mat, i.e., "As abbreviations of column names from a table, c_name I pCd I dt stand for Customer Name Product Code I Date." We extracted the answer in the predicted sequence when the <EOS> token or the period token is met and then split the answers for each query name with the separator.
Recent advances in pre-trained LMs (Radford et al., 2019; Brown et al., 2020) have shown a strong abil- ity to generate fluent text and the "emergent" per- formance boost when scaling up LMs (Wei et al., 2022). Therefore, evaluated the performance of both small and large LMs for NAMEGUESS. We adopted two learning paradigms. First, we em- ployed prompt-based learning with LLMs without tuning model parameters. Second, we fine-tuned small LMs. In particular, we utilized the super- vised fine-tuning (Schick and Schütze, 2021a,b) paradigm with task-specific prompts.
Training. We fine-tuned pre-trained LMs by contextualizing column query names with table content, incorporating sampled cell values and ta- ble schema data. To limit the sequence length of a linearized table, we selected N cell values in the corresponding column (after removing duplicates) for each query name. We truncated cell values when they had more than 20 characters. Moreover, instead of predicting individual query name sepa- rately, we jointly predicted K query names. The columns are stacked together to form the table con- text data This structured input is then serialized and combined with a task prompt q. Specifically,
5 Experiments
We performed comprchensive set of experiments to answer (1) Can NAMEGUESS be solved as natural language generation task? (2) How does
"Column names: (m1 ) {œx] <SEP> row 1: l......, (ck) <SEP> row 2: [c]
fine-tuning and scaling the number of parameters help the model handle NAMEGUESS? (3) Can table contents aid disambiguation?
I <SEP>... row N: [c ). 1",

---

5.1 Evaluation Metrics
EM
Fl
Hert F1
Model
q
t'tq
1'+q
1'+g
We use three metrics for evaluation: exact match accuracy, Fll scores based on partial matches, and
GPT-2 (124M)
10.4
25.7
27.9
46.4
37.2
52.6
GPT-2
35.1
37.8
56.5
59.5
61.6
63.6
GPT-Nco (1,3R)"
38.3
41.6
58.8
62.4
62.4
66.1
Bert FI scores based on semantic matches.
GPT-Neo (2780
40.6
43.8
61.4
64.7
65.7
68.2
Exact Match (EM). Similar to the exact match for question answering, the exact match is computed based on whether the predicted column name is identical to the ground truth after normalization (ig- noring cases, removing punctuations and articles). F1 Score. Computed over individual normalized tokens the prediction against those in the ground truth, as precision (precll(priission + recall), where precision and recall are computed by the number of shared tokens in the prediction and ground truth label, respectively.
Human
43.4
66.5
65.4
Palcon-Inst. (40B)
51.2
53.4
68.4
75.6
72.2
73.3
L.aMA (65B)
53.3
56.2
69.6
71.2
73.3
74.0
GPT-4
69.3
73.4
82.2
85.5
81.6
86.2
Table Overall exact match (EM), FI, and Bert-FI scores (%) of fine-tuned models, LLMs (non-finetuned), and human performance. Fine-tuned models are marked with t' q indicates the results of incorporating sampled table contents, and q refers to only using task prompts without table contents.
BertScore. The similarity score for each token in the predicted phrase with that in the reference phrase computed using pre-trained contextual em- beddings (Zhang al., 2020b). demonstrates better alignment with human judgment sum- marization and paraphrasing tasks than existing sentence-level and system-level evaluation metrics. We use rescaled BertScore F1 and roberta_larg _lage (Liu et al., 2019) for contextual embeddings.
have successfully extracted answers. The scores in the tables are calculated based on the predictions with successful extractions.
Effect of Model Size. From Table we see that among the fine-tuned models, the GPT-2- 124M model exhibits particularly poor perfor- mance, while the fine-tuned GPT-Neo-2.7B model achieves the highest overall EM, FI, and Bert FI scores. With similar number of parameters, the GPT-Neo-1.3B model has a 3.8% higher overall EM score than GPT-2-1.5B. With a much larger size, GPT4 achieves 29.6% higher EM than the best fine-tuned model.
5.2 Representative Methods
We fine-tuned GPT-2 (Radford et al., 2019) and GPT-Neo (Black al., 2021) models from Hug- ging Face. We conducted preliminary experi- ments using the pre-trained small LMs without fine-tuning but consistently obtained incorrect re- sults. We also evaluated non-finetuned L.Mss, in- cluding Falcon-40B-Insoe
Effect of Fine- Fine-Tuning. Without fine-tuning, the small to medium models achieve <1% overall EM scores, producing almost random predictions. How- ever, the NAMEGUESS training data allows a se- ries of fine-tuned small to medium-sized models to approach human performance with much cheaper inference cost than LLMs.. Still, big gap exists between these fine-tuned models and LLMs.
(Almazrouci et al., 2023), LLaMA-65B (Touvron et al., 2023), and GPT-4 (OpenAI, 2023) using the same prompt. Fur- thermore, we collected human performance from 6 annotators on 1,200 samples from the evaluation set, including 300 from each difficulty level. The annotators had access to 10 sampled cell values for each column. Detailed setups are Appendix A.5.
Human Performance. One important observa- tion is that human performance on NAMEGUESS test set (also numan-annotaan is far from perfect. However, the fine-tuned models can slightly exceed human performance (the fine-tuned GPT-Nco-2.7B is 0.4% higher in EM). Intuitively, expanding from query name into logical name y is much more challenging than reverse since requires deeper understanding of the meaning and context of the ab- breviations to identify and reconstruct the original phrase accurately.
5.3 Results and Discussion
The main results, including models with and with- out table context data, are in Table 4. Table 5 reports the EM results on four hardness levels, and the Fl and Bert-Fl results are in Appendix Ta- ble 6. The responses of Falcon-instruct, LLaMA, and GPT-4 models may not match the demonstra- tion example, resulting in potential answer extrac- tion failures. In total, 92% of malcon-instruct--055 96% of 1LMMA655B, and 99% of GPT-4 examples
Effect of table context data. Expanding abbrevi- ated column names may require deep understand- ing of table content. For example, "E_NAME" can represent "Employer Name" instead of "Employee

---

Med-
Extra
predefined labels (classification task), while the lat- ter refines tokens within names, often at the word level (e.g.. "c_name" expands to "customer name").
Easy
Hard
ium
Hard
GPT-2 (121M)
41.4
30.3
19.0
16.1
GPT-2 (1.50)
53.8
42.7
32.3
21.1
GPT-Neo (1.38)
54.3
47.2
36.8
23.6
Table question answering (Yin et al., 2020; Herzig et al., 2020; Yang et al., 2022; Xie et al., 2022) re-
GPT-Neo (2.78)
55.3
50.4
38.0
27.2
Falcon-Ine Inst. (10B)
62.2
59.3
49.0
36.5
quires models to understand both tables and natural language questions and to perform reasoning over the tables. Other tasks, such as table description generation (Gong al., 2020), table fact verifica- tion (Eisenschlos et al., 2021), and formula predic- tion (Cheng al., 2022), also require meaningful column names to aid the model understanding the overall semantic meaning of tables.
LLaMA (1SB)
62.2
62.9
52.3
37.9
GPT-4
71.7
79.9
72.0
54.7
Human
53.7
53.7
33.7
30.2
Table 5: Exact match scores (%) of fine-tuned models, LLMs (non-finetuned), and human performance for four difficulty levels. indicates fine-tuned models.
Name" with company names among column val- ues. By comparing the performance of fine-tuned and non-fine-tuned models with sampled table con- tent (t' + g) and without table content (g only) in Table 4, we find that incorporating sampled table contents can increase the performance of the models, with a huge boost of 15% for the smallest GPT-2 (124M) model.
6.2 Abbreviation Expansion and Acronym Disambiguation
Abbreviations are widely used in social network posts (Gorman et al., 2021), biomedical articles (Yu et al., 2002), clinic notes (Wu et al., 2015) and scientific documents (Zilio al., 2022; Pouran Ben Veyseh et al., 2020). Abbreviation expan- sion and acronym disambiguation tasks are typ- ically formulated as classification problems (Am- mar et al., 2011; Pouran Ben Veyseh et al., 2020), which involve selecting an expansion for an ab- breviation from set of candidates based on the context information. The ad hoc abbreviation ex- pansion task (Gorman al., 2021) is similar to the NAMEGUESS task but limits the per-token transla- tion imposed on the dataset creation process and the developed solutions. Meanwhile, NAMEGUESS is a natural language generation problem suitable for a wide range of lengths of abbreviations and expan- sions. Regarding abbreviation expansion, our most relevant work is by Cai et al. (2022). However, this work primarily addresses text messages/SMS ab- breviations, aiming reduce message length and minimize typos. In contrast, our task focuses on generating meaningful and human-readable expan- sions for column name-based abbreviations.
Difficulty Breakdowns. We observe trend of decreasing exact match scores more difficult examples. From the difficulty breakdown, the fine- tuned small LMs can outperform humans on easy examples but are worse on extra hard examples. Conversely, GPT-4 outperforms all other models and human results, especially in medium and hard divisions. The LLMs are infused with broader knowledge regarding hard and extra hard examples. They are better at interpreting the exact meaning of the misleading abbreviations that often involve uncommon acronyms words or ambiguous ways of combining abbreviations.
6
Related Work
6.1 Table Understanding and Metadata Augmentation Tasks
Great strides have been achieved in table under- standing (Dong et al., 2022). Descriptive column names are crucial for task performance, aiding the model comprehending table semantics and con- ducting cell-level reasoning. Column type detec- tion (Hulsebos et al., 2019; Zhang et al., 2020a; Suhara et al., 2022) and column relationship iden- tification (Deng et al., 2022; lida al., 2021; Wang et al., 2021) involve assigning predefined types, like semantic labels or database column re- lationships. Semantic column type detection and NAMEGUESS, though related to columns, have distinct objectives. The former predicts types with
Conclusion and Future Works
We introduced task related expanding the commonly-used abbreviated column names in tab- ular data, developed and created two benchmark datasets to facilitate the study of this task, and ana- lyzed the performance of many language modeling methods to solve this task. One future direction is to utilize similar examples that provide contextual information to solve this task, preferably feeding these examples through in-context learning.

---

8 Ethics Statement
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pages 1877-1901.
The human annotations, including abbrevi- ated/logical column names for the evaluation set, were collected through hired annotators from data annotation service. Annotators were instructed to strictly refrain from including any biased, hate- ful, or offensive content towards any race, gen- der, sex, or religion. The annotations passed through audits, where they were examined by a separate group of annotators and reached 96.5% agreement ratio. The human performance on the NAMEGUESS test set was collected from database and dialogue/linguisties experts.
Shamqing Cai, Subhashini Venugopalan, Katrin Tomanek, Ajit Narayaman, Meredith Morris, and Michael Brenner. 2022. Context-aware abbreviation expansion using large language models. Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies (NAACL- HLT), pages 1261-1275.
9 Limitations
One limitation of our work is the need for real rela- tional database tables. The training and evaluation sets we used were all public web-related tables, which generally have fewer rows and lack the meta- data of primary and secondary keys. This work is just the first step to introduce this topic to the NLP community, and further research is needed to improve the performance with methods that better capture context around relational data. Moreover, the scope of handling omitted information in the original column names, like column header "glu- cose", which stands for "fasting glucose", is be- yond NAMEGUESS. One possible solution is to collect and utilize more table metadata information. For example, if the table name contains the term "fasting", then the original column name "glucose" will most likely be inferred as "fasting glucose".
Zhoujun Cheng, Haoyu Dong. Ran Jia, Pengfei Wu. Shi Han, Fan Cheng, and Dongmei Zhang. 2022. FORTAP: using formulas for
aware table pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1150-1166.
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2022. Turl: Table understanding through repre- sentation learning. ACM SIGMOD Record, 51(1):33- 40.
Haoyu Dong, Zhoujun Cheng, Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dong- mei Zhang. 2022. Table pre- training:: A survey model architectures, pre-training objectives, and downstream tasks. In Proceedings of the Thirty- First International Joint Conference on Artificial Intelli- gence (IJCAT), pages 5426-5435.
Julian Eberius, Katrin Braunschweig, Markus Hentsch, Maik Thiele, Ahmad Ahmadov, and Wolfgang Lehner. 2015. Building the dresden web table corpus: A classification approach. In 2015 IEEE/ACM 2nd International Symposium on Big Data Computing (BDC), pages 41-50 IEEE
References
Ebtesan Almazrouei, Hamza Alobeidli, Abdulaziz Al- shamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Hes- low, Julien Launay. Quentin Malartic. Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. Falcon-40B: an open large language model with state-of-the art performance.
Julian Martin Eisenschlos. Maharshi Gor, Thomas Müller, and William W Cohen. 2021. Mate: Multi- view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7606-7619.
Waleed Ammar, Kareem Darwish, Ali Kahki, and Khaled Hafez. 2011. Ice-tea: in-context expansion and translation of english abbreviations. In Interna- tional Conference on Intelligent Text Processing and Computational Linguistics (CICLing), pages 41 41-54. Springer.
Kawin Ethayarajh. Yojin Choi, and Swabha Swayamdipta. 2022. Understanding dataset difficulty with V-usable information. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pages 5988 -6008
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh- Tensorflow.
Christiane Fellbaum. 1998. WordNet: An electronic lexical database. MIT press.

---

Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin. Wei Bi, Xiaojiang Liu, and Ting Liu. 2020. TableGPT: Few-shot table-to-texi generation with ta- ble structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguistics ( (COLING), pages 1978-
Amir Pouran Ben Veyseh, Franck Dernoncourt, Quan Hung Tran, and Thien Huu Nguyen. 2020. What does this acronym mean? introducing new dataset for acronym identification and disambigua- tion. In Proceedings of the 28th International Confer- ence Computational Linguistics (COLING), pages 3285-3301.
1988.
Kyle Gorman, Christo Kirov, Brian Roark, and Richard Sproat, 2021. Structured abbreviation expansion in context. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 995-1005.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei. 1lya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mäller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training- In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics (ACL), pages
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research (JMLR), 21(1): 5551.
Madelon Hulsebos, Çagatay Demiralp, and Paul Groth. 2023. Gittables: A large-scale corpus of relational tables. Proceedings of the ACM on Management of Data, 1(1):1-17.
Brian Roark and Richard Sproat. 2014. Hippocratic abbreviation expansion. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 364-369.
Madelon Hulsebos, Kevin Hu, Michiel Bakker, Emanuel Zgraggen, Arvind Satyanarayan, Tim Kraska, Ça- gatay Demiralp, and César Hidalgo. 2019. Sherlock: A deep learning approach to semantic data type de- tection. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD), pages 1500-1508.
Timo Schick and Hinrich Schüitze. 2021a. Few-shot text generation with natural language instructions. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 390-402.
Timo Schick and Hinrich Schütze. 202 lb. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages
Hiroshi lida Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. TABBIE: pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL-HLT), pages 3446-3456.
Yoshihiko Suhara, Jinfeng Li. Yuliang Li, Dan Zhang. Çagatay Demiralp, Chen Chen, and Wang-Chie -hhiw Tan. 2022, Annotating columns with pre-trained lan- guage models. In Proceedings of the 2022 Inter- national Conference on Management of Data (SIG- MOD), pages 1493-1503.
Christos Koutras George Siachamis, Andra Ionescu, Kyriakos Psarakis, Jerry Brons, Marios Fragkoulis, Christoph Lofi, Angela Bonifati, and Asterios Katsi- fodimos. 2021. Valentine: Evaluating matching tech- niques for dataset discovery. In 2021 IEEE 37th In- lernational Conference on Data Engineering (ICDE), pages 468-479.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi. Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (FMNNPP), pages 9275-9293.
Edward M Leonard. 2011. Design and implementation of an enterprise data warehouse. Marquette Univer- sity.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeulemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 16692.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 1337
S. Melnik, H. Garcia-Molina. and E. Rahm. 2002. Sim- ilarity flooding: versatile graph matching algorithm and its application to schema matching. In Proceed- ings 18th International Conference on Data Engi- neering (ICDE). pages 117-128.
Daheng Wang, Prashant Shiralkar, Colin Lockard, Binx- uan Huang, Xin Luna Dong, and Meng Jiang. 2021. Ten: Table convolutional network for web table in- terpretation. In Proceedings of the Web Conference 2021, page 4020-4032.
OpenAl. 2023, Gpt-4 technical report. arXiv preprint arX1v:2303.08774.

---

Jason Wei. Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Transactions on Machine Learning Research.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations (ICLR).
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint urXiv:1709.00133.
Yonghui Wu, Jun Xu, Yaoyun Zhang, and Hua Xu. 2015. Clinical abbreviation disambiguation using neural word embeddings. In Proceedings of BioNLP 15, pages 171-176.
Leonardo Zilio, Hadeel Saadany, Prashant Sharma, Diptesh Kanojia, and Constantin OrÅfsan. 2022. Plod: An abbreviation detection dataset for scien- tific documents. In Proceedings of the Language Resources and Evaluation Conference (IREC), pages 680-688
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang. Chengzu Li, Connor Boyle. Ansong Ni, Ziyu Yao. Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. Unifiedskg: Unifying and tastii structured knowledge grounding with text-to-text language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 602-631.
Jingfeng Yang, Aditya Gupta, Shyam Upadhyay. Luheng He, Rahul Goel, and Shachi Paul. 2022. TableFormer: Robust transformer modeling for table- text encoding. In Proceedings of the 60th Annual Meeting of Association for Computational Lin- guistics (ACL), pages 528-537.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data, Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 8413-8426.
Hong Yu, George Hripesak, and Carol Friedman. 2002. Mapping abbreviations full forms in biomedical ar- ticles. Journal of the American Medical Informatics Association, 2322222-222.
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga. Dongxu Wang. Zifan Li. James Ma, Irene Li. Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A large-scale labelel dataset for complex and cross pross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3911- 3921.
Dan Zhang, Madelon Hulsebos, Yoshihiko Suhara, Çağatay Demiralp, Jinfeng Li. and -hige Tan. 2020a. Sato: Contextual semantic type detection in tables. Proc. VLDB Endow, 13(12):1835-1888
Shuo Zhang and Krisztian Balog. 2020. Web table ex- traction, retrieval, and augmentation: A survey. ACM Transactions on Intelligent Systems and Technology. 1-3351133.

---

D_ID D_ NAME
E_NAME
10
Accounting
Allen
NAME
COMM
10
Accounting
Ward
Allen
14000
.4
Ward
13500
.3
Depart
Department Name
Name
Employec
Last
me
Name
30
Research
Martin
Allen
Russell
Sales
Turner
Ward
Partners
(a) An example of unionable relation extraction task.
(b) An example of joinable relation extraction task.
Figure 6: Unionable/Joinable relation extraction tasks with abbreviated column names. The column names in blue are the corrupted names. Column D_ID should match Department__ue D_NAME with DepattmmetNNNme and EAME with Name for unionable column pairs. Column E_NAME matches with column Employee_Nor Name for the joinable relation.
Pick#
CFL_team
Col.
Question:
How many CFL teams from York College?
Pick
CFL Team
College
27
Hamilton Tiger-Cats
Wilfrid Laurier
SQL:
Calgary Stampeders
York
SELECT COUNT CFL Team
29
Ottawa Renegades
California
FROM CFLDraft
30
Toronto Argonauts
York
WHERE College = "York"
Results:
Table name: CFLDraft
2
Figure 7: An table Question Answer (QA) task example. After corrupting the column names in the table (i.e., the names blue), the keyword the question 'college' may fail to locate the abbreviated column name Col..
A Appendix
tive relationships, is available, the lack of descrip- tive column names can hinder the effectiveness of schema-baser methods in identifying unionable and joinable columns.
A.1 The Effect of Abbreviated Column Names Table Understanding Tasks
Table Question Answering Table question an- swering (QA) aims derive answers from tables for natural language questions. Typically, the task requires two steps: translating natural language questions to corresponding SQL queries and then extracting the answers from the SQL queries. In order to test the effect of the abbreviations of the column names for Table QA, used the abbre- viated method in section 3.2.2 to corrupt the col- umn names tables of the WikiSQL (Zhong al., 2017) dataset and used the same training and eval- uation script in UnifiedSKG (Xie et al., 2022) with the T5-large model. The accuracy drops from 84.32 to 80.49, recorded in Table 1.
Unionable and Joinable Relation Detection. The unionable and joinable column relation extraction tasks are illustrated in Figure 6. The original re- call score of the schema-based relation detection (with expanded column name) in Table 1 is cited from the Valentine paper (Koutras al., 2021). The datasets used in Valentine were fabricated by corrupting the column names or the column cells. The results of the original datasets have the same column names in two tables, so the recall score is 1.0. The best schema-based method (i.e., Simi- larity Flooding (Melnik al., 2002)) drops from 1.0/1.0 to 0.63/0.56 (an average score of 0.595) unionable/joinable relation detection tasks with expanded and abbreviated column names. As il- lustrated in the Valentine study, even when other schema information, such as data types and transi-

---

A.2
The Logical Name Identification Algorithm
A.4 Annotation Interface
Figure shows the interface that the annotators used to produce both the abbreviated column names ("Cryptic Variant") and well-curated column names ("Well-Curated Variant") based on the column con- tents ("Original Column Name", "Sampled Col- Values") and table metadata ("Table Name", "Table Category", and "Table Description"), The " "Comments" " column includes some explanations or clarifications for ambiguous samples.
See Algorithm 12.
Algorithm Logical name identification
procedure RAGICAL_NAMEEE. if V then return
well-curated
else
x <- tokeriize(z)) x; E x do if fisdigi((a.) then:
continue
A.5 Experimental Setup
34 +- (emmmatize(s.)
if at & then return O
All fine-tuning experiments on GPT-2', GPT-2- XL2, GPT-Neo-1133D GPT-Neo-2.7IB and infer- ences were run using eight NVIDIA A100 GPUs. The inference results from GPT-4 was generated using OpenAl API calls The model weights for LLaMA-65B and Falcon-Instruct-400 models were downloaded and hosted using Text Genera- tion Inference (TGI) toolkit 8 We used a global batch size of 128 and a learning rate of 10 dur- ing the fine-tuning process. We used beam search with a beam size of 5 during inference, which per- formed better than decoding by sampling. The results of the evaluation benchmark reported in the experimental tables were obtained from the best checkpoint of each model on single run.
> not well-curated
return
P well-curated
A.3 Training/Evaluation Set Details
Some variations in the table size are included the training and evaluation set. As seen from Figure 8, most tables have fewer than 100 columns, and there could be some small tables with fewer than five columns since the columns that initially had abbreviated ambiguous names were filtered before generating abbreviated columns. We limit the num- ber of rows per table for training and evaluation sets to less than 1000.
Spie
Tran
Test
Counts
10a
10a
101
10a
100
150
200
Col/Table
Spli:
Tain
10
Counts
10
Ihttps://hugginggfac.cc co/gtt2
hntpps://huggingffce.o////
200
400
600
800
1000
# Row/Col
"https://platform. http://rrmmpppaa...o..o.oo
Figure 8: Additional statistics for NAMEGUESS train- ing and test datasets. Top: The distribution of the num- ber of columns per table. Bottom: The distribution of the number of rows (cells) per column.
hittos://ai..etta.com//b/
'https://huggingface wwttiiiee/
falcon-40b- instruct.
'https://gi thub. con/
con/huggingface//

---

Easy
Medium
Hard
Extra Hard
Model
EM
F1
B-F1
EM
F1
B-F1
EM
F1
B-F1
EM
F1
B-F1
GPT-2 (12MM
41.4
65.2
66.9
30.3
44.9
54.4
19.0
44.4
49.2
16.1
38.4
42.8
GPT-2 (158))
53.8
73.4
73.7
42.7
58.7
65.5
32.3
59.4
62.1
21.1
46.8
50.2
GPT-Neo (1.bD
54.3
72.8
74.4
47.2
62.4
68.5
36.8
62.9
64.8
23.6
48.1
51.6
GPT-Neo (277))
55.3
73.8
76.5
50.4
64.7
70.7
38.0
64.8
66.4
27.2
53.0
56.0
Falcon-Inst (00))
62.2
73.6
77.2
59.3
71.2
76.0
49.0
71.1
72.2
36.5
58.2
62.0
LLaMA (638)
62.2
73.9
77.2
62.9
72.1
76.6
52.3
72.7
73.6
37.9
57.9
60.9
GPT-4
71.7
81.1
81.2
79.9
87.6
89.5
72.0
87.3
86.8
54.7
74.4
75.7
Human
53.7
74.4
74.8
53.7
69.3
72.8
33.7
64.7
59.9
30.2
54.4
48.3
Table Exact match (EM), F1, and BertScore-F1 (B-F1) scores (%) of fine-tuned models, LLMs sion-finetuned). and human performance for four difficulty levels. Fine-tuned models.
A.6
Ablation Studies
A.6.1
Difficulty Breakdowns.
The full results all the models on four different difficulty levels are in Table 6.
A.6.2
The Effect of Different Jointly
Predicted Query Column Names K and Different Number of Sampled Rows N.
We evaluated the exact match (EM) scores ied values of K and N the LLaMA-65B model. From Table 7, we note that an increase the values of K and N results in rise in EM scores, although the rate of increase tends to diminish for higher K and N values. Thus, we set the values of and N at 10 for other models in the experiments.
#column name(K)
N=1
N= 10
#sampled row (N)
K=1
45.7
49.0
K=5
54.3
55.5
K=10
56.0
56.2
Table 7: Exact match scores (%) with different K and values for the LLaMA-65B model.

---

Table
Tabll
Table
Table
Table
Column
Original Column
Catepory
Economy and
opendata arat
Colleges Francisco
Community
Public private colle
Web Address
Economy
100
99 sf Loopenddte araf-pcte
Collegees in San Francisco [2011]
Community
Public and private colle 6
Location
Regression
101
st t.opendata vown faak
Housing Simulattinn Model
Community
In order to understand
year
Regression data for inclusionary
Economy and
opendata vown.
Housing
Model
Community
Rogression data for inclusionary Housing Simulation Model
Economy and Community
103
Lopeeddat vewn-fZak Ftak
order understand
NASDAQ
Regression data for inclusionary Housing
Economy and Economy and Community
Rogression data for inclusionary
105
oppenddtt
Housing Simulation Model
order understand
Given
Regression inclustonary Housing Simulation Model
Economy Community
106
105 sf opendeta vown-faak
In order to understand
Area
Sampled Column
Values
Cryptic Variant
Well-Curated Variant
Jomments
http://www.hteee
http://www.esll.ed.eo
http://www.ate www.pacdemmart..co
webAddy
Web Address
['atiitde::
37.7811643,,
"lnngiudee;;
122.4140251)
(llttitude:
37.79271570'
longttude:
122.42339857
[llatitude's
'37.77782233
longitude's
122.4103705')
33.754886837'
llngttudees
122.4203047)
(latiitude'
37.78574019/
llngttude'
122. (23.4030025))
('latitude':
Tongitude's
122. 122.45093511)
('latitude':
37.779962844,
longitude::
122.4109312)
Location
2001 2002 2003
2004 2005
2006
66.8855
29.23575
0.0 50.1526
am "Inc" for "including"
| 1950.400024
1385.31002
2175.439941
2205. 320068 2415.290039
2852.280029
mddq
Nasdoq
RETAIL/ENT
MIXED
MIPS
RESIDENT VISITOR MIXRES
Indusegivn
P
zonin givn
Zoning
28351.820021
8975.3538911
31286.901308
09111.092998
115221
56258.876269
AREA
Area
Figure 9; A screenshot for the interface used by annotators.


Reviews:
[{"review_id": "qpMP7t7wEL", "reviewer": "Reviewer_MEbZ", "summary": "", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful feedback and suggestions on our paper. We have carefully considered the points raised and would like to address them in our research.\n\n- **Handling Omitted Information**:\nReviewer 1 has raised a valid concern regarding the challenge of dealing with omitted information when using abbreviated column names in practical applications. We agree that this is an important aspect to consider. Our current problem scope does not directly address the semantic meaning expansion problem. We will acknowledge this limitation in the paper. Moreover, as outlined in the *future works* section, we believe that the incorporation of a codebook that associates abbreviations with their expanded meanings (including the omitted information) could mitigate the omitted information issue. As suggested in the feedback, if “glucose” stands for “fasting glucose”, then we can use this example as an in-context demonstration to help interpret similar terms. Another possible solution is to collect and utilize more table metadata information, for example if the table name contains the term “fasting”, then the original column name ‘glucose’ is most likely to be inferred as “fasting glucose”.\n\n- **Performance Gain in NLP tasks with Expanded Column Names**:\nWe appreciate Reviewer 1's insights into the capabilities of LLMs in understanding abbreviations. We have compared the performance of using abbreviated and expanded column names on three table understanding tasks, i.e., text2SQL, schema-based relation detection, and table QA in Table 1 of the *introduction* section. The performance drops on all the tasks when they have abbreviated column names. These results can show the benefits of the expanded column names for these NLP tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I am always greatly appreciating papers that give a presentation of new tasks, and provide the provision of new datasets and evaluate benchmarks. This is the primary reason for my positive comments (constituting over 90%), as such actions infuse new vitality into the field! Clearly, this paper deserves an acceptance.\n\n- The details are all clear, and the results look credible.", "reasons_to_reject": "I have few concerns that should be considered / or explained (e.g., in the limitation section):\n\n- In practical work, the challenge with abbreviated column names in tables often lies not in the recognization of an abbreviation, but rather in dealing with omitted information. For instance, in diabetes datasets, \"glucose\" might refer to random glucose, fasting glucose, and so on. Similarly, measurement methods can exhibit considerable variation due to different techniques being employed. Shouldn't we be focusing more on addressing these kinds of issues? Of course, I'm not a nitpicking reviewer – this paper is already quite good. However, I still encourage you to provide some explanations and analyses.\n\n- From my experience, LLM embeddings have shown good understanding for abbreviations. I'm not sure if the presented abbreviation cases are not particularly challenging or important (GPT-4 could easily handle this issue, and I personally think the extra hard case you give is not very difficult). Could you showcase and analyze how, through column name expansion, we can achieve significant gains in some certain NLP tasks? Maybe an additional experiment is required.", "questions_to_authors": "See above.", "paper_topic_and_main_contributions": "In this work, authors introduced an innovative NLP task known as \"column name expansion,\" along with a collection of datasets that encompasses human evaluations, data difficulty breakdown, and benchmark evaluations. In this paper, the detailed data collection steps, difficulty breakdown, logical name identification, abbreviation generation, human annotations are clearly described.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A.", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A.", "Justification_for_Ethical_Concerns": "N/A."}, "extra_text": {}}, {"review_id": "4DdwAX7sz1", "reviewer": "Reviewer_4P1D", "summary": "", "questions": "Q1.\nWhy the partition of the datasets with city open data only as test data?\n\nQ2.\nThere are several choices that lead to a very well curated dataset, which may not reflect the reality for datasets in the wild.\nWhy constraining the human annotation process to the rules of the generator? (point 4 in 3.3.1)\nWhy col names unclear or difficult have been removed? (again 3.3.1) this makes the dataset much easier and far away from reality\n\nQ3\nSec 3 states 8.9 tables, but  table 2 and fig 4 states 895 - which number is correct?\n\nQ4\nwhy reporting only the predictions with successful extractions (sec 5.2)? this seems to possibly favor LLMs - are these examples removed also from the other models? I d argue that LLMs \"fail\" in this case and should be reported", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate Reviewer 2's insightful feedback to enhance the clarity and accuracy of our work. We have carefully considered each of your questions and concerns and would like to address them accordingly.\n\nQ1. **Choice of Test Data Partition**: The decision to use part of the city open data as the test set is motivated by the practical constraints of human annotation costs. The choice of city open data as the test set was primarily driven by its larger table size and more diverse column headers, which makes it a suitable representative for evaluation.\n\nQ2. **About the Guidance of Human Annotation**: 1) For the evaluation datasets, we have kept both well-curated and abbreviated column names to capture the realistic scenarios encountered with real-world datasets. To make this clear, we will edit Point 3 of Section 3.3.1 as \"*Determine if the original column name is abbreviated or well-curated. If in a well-curated form, annotators are asked to provide an \"abbreviated variant\". Otherwise, annotators will add both a \"well-curated variant\" and an \"abbreviated variant\" to the benchmark*.\" 2) With Point 4 of Section 3.3.1 \"*When creating the abbreviated names, the abbreviated words can be combined using any of the combining rules defined for the abbreviation generator*\", we will clarify that it serves as a guidance rather than a strict limitation on how annotators should combine abbreviated words. This guidance is to ensure the diversity of annotations like \"*word-level abbreviation*\" and \"*acronym extraction*\". 3) We did not remove unclear or difficult abbreviated column names, but the annotations. To make this clear, we will edit the sentence \"*Note that the annotated column names that ... were discarded from the dataset.*\".\n\nQ3. **Inconsistency in Table and Section Numbers**: The accurate number of tables in the evaluation dataset is 895, as reflected in Table 2 and Figure 4. We will fix this in the final version of the paper.\n\nQ4. **Reporting Predictions with All Examples**: We understand your point and agree that it is crucial to provide a more comprehensive view of the model performance. To address this, we will augment the results in Table 4 by including, for example, the exact-match scores under $t'+q$ for Falcon-Inst.(40B), LLaMA(65B), and GPT-4, which are 49.1%, 54.0%, and 72.7%, respectively. This adjustment will provide a more balanced perspective on the performance of all models.\n\n\n\n**Presentation Improvements**:\n\n1) in 3.3.1, clarify what is the measured agreement: is this about giving the same abbreviation for a given column?\n \n- **Answer**: We had a separate group of annotators to do the quality audit. To determine if an annotation is reasonable, we employed a criterion where, if two out of three annotators are in agreement, the annotation is considered to pass this agreement measure.\n\n2. how the probabilities in 3.2.2 are assigned?\n\n- **Answer**: The probabilities assigned in section 3.2.2 are pre-assigned by the developers. This assignment is undertaken to maintain statistics that resemble those found in real-world datasets. By incorporating these probabilities, we strive to create a synthetic dataset that retains realistic characteristics, thus enhancing the applicability and relevance of our approach.\n\n3. in 3.1, \"Half of the column names being duplicates\" - how is this possible? all col names should be different, in general.\n\n- **Answer**: The reference to \"Half of the column names being duplicates\" pertains to the potential occurrence of identical column names across different tables. In real-world scenarios, certain column names, such as \"customer id,\" might appear repeatedly in various tables to signify the same type of information. To clarify, we do not mean that the column names within a single table are duplicated, but rather that identical column names may occur in different tables due to the nature of the data being represented.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A1\nPertinent and practical problem in dealing with tabular data. \nThe approach of treating column name expansion as a natural language generation issue is sound.\n\nA2\nThe creation of a large-scale training set and a human-annotated benchmark is a valuable contribution. The creation process shows rigor and understanding of the challenges in expanding abbreviated column names (abbreviation ambiguity, developer conventions, and domain-specific terminologies). \n\nA3\nThe evaluation of LLM against human performance on the NameGuess task is a good first reference for future research for this task.", "reasons_to_reject": "R1.\nVery limited coverage of work done on profiling tables to obtain metadata.\nThe paper should better position the work\nWhile the mentioned work (including turl) are not immediate baselines, it is not clear why tabular language models cannot be used immediately as a solution for the proposed task. Other related work include\n\nYoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çagatay Demiralp, Chen Chen, Wang-Chiew Tan:\nAnnotating Columns with Pre-trained Language Models. SIGMOD Conference 2022: 1493-1503\n\nDan Zhang, Yoshihiko Suhara, Jinfeng Li, Madelon Hulsebos, Çagatay Demiralp, Wang-Chiew Tan:\nSato: Contextual Semantic Type Detection in Tables. Proc. VLDB Endow. 13(11): 1835-1848 (2020)\n\nEnzo Veltri, Gilbert Badaro, Mohammed Saeed, Paolo Papotti:\nData Ambiguity Profiling for the Generation of Training Examples. ICDE 2023: 450-463\n\nnotice how the last one covers the ambiguity problem that is mentioned in the submission but not really tackled.\n\nR2. \nThe task is heavily dependent on training data generated using abbreviation lookups and probabilistic rules, i.e., with a synthetic approach. These rules might not scale well with strings that do not adhere to common abbreviation patterns or for languages other than the one studied.\n\nR3.\nSome important decisions in the study are not clear and require justifications. My major concerns are \n(i) the abbreviations in the test datasets are designed to be consistent with the abbreviation created for the synth training data;\n(ii) datasets have been polished of hard examples, thus widening the gap with real ones (see also R2)", "questions_to_authors": "Q1.\nWhy the partition of the datasets with city open data only as test data?\n\nQ2.\nThere are several choices that lead to a very well curated dataset, which may not reflect the reality for datasets in the wild.\nWhy constraining the human annotation process to the rules of the generator? (point 4 in 3.3.1)\nWhy col names unclear or difficult have been removed? (again 3.3.1) this makes the dataset much easier and far away from reality\n\nQ3\nSec 3 states 8.9 tables, but  table 2 and fig 4 states 895 - which number is correct?\n\nQ4\nwhy reporting only the predictions with successful extractions (sec 5.2)? this seems to possibly favor LLMs - are these examples removed also from the other models? I d argue that LLMs \"fail\" in this case and should be reported", "paper_topic_and_main_contributions": "This paper is about naming conventions in tabular data, specifically focusing on the challenge of abbreviated column headers that can cause confusion and impact data comprehension and accessibility. It introduces a new task, called NameGuess, which aims to expand abbreviated column headers into full names, formulating it as a natural language generation problem. \n\nThe main contributions are \n1. New task modeled as natural language generation problem. \n2. A data generation method that produces a large-scale training set consisting of 384K abbreviated-expanded column pairs.\n3. Human-annotated evaluation benchmark with 9.2K examples drawn from real tables. This benchmark spans various difficulty levels, providing a reference for comparing the effectiveness of various solutions.\n4. Comprehensive evaluation of LLMs of varying sizes and training strategies, including comparison to human performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çagatay Demiralp, Chen Chen, Wang-Chiew Tan:\nAnnotating Columns with Pre-trained Language Models. SIGMOD Conference 2022: 1493-1503\n\nDan Zhang, Yoshihiko Suhara, Jinfeng Li, Madelon Hulsebos, Çagatay Demiralp, Wang-Chiew Tan:\nSato: Contextual Semantic Type Detection in Tables. Proc. VLDB Endow. 13(11): 1835-1848 (2020)\n\nEnzo Veltri, Gilbert Badaro, Mohammed Saeed, Paolo Papotti:\nData Ambiguity Profiling for the Generation of Training Examples. ICDE 2023: 450-463", "Typos_Grammar_Style_and_Presentation_Improvements": "- in 3.3.1, clarify what is the measured agreement: is this about giving the same abbreviation for a given column?\n\n- it is not clear how the probabilities in 3.2.2 are assigned \n\n- in 3.1, \"Half of the column names being duplicates\" - how is this possible? all col names should be different, in general"}}, {"review_id": "FbutScyVK8", "reviewer": "Reviewer_hjkH", "summary": "", "questions": "1. The evaluation setup consists of decoder-only models; How do the encoder-decoder models, for e.g., let's say flan/t5-* family of models, perform on this task?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the insightful feedback from the reviewer, which highlights a pertinent aspect of our evaluation setup.\n\n1. **Evaluation Setup**: It's worth noting that our current access is limited to large-scale out-of-the-box decoder-only models, like GPT-based models, Falcon, and LLaMA models. As a result, we fine-tuned smaller decoder-only models to maintain a balanced and fair comparison. We recognize the significance of this aspect and plan to explore and incorporate more models, including encoder-decoder variants, in future research endeavors. This expansion will provide a broader understanding of model performance across different architectures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The significance of the NAMEGUESS task has been properly motivated. It is an essential task at the intersection of NLP and Database communities, which could have exciting implications within enterprise settings, wherein data assets typically sit within SQL databases, and their metadata, for example, column headers, are generally abbreviated.\n\n2. The paper is well-written and easy-to-follow. The analysis of the results and the ablation studies provide a complete picture of how the results vary with changing hyper-parameters/table contexts. Additionally, the benchmark dataset, if made publicly available, would engage and motivate researchers to build improved models on this task.\n", "reasons_to_reject": "None", "questions_to_authors": "1. The evaluation setup consists of decoder-only models; How do the encoder-decoder models, for e.g., let's say flan/t5-* family of models, perform on this task?", "paper_topic_and_main_contributions": "In this paper, the authors have introduced a new task called NAMEGUESS, which tackles expanding abbreviated column names in tabular data. Abbreviating column names is common while building tables within standard database systems; it is convenient for representation and use in code. However, it negatively impacts understandability and causes problems for downstream algorithmic tasks, such as text2SQL, TableQA, Column Relation discovery, etc. The authors have empirically demonstrated this assertion via experiments in Table 1, thereby firmly stating why this task is essential.\n\nThe authors introduce NAMEGUESS as a natural language task, unlike existing textual abbreviation expansion tasks, which tend to use a classification-style approach via a predefined set of candidate expansions. This makes NAMEGUESS a more challenging task that is much more aligned to the real world use-cases. For empirical evaluations, the authors have built a large-scale training/test dataset for this task.\n\nFor building the training dataset, the authors took the expanded column names as input and used the convention of the database developers to convert the expanded column names into abbreviated names. The tables were collected from seven public tabular dataset repositories (Sec 3.1). An expanded column name from a table is regarded as well-curated if all the tokens in this column name were found in a predefined vocabulary. Only well-curated expanded column names are converted into abbreviated names. The abbreviation generation step is largely rule-based, consisting of four primary schemes: Word-level abbreviation, Acronym extraction, word removal, and word-order change. This procedure yields 384,333 columns across 163,474 tables.\n\nA human-annotated evaluation dataset was built using 15 human annotators, wherein the annotators were asked to create an \"abbreviated variant\" for all the \"well-curated\" original column names using similar rules as the abbreviation generation step above. This step yields an evaluation benchmark consisting of 9,218 examples across 8,965 tables.\n\nThe authors compare the performances of small and large Language models (LMs) for this task in a zero-shot/finetuned setting. For finetuning, the authors finetuned the task-specific prompts. The training procedure was done using decoder-only models in an autoregressive manner. The model was queried for prediction until either an <EOS> or period token was returned. The authors analyzed different LMs on this task across different paradigms, such as model sizes, effect-of-finetuning, significance of table context data, etc., and provided a comprehensive analysis of various model performances.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are positive about the paper. They find the task well-motivated and the paper well-written. All reviewers rate the soundness of the paper high and two of the three are highly excited to see the paper published (the third is ambivalent). The reviewers raised issues that seem easy to fix before the camera-ready. All reviewers consider the paper reproducible.
Decision: Accept-Main