ROBUT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations
Year: 2023

Abstract:
Despite significant progress having been made in question answering on tabular data (Table QA), it’s unclear whether, and to what extent existing Table QA models are robust to taskspecific perturbations, e.g., replacing key question entities or shuffling table columns. To systematically study the robustness of Table QA models, we propose a benchmark called ROBUT, which builds upon existing Table QA datasets (WTQ, WIKISQL-WEAK, and SQA) and includes human-annotated adversarial perturbations in terms of table header, table content, and question. Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training, which significantly improves the robustness of Table QA models. Our data and code is publicly available at https: //github.com/yilunzhao/RobuT.

heading: Future Work
text: We will explore how to improve robustness without compromising accuracy in our future work. 6. 3 Analysis
To evaluate the quality of adversarial example generation, we conducted human evaluations to compare the quality of the examples generated by the LETA framework with those created by human annotators. We further provided case studies on common errors made by LETA. Comparison with Human Annotation For each perturbation type, we sampled 100 adversarial examples from both human annotation and LETA. Error Type
Example
Change original meaning
Original: How many districts were created in the 1900’s? Paraphrased: How many districts were created in the nineteenth century? Explanation: Should be twentieth
Mismatch with given prompt For the prompt of replacing carrier phrase Original: How many players scored more than 7 points? Paraphrased: How many athletes scored more than 7 points? Explanation: Should paraphrase the carrier phrase How many
Information missing Original: What are the names and stock code of companies whose headquarters are located in the United States? Paraphrased: Name some companies whose headquarters are located in the United States. Explanation: stock code is missing
HallucinationOriginal: What is the name and nation of the singer who have a song having “Hey” in its name? Paraphrased: What is the name and nation of the singer having a song named “Hey Ya!” Explanation: “Hey Ya!” does not appear in the given context
Table 6: Case study for common errors made by the LETA framework for NLQ perturbation. The colored text highlights model errors. Two evaluators were then asked to rate each sample on a scale of 1 (worst) to 5 (best) and determine which example was better, between the one created by human annotators and the framework. We also estimated the annotation cost of each perturbation type for both methods. The results in Table 5 demonstrate that LETA achieves comparable performance to human annotators for table header and NLQ perturbations, with much lower annotation cost. However, it still lags behind human annotators in terms of table content perturbations, we leave future work to design more effective prompting methods for table content augmentation. Error analysis of LETA generation Table 6 shows examples of perturbed questions generated by the LETA framework. We identified the following common mistakes that LETA are likely to make: 1) changing the original meaning of the questions; 2) not consistent with the demonstration in the given prompt; 3) missing important information from the original question; and 4) hallucination