Title: Parameterizing Context: Unleashing the Power of Parameter-Efficient Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing
Venue: NeurIPS 2023
Accepted: yes
Keywords: semantic parsing

continual learning

few-shot learning
Abstract: Continual table semantic parsing aims to train a parser on a sequence of tasks, where each task requires the parser to translate natural language into SQL based on task-specific tables but only offers limited training examples. 
Conventional methods tend to suffer from overfitting with limited supervision, as well as catastrophic forgetting due to parameter updates.
Despite recent advancements that partially alleviate these issues through semi-supervised data augmentation and retention of a few past examples, the performance is still limited by the volume of unsupervised data and stored examples.
To overcome these challenges, this paper introduces a novel method integrating parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) for training a continual table semantic parser. Initially, we present a task-adaptive PEFT framework capable of fully circumventing catastrophic forgetting, which is achieved by freezing the pre-trained model backbone and fine-tuning small-scale prompts. 
Building on this, we propose a teacher-student framework-based solution. The teacher addresses the few-shot problem using ICT, which procures contextual information by demonstrating a few training examples. In turn, the student leverages the proposed PEFT framework to learn from the teacher's output distribution, and subsequently compresses and saves the contextual information to the prompts, eliminating the need to store any training examples.
Experimental evaluations on two benchmarks affirm the superiority of our method over prevalent few-shot and continual learning baselines across various metrics.

Parameterizing Context: Unleashing the Power of Parameter-Efficient Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing

Abstract

Continual table semantic parsing aims to train a parser on a sequence of tasks, where each task requires the parser to translate natural language into SQL based on taskspecific tables but only offers limited training examples. Conventional methods tend to suffer from overfitting with limited supervision, as well as catastrophic forgetting due to parameter updates. Despite recent advancements that partially alleviate these issues through semi-supervised data augmentation and retention of a few past examples, the performance is still limited by the volume of unsupervised data and stored examples. To overcome these challenges, this paper introduces a novel method integrating parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) for training a continual table semantic parser. Initially, we present a task-adaptive PEFT framework capable of fully circumventing catastrophic forgetting, which is achieved by freezing the pre-trained model backbone and fine-tuning small-scale prompts. Building on this, we propose a teacher-student framework-based solution. The teacher addresses the few-shot problem using ICT, which procures contextual information by demonstrating a few training examples. In turn, the student leverages the proposed PEFT framework to learn from the teacher's output distribution, then compresses and saves the contextual information to the prompts subsequently, eliminating the need to store any training examples. Experimental evaluations on two benchmarks affirm the superiority of our method over prevalent few-shot and continual learning baselines across various metrics.

1 Introduction

Tabular data is a crucial source of information for human decision-makers in various domains, including healthcare [1], retail [2] and finance [3]. Although structured query language (SQL) provides an efficient means of accessing this data, a natural language interface would make it more accessible to non-technical users. Consequently, there has been a growing interest in Table Semantic Parsing (TSP), which involves mapping natural language queries over tabular data to formal programs.

Current research [4, 5, 6] on TSP have covered many scenarios, most of which assume the datasets are static. In contrast, recent work [7] highlights the fact that new tables are often constantly emerging in response to changing circumstances, leading to a continual stream of TSP tasks. With this comes two new challenges: a) few-shot problem [8, 9, 10]: For a new task against unseen tables, the annotated data available in a short period is typically limited, causing the parser to be prone to over-fitting. b)

Task 1

Flights

…

Task 2

Geography

Task 3

Movies

PLM

PLM

PLM

Input

Prompt 1

Prompt 2

Prompt 3

Input

Input

Few-Shot

Few-Shot

Few-Shot

Overfitting

PLM

Demo 1

Input

PLM

Demo 2

Input

PLM

Demo 2

Input

Catastrophic

Forgetting

PEFT

ICT

catastrophic forgetting [11, 7]: After training on a new task, the performance of the parser on the previous task may plummet attributed to parameter updates. While [7] mitigates these challenges using semi-supervised learning and replaying past examples, however, its performance is still limited by the amount of unsupervised data and replayed examples. More realistically, in some scenarios involving data privacy and timeliness, such as patient cases and transaction records, both unsupervised data and past examples may not be available in the following tasks.

Fortunately, recent research yields new ideas to tackle the issues. a) in-context tuning (ICT) [12, 13, 14] enhances models' few-shot learning capability using demonstrations containing a few training examples. b) parameter effective fine-tuning (PEFT) [15, 16, 17] fundamentally eliminates catastrophic forgetting by avoiding parameter updates of the pre-trained model (PLM) during the training stage and instead tuning a group of parameters of bearable size individually for each task. Nevertheless, the two solutions can merely handle the corresponding challenge while still suffering from the other. As is illustrated in Figure 1: for ICT, the invisibility of past demonstrations causes catastrophic forgetting when the model experiences a new task; for PEFT, though proposed to improve generalization, it still suffers from severe overfitting as only a few training examples are available for each new task.

In this paper, we propose a new method to simultaneously tackle both of these challenges by integrating PEFT with ICT. We start with a task-adaptive parameter-efficient continual TSP framework, which freezes the backbone PLM and only tunes the additional prompt embeddings for each distinct task. Unlike existing methods [16, 17], our backbone PLM undergoes fine-tuning on the initial task to facilitate adaptation to the TSP task format. Based on this foundation, we propose a Context-Compressed Continual table semantic parser (C3). C3 employs a teacher-student framework, comprising two parsers that symbolize the TEACHER and the STUDENT, respectively. For each task, the TEACHER utilizes ICT to glean contextual information from semantically akin demonstrations extracted from the training set, thereby mitigating the risk of overfitting on the few-shot data. Subsequently, the STUDENT adopts our proposed parameter-efficient framework to fine-tune the prompt embeddings and learns the output distribution of the TEACHER for each training example. This method ensures that the contextual information gleaned by the TEACHER can be preserved and compacted into a parameterized form, which can be reloaded later even when the demonstrations of past tasks are no longer available. Extensive experiments performed on two TSP task streams substantiate the effectiveness of our method, which surpasses all compared baselines, achieving state-of-the-art results across multiple metrics. In summary, the contributions of this paper include:

Weintroduce a task-adaptive, parameter-efficient continual framework for table semantic parsing. It can completely avert catastrophic forgetting by tuning only 0.05% of the original PLM parameters.

We propose to utilize the teacher-student framework to fuse PEFT and ICT so that the table semantic parser can benefit from both their advantages in solving few-shot data and catastrophic forgetting. To the best of our knowledge, this is the first time that the two technologies have been integrated into the scenario of the task streams.

We create two task streams derived from two mainstream table semantic parsing benchmarks, and carry out extensive experiments using these task streams. The results demonstrate that our method surpasses existing competitors, achieving state-of-the-art performance across multiple metrics.

2 Preliminaries

2.1 Table Semantic Parsing

Formally, given a natural language question Q and a schema S = ( C , T ) of multiple tables, the goal of TSP is to generate a SQL query Y = F θ ( Q , S ) , where F θ denotes a neural semantic parser with parameters θ . T = { t 1 , . . . , t |T | } and C = { c 1 , . . . , c |C| } are the sets of table names and column names, respectively, where each column c i ∈ C belongs to only one table t j ∈ T .

In this paper, we focus on TSP in this paper for the following reasons: a) TSP is important for enhancing various fields through its role in Natural Language Interfaces to Databases (NLIDB). b) TSP is a challenging task requiring comprehensive understanding and mapping of queries to structured tables, presenting a few-shot learning challenge due to the difficulty in obtaining supervised data. c) TSP provides objective evaluation metrics, as minor errors in SQL queries can lead to significant outcome discrepancies, thereby offering a true reflection of our method's performance.

2.2 Problem Formulation

In this paper, we assume that F θ is no longer limited to fixed training and testing data, but has to face a continual stream of TSP tasks, each corresponding to a different set of tables. Formally, F θ is sequentially trained on a stream of K tasks {D 0 , D 1 , . . . , D K -1 } . Each task D i consists of a training set D i train , a validation set D i valid , and a test set D i test . Each example e = ( X , Y ) , where X = ( Q , S ) , is the input and Y denotes the target SQL. For ∀D i , ∀D j , if i = j , then S ( D i ) ∩ S ( D j ) = ∅ , where S ( D i ) denotes the total set of corresponding table schemas in D i . Here S also serves as a task identifier for each e , enabling efficient indexing of the corresponding task. Our ultimate objective is to train a F θ that can achieve good accuracy on each D i test after experiencing all K tasks sequentially.

̸

3 Parameter-Efficient Continual Table Semantic Parsing

We start with a vanilla parameter-efficient continual TSP framework, which trains a backbone semantic parser with a task-adaptive continual prompt-tuning method.

3.1 Backbone Semantic Parser

The pre-trained Text-to-Text Transfer Transformer (T5) [18] is selected as our backbone parser F θ for two primary reasons. Firstly, prior studies [16] have shown that the combination of T5 with PEFT methods can result in significant performance gains across various NLP tasks. Secondly, T5 has demonstrated robust capabilities [19] in generating logical forms on a widely-used TSP dataset Spider [5]. In particular, each input pair X = ( Q , S ) is flattened into a plain text,

where c t i j denotes the j -th column name of the i -th table, and " : ", " , ", and " | " are predefined separators. Subsequently, X ′ is fed to the parser F θ and the generative probability of a predicted SQL ˜ Y is estimated by P ( ˜ Y|X ′ , θ ) = ∏ | ˜ Y| j =1 P (˜ y j | X ′ , ˜ y <j , θ ) , where y j ∈ V denotes the j -th token of ˜ Y and V is the vocabulary of the T5 tokenizer. X ′ are the hidden states of X ′ obtained by the T5 encoder, and P (˜ y j | X ′ , ˜ y <j , θ ) denotes the output probability of ˜ y j from the T5 decoder.

3.2 Task Adaptive Continual Prompt Tuning

Prompt tuning [20], as a representative of PEFT-based methods, has been widely applied in recent NLP tasks [16, 17] owing to its simplicity. Inspired by these works, we employ prompt tuning to conduct the PEFT training framework for F θ . Formally, each input ¯ X in task D k is prefixed with M special prompt tokens P k = p k 1 p k 2 . . . p k M . During training, only the embeddings of these tokens are updated, rather than the parameters of the entire T5 model. More specifically, the training process can be divided into the following two phases.

Task

𝒊 - 𝟏

…

Task

𝒊

Task

𝒊 + 𝟏

Teacher

Demo

Input

Demo

Input

Demo

Input

Teacher

Teacher

Student

Student

Student

Distribution

Input

Prompt

𝑖 - 1

Distribution

Distribution

Input

Prompt

𝑖

Input

Prompt

𝑖 + 1

𝐷

!"

𝐷

!"

𝐷

!"

…

3.2.1 Initial Task Adaptation

Intuitively, the original T5 is not well-suited to TSP, as it is not exposed to any logical form during pre-training and is never tasked with predicting structured outputs. While this limitation can be readily addressed through fine-tuning, we hypothesize that tuning prompts alone may not be sufficient.

Therefore, we first fine-tune T5 with D 0 train to obtain a set of task-adaptive parameters θ that will be frozen on D 1 , . . . , D K -1 . Notably, here the prompt embeddings are tuned as well as θ because we expect to acquire a good initialization for both of them. In particular, for each e = ( X , Y ) ∈ D 0 train , the following loss is optimized using the AdaFactor [21] algorithm,

where [; ] represents the concatenation of matrices, P 0 ∈ R M × d denotes the tunable embeddings of P 0 , and ¯ P 0 ∈ R M × d represents its hidden states from the T5 encoder. In our experiments, P 0 is randomly initialized. After multiple iterations, the pair of θ and P 0 with the highest validation accuracy, denoted by ( θ ∗ , P ∗ ) , is selected for initializing the subsequent K -1 tasks.

3.2.2 Continual Prompt Tuning

Unlike existing methods [22], for D i ( 0 < i < K ), we initialize all P i with P ∗ instead of P i -1 because our experimental results (detail in Section 5.3.2) reveal that the latter does not yield significant performance improvements. Furthermore, this strategy enhances the flexibility of the proposed framework by eliminating the dependency on the order of subsequent tasks, as all prompts share a common initialization. During training, P i is updated with the gradient ∇ P i ∑ e ∈D i train L ( e ; θ ∗ , P i ) using the AdaFactor optimizer, where L ( e ; θ ∗ , P i ) is calculated by Equ. (2). We experimentally set the prompt dimension d = 512 and the prompt number M = 150 , leading to a total of only 0 . 7 million parameters. This size is negligible ( 0 . 05 %) compared to the entire T5-BASE of 220 million parameters. Consequently, the cost of storing all P 1 , . . . , P K -1 is usually acceptable. During inference, given a test example e i test ∈ D 0: K -1 test , its task index i can be first identified using its table schema S . Then, F can solve it by loading the saved ( P i , θ ∗ ) without any forgetting.

In theory, our framework can also be expanded to the tasks whose tables have not been seen in D 0 , . . . , D K -1 . The prediction can be made using simple heuristics, such as averaging all P i . Since this scenario involves the zero-shot learning problem, we leave it for future work.

4 C3 Semantic Parser

Figure 2 depicts the illustration of our proposed Context-Compressed Continual table semantic parser (C3). C3 is typically composed of two parsers (e.g., T5) of identical architecture (optional), namely TEACHER F tea and STUDENT F stu, and each possesses its own individual parameters during training.

4.1 Context-Enhanced Teacher Parser

The sole purpose of F tea is to gain the few-shot learning capability and thereby in-context tuning (ICT) is employed to train F tea. Note that F tea only focuses on each current task D i and does not care if the past tasks are forgotten. Thus, it is sequentially trained on D 0 , . . . , D K -1 without any continual

learning. During D i , the parameters of F tea, denoted by θ i tea , is initialized with θ i -1 tea . Afterwards, θ i tea is updated by optimizing L ( x ; θ tea ) = -∑ | ˜ Y| j =1 log P (˜ y j | C , y <j , θ tea ) . Here C denotes the hidden states of the input text, C , which is constructed by retrieving and formatting the demonstrations.

4.1.1 Demonstration Retrieval

The key to ICT lies in identifying the suitable demonstration that effectively facilitates the learning process of the model within the appropriate context. Recent studies [23, 24] have reached a consensus that the closer the demonstration aligns with the example being queried, the greater the benefits of ICT. Thus, for each e , we retrieve the demonstration by comparing its semantic similarity to the candidate demonstrations. To maintain overall uniformity, we select the original T5 checkpoint which was fine-tuned on the Semantic Textual Similarity Benchmark (STS-B) [25], as an off-the-shelf retriever R . Specifically, for example e to be queried and a candidate demonstration e ′ j ∈ D i train \ { e } , we first wrap them into a template T ( e, e ′ j ) = stsb sentence1: Q , sentence2: Q ′ j , where Q and Q ′ j represent the natural language questions of e and e ′ j , respectively. Here the target SQL Y is not considered because it is invisible in the testing stage. Subsequently, the similarity score of e ′ j , denoted by σ ( e ′ j ) , is calculated by σ ( e ′ j ) = R ( T ( e, e ′ j )) ∈ [0 , 5] . To circumvent the noise, we set a threshold η and eventually select the topr e ′ j ∈ D i train \ { e } whose σ ( e ′ j ) ≥ η as the demonstration of e . In fact, not every example e ∈ D i train has sufficient demonstrations. To retain F i tea 's ability in handling no-demonstration examples, we apply a simple but effective context mixing strategy during training: both C and X ′ are engaged in the training for each e ∈ D i train if it has r demonstrations, otherwise only X ′ is engaged.

4.1.2 Demonstration Formatting

r retrieved demonstrations e 1 , . . . , e r compose the input C = Q 1 |Y 1 | . . . |Q r |Y r ||X ′ , where both | and || are separators, and X ′ is obtained by Equ. (1). The decision to exclude table schemas S from C is motivated by one main consideration: the selected demonstration examples and the example e to be queried are likely to be associated with the same batch of tables due to their similarity. Therefore, the explicit information regarding schemas need not be provided repeatedly outside of X ′ , which helps to streamline the input to improve efficiency.

4.2 Context-Compressed Student Parser

F i stu follows the PEFT framework introduced in Section 3 and is dedicated to compressing and preserving the few-shot capability F i tea learns from the demonstration into the prompt embeddings. In this way, F i stu can reproduce the few-shot performance by loading the saved prompt embedding, without accessing any examples or demonstrations of previous tasks. Inspired by [26], We hypothesize that this goal could be accomplished as much as possible by having F i stu learn the F i tea 's output distribution. More concretely, we minimize the KL divergence between F i stu 's per-step autoregressive distribution and the F i tea 's, denoted by D KL ( P ( θ tea ) ∥ P ( θ stu )) where P ( θ ) = P ( y j | C , y <j , θ ) . During training, the teacher-forcing strategy is applied to ensure that the inputs to the decoders of F i tea and F i stu are consistent at each decoding step. In addition, to guarantee the quality of supervision, we directly use the gold SQL Y instead of P ( θ tea ) if F i tea 's prediction for a given input X is incorrect.

5 Experiments

5.1 Experimental Setup

5.1.1 Datasets & Evaluation Metrics

Our experiments involve two widely-used TSP datasets: WikiSQL [4] is a dataset for single-table semantic parsing in which each SQL query corresponds to a single table and adheres to a simplified SQL syntax. Spider [5] is a more challenged dataset in which each SQL requires the JOIN operation of multiple tables and contains complicated syntax including GROUP BY even nested query.

As explicated in Section 2.2, we constructed two task streams based on the above datasets. The first one, Spider-Stream , is established by merging the Spider training and validation sets and splitting

2

4

6

8

10

12

14

16

0

200

400

600

Example Number

1

2

3

4

5

6

7

8

9

10 11

0

200

400

Example Number

D

i

train

D

i

valid

D

i

test

them into 16 tasks according to the domain of the tables. It is used to reflect the impact of the changing table domains. The second one, Combined-Stream , is a merged task stream encompassing 11 tasks. The first 8 tasks are randomly selected from the Spider-Stream and encompass complex SQL syntax, whereas the last 3 tasks are collected from WikiSQL and involve only simple SQL syntax. It is utilized to assess the dual impact of table domains and target SQL structures.

To highlight the few-shot challenge, the majority of tasks D i train have |D i train | < 500 . For each method, we evaluated its performance in three random task orders, to eliminate any potential influence of specific tasks. Considering that in real applications, models are usually fully trained in certain domains before facing the task stream, we combined the first N (6 for Spider-Stream and 5 for Combined-Stream) tasks of each task order into a single task. This practice enables all methods to have full adaptation to the TSP task format. In addition, for Combind-Stream, to aggravate the gap between adjacent tasks, for any task order, we always make the tasks of the even index (0, 2, . . . ) from Spider and the tasks of the odd index (1, 3, . . . ) from WikiSQL.

Following [27, 11], we adopted four metrics: 1) Task Accuracy: TA = 1 K ∑ K -1 i =0 a i,K -1 ; 2) Example Accuracy: EA = a D (0: K -1) test ; 3) Initial Accuracy: IA = 1 K ∑ K -1 i =0 a i,i ; 4) Memory Decay: MD = 1 K -1 ∑ K -2 i =0 a i,K -1 -a i,i ; where a i,j denotes the accuracy on D i test after the training of D j . TA and EA reflect the overall performance at the task level and example level, respectively. MD measures the extent of forgetting. IA evaluates the performance when the parser is trained on a new task for the first time, considering no forgetting.

5.1.2 Compared Methods & Implementation Details

Our compared methods can be categorized into three groups: a) FINE-TUNING, which trains the parser sequentially without any strategies; b) Few-Shot Learning baselines, which contain MAML [8] and ICT [13]; c) Continual Learning baselines, which consist of EWC [28], HAT [29], EMR [27], EMAR [30], APPER [31], and TR [11]. To increase the competitiveness of these baselines, we utilized pre-trained GRAPPA-LARGE [32] for TSP in conjunction with SemQL [33], an intermediate representation for SQL. To ensure a fairer comparison, we conducted experiments applying the best-performing methods on GRAPPA, EMR and EMAR, to T5-BASE as complementary baselines. Additionally, we compared our methods with SFNet [7] which utilizes extra unsupervised data. Note that we only apply [7] on Combined-Stream for the reason that Combined-Stream randomly selects a portion of samples from the original Spider and WikiSQL datasets which enables the rest to be used as unsupervised data to fulfill the scenario requirements. We also set an upper boundary method MULTI-TASK. For each task D i , it trains F θ with D (0: i ) train .

Our method ran on one NVIDIA RTX 4090 GPU and the hyperparameters were decided with the validation sets (See Appendix for details): a) prompt length M is set to 150 ; b) demonstration number r and threshold η are set to 1 and 4 . 0 , respectively; c) the batch size is 12 and the learning rates are set to 0 . 3 and 1 × 10 -4 for prompt-tuning and fine-tuning, respectively; d) early stopping is performed after 1000 epochs and the patience is set to 10 evaluations, which is performed every 50 epochs. e) T5-LARGE is always employed as F tea. All of our data and codes are publicly available 1 .

5.2 Overall Results

Table 1 presents the overall performance of the methods. The results indicate that most of the compared methods exhibit relatively lower effectiveness and more severe forgetting on CombinedStream compared to Spider-Stream, even though their backbone models have been pre-trained for the

0

1

2

3

4

5

6

7

8

9 10

50

60

70

TA (%)

0

1

2

3

4

5

6

7

8

9 10

50

60

70

EA (%)

0

1

2

3

4

5

6

7

8

9 10

-

20

-

10

0

MD(%)

FINE-TUNING

MAML

ICT

EWC

HAT

EMRGRAPPA

EMARGRAPPA

EMRT5-BASE

EMART5-BASE

APPER

TR

SFNet

PEFTT5-BASE

PEFTT5-LARGE

C3T5-LARGE

MULTI-TASK

0

1

2

3

4

5

6

40

60

TA (%)

0

1

2

3

4

5

6

20

40

60

EA (%)

0

1

2

3

4

5

6

-

60

-

40

-

20

0

MD(%)

tabular data. Surprisingly, our proposed PEFT framework outperforms all baseline models despite the absence of any explicit provision to address the few-shot problem, mainly due to its ability to retain previously learned knowledge. Moreover, our proposed C3 further improves the overall performance (4.9% & 5.1% in terms of TA) and achieves state-of-the-art results for both datasets. Of particular note is that even with a smaller backbone model (T5-BASE, 220M), C3 outperforms all compared methods based on GRAPPA-LARGE (340M). Remarkably, it manages to approach the upper bound of performance (MULTI-TASK) on Combined-Stream.

Compared to ICT, MAML performs poorly on both datasets, especially in terms of MD, even less than FINE-TUNING. This finding may point out that it is more profound and irreversible for parameter updates. The replay-based EMR and EMAR outperform other continual learning baselines in terms of all metrics, which suggests that retaining past instances may be more direct and effective for mitigating forgetting, particularly on Combined-Stream where the task varies more. Unfortunately, in many cases involving data privacy, access to past examples is disabled, rendering these methods limited. Conversely, our proposed C3 not only circumvents data privacy concerns and has a broader range of applications, but also achieves superior results without using any past examples.

5.3 Detailed Results and Analysis

5.3.1 Performance Till the Seen Tasks

Figure 4 displays the methods' performance till the seen tasks on both datasets. Our proposed C3LARGE (blue) consistently exhibits the highest performance across all evaluated time steps. Moreover, this performance advantage becomes increasingly pronounced as the number of tasks in the continual learning regime grows. Interestingly, the EA metrics on the Combined-Stream for compared methods always show oscillation. The performance troughs on the even-indexed tasks suggest a dramatic forgetting of the harder Spider tasks after the model is trained on an easier WikiSQL task.

5.3.2 Ablation Study

We compared the performance of the proposed C3 with the following settings:

w/o TEACHER : we removed F tea to verify its contribution;

w/o STUDENT : we removed F stu and only employed F tea (T5-LARGE) to perform ICT, to evaluate the contribution of PROMPT-TUNING;

w/o In-Context Tuning : we used FINE-TUNING to train F tea to assess the necessity of ICT.

w/o Task Adaptation : we used PROMPT-TUNING to train F stu from D 0 .

w Continual Initialization : we initialize the prompt embeddings P i with P i -1 instead of P ∗ .

Table 2 shows the TA (%), EA (%), and MD (%) of different settings. Our C3 equipped with all components performs best in terms of all metrics. Dramatic performance degradation demonstrates that the adaptation of the PLM to the task format is critical to the effectiveness of prompt tuning. Removing F tea results in an approximate 2% drop in terms of both TA and EA, which proves the TEACHER's few-shot learning capability contributes to the entire model. With the exclusion of STUDENT, the TA exhibits a more pronounced decrease on Combined-Stream (-6.8%) compared to Spider-Stream (-2.8%) because Combined-Stream involves challenges of both domain changes and SQL structure changes. The performance drops (-2.3% & -0.7%) brought by abandoning ICT reveal the necessity of contextual information. Its smaller contribution on Combined-Stream is probably due to the relative simplicity of the WikiSQL tasks, which does not require a demonstration to make

1

5

20

75 150

20

40

60

TA (%)

T5-SMALL

1

5

20

75 150

20

40

60

2

.

0

3

.

0

4

.

0

74

76

78

IA (%)

2

.

0

3

.

0

4

.

0

70

72

r

= 1

∗

r

= 1

r

= 2

r

= 3

a correct prediction. In contrast to [16]'s assertion, our experiments did not show any improvement with Continual Initialization. One possible reason is the large gap between different tasks.

5.3.3 Impact of Large Language Models

Considering the amazing potential of Large Language Models (LLMs), such as GPT-3 [34] and GPT-4 [35], in the NLP community recently, we also evaluate the performance of C3 by replacing F tea from T5-LARGE with the text-davinci-003, an LLM of the GPT family that excels in textual tasks. Here we did not choose code-davinci-002, which specifically handles code tasks, and the more widely used gpt-3.5-turbo, because the API of the former is deprecated and the latter can not provide output distribution for each decoding step. Referring to [26], we align the GPT tokenizer with the T5 tokenizer in order to calculate D KL (details in Appendix). Here we let text-davinci-003 perform inference based on the extracted demonstration without tuning its parameters, so C3 in this scenario is independent of the task order. The experimental results are shown in Table 3. Despite not having undergone any fine-tuning, using GPT has achieved close or even better results compared to using T5-LARGE. This proves, to some extent, that our proposed method is not limited to a specific architecture (encoder-decoder vs. decoder-only) and size (million-level vs. billion-level) of PLM.

5.3.4 Impact of Different Prompt Lengths & PLM Sizes

We varied the backbone PLM in {T5-SMALL, T5-BASE} and prompt length M in { 1 , 5 , 20 , 75 , 150 } for our proposed C3 on the two task streams. The experimental results are shown in Figure 5, which indicates that: When fixing the prompt length, increasing the backbone PLM size improves the TA; When the PLM size is fixed, increasing the prompt length improves the overall performance in general. Moreover, our findings indicate that extending the prompt length from 20 to 75 enhances TA more significantly than increasing it from 75 to 150 on both task streams.

5.3.5 Impact of Different Demonstration Numbers & Thresholds

We also varied the demonstration number r in { 1 , 2 , 3 } and threshold η in { 2 . 0 , 3 . 0 , 4 . 0 } for the TEACHER F tea in our proposed C3. Figure 5 shows IA (%). We observed that removing the context mixing strategy leads to severe performance degradation, proving our hypothesis that there exists many examples lack of sufficient demonstrations. When using context mixing, the performance is not significantly affected by the threshold and number of demonstrations, but a trade-off between performance and efficiency is achieved with a threshold of 4 . 0 and using only one demonstration.

5.3.6 Performance of TEACHER Parser with Always-Visible Demonstrations

While in many works involving in-context learning there exists an always-visible demonstration pool (e.g., public or synthetic datasets) from which we can select demonstration for all tasks, resources for TSP tasks are quite limited due to high expense of annotation. Therefore, C3 applies a practical strategy which samples demonstrations from every incoming task. However, to study the above sampling strategy, we conducted an additional experiment assuming that D 0 train , the training set for task 0, is an always-visible public demonstration pool from which all demonstrations for subsequent tasks are sampled. Table 4 shows IA (%) of a T5-LARGE based teacher parser. From the results, when sampling the demonstration from D 0 train , the performance of the teacher parser shows a significant drop on both datasets. This reflects the conclusions in [23, 24], which state that performance gains are maximized only when in-context learning utilizes demonstrations similar to the test examples.

For domain-specific databases, there is no guarantee that similar demonstrations will be found using public data resources. Our results prove that that the quality of the demonstration has a great impact on the performance of ICT.

6 Related Work

Table Semantic Parsing In recent years, there has been a notable increase in interest concerning joint textual-tabular data understanding problems [36, 37], especially in table semantic parsing (TSP) [4, 5]. This trend can be attributed to the fact that many real-world datasets contain a combination of structured and unstructured data, and a comprehensive understanding of such data requires the ability to analyze and interpret both forms effectively. The related methods can be divided into three directions according to the application scenarios, namely single table [4, 38, 10, 39], multitable [5, 33, 40, 41] and conversational tasks [42, 6, 43, 22].

Parameter Efficient Continual Learning Prompt tuning was first systematically studied by [20], who demonstrated that fine-tuning language models on a small number of task-specific input prompts could result in substantial performance improvements with much lower computational costs compared to full-model fine-tuning. Afterward, several works [44, 17, 16, 45] applied it to the continual learning scenario and achieved promising results for mitigating catastrophic forgetting. Unlike them, we leverage in-context tuning to compensate for the shortcomings of prompt tuning on few-shot tasks.

Few-shot In-Context Learning Recently, in-context learning has emerged as a promising approach to improve the performance of large language models (LLMs) such as GPT-3, particularly in scenarios where only limited training data is available [34]. However, LLMs are often computationally expensive to train and inference, which limits their practical use in many real-world applications. To address this limitation, recent studies [12, 13, 14] focus on extending the capabilities of in-context learning to medium- or small-scale pre-trained language models (PLMs) by tuning them with the demonstrations. Inspired by these works, we propose to utilize ICT in a parameter-efficient framework as a means to acquire contextual information.

7 Conclusion & Future Work

In this paper, we present a few-shot continual learning method that integrates parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) to handle the task stream in table semantic parsing. The teacher model is designed to extract contextual information from sampled demonstrations for few-shot learning, while the student model is tasked with learning the teacher's output distribution. It employs parameter-efficient prompt-tuning to retain its capabilities and to entirely eradicate catastrophic forgetting. Through extensive experimentation, we've demonstrated that our method surpasses all compared baselines. In our future research, we aim to investigate strategies to ensure the model's zero-shot predictive capabilities, while simultaneously preventing catastrophic forgetting. Theoretically, our proposed method is generalizable and has the potential to be adapted to other NLP tasks because it is not limited to specific backbone models and does not involve task-specific modules. We leave the evaluation of C3 on other tasks to our future work.

References

Wang, P., T. Shi, C. K. Reddy. Text-to-sql generation for question answering on electronic medical records. In Y. Huang, I. King, T. Liu, M. van Steen, eds., WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020 , pages 350-361. ACM / IW3C2, 2020.

Yuan, Y. Analysis and design of human resource management model of retail enterprises under the background of big data. In M. A. Jan, F. Khan, eds., Application of Big Data, Blockchain, and Internet of Things for Education Informatization , pages 31-37. Springer International Publishing, Cham, 2021.

Stockinger, K., N. Bundi, J. Heitz, et al. Scalable architecture for big data financial analytics: user-defined functions vs. SQL. J. Big Data , 6:46, 2019.

Zhong, V., C. Xiong, R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR , abs/1709.00103, 2017.

Yu, T., R. Zhang, K. Yang, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In E. Riloff, D. Chiang, J. Hockenmaier, J. Tsujii, eds., Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018 , pages 3911-3921. Association for Computational Linguistics, 2018.

Yu, T., R. Zhang, H. Er, et al. Cosql: A conversational text-to-sql challenge towards crossdomain natural language interfaces to databases. In K. Inui, J. Jiang, V. Ng, X. Wan, eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pages 1962-1979. Association for Computational Linguistics, 2019.

Chen, Y., X. Guo, T. Wu, et al. Learn from yesterday: A semi-supervised continual learning method for supervision-limited text-to-sql task streams. CoRR , abs/2211.11226, 2022.

Guo, X., Y. Chen, G. Qi, et al. Improving few-shot text-to-sql with meta self-training via column specificity. In L. D. Raedt, ed., Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022 , pages 4150-4156. ijcai.org, 2022.

Chang, S., P. Liu, Y . Tang, et al. Zero-shot text-to-sql learning with auxiliary task. In The ThirtyFourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 7488-7495. AAAI Press, 2020.

Chen, Y., X. Guo, C. Wang, et al. Leveraging table content for zero-shot text-to-sql with meta-learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, ThirtyThird Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021 , pages 3992-4000. AAAI Press, 2021.

Li, Z., L. Qu, G. Haffari. Total recall: a customized continual learning method for neural semantic parsers. In M. Moens, X. Huang, L. Specia, S. W. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 3816-3831. Association for Computational Linguistics, 2021.

Min, S., M. Lewis, L. Zettlemoyer, et al. Metaicl: Learning to learn in context. In M. Carpuat, M. de Marneffe, I. V. M. Ruíz, eds., Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022 , pages 2791-2809. Association for Computational Linguistics, 2022.

Chen, Y., R. Zhong, S. Zha, et al. Meta-learning via language model in-context tuning. In S. Muresan, P. Nakov, A. Villavicencio, eds., Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 719-730. Association for Computational Linguistics, 2022.

Hu, Y., C. Lee, T. Xie, et al. In-context learning for few-shot dialogue state tracking. In Y. Goldberg, Z. Kozareva, Y. Zhang, eds., Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 2627-2643. Association for Computational Linguistics, 2022.

Lester, B., R. Al-Rfou, N. Constant. The power of scale for parameter-efficient prompt tuning. In M. Moens, X. Huang, L. Specia, S. W. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 3045-3059. Association for Computational Linguistics, 2021.

Zhu, Q., B. Li, F. Mi, et al. Continual prompt tuning for dialog state tracking. In S. Muresan, P. Nakov, A. Villavicencio, eds., Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 1124-1137. Association for Computational Linguistics, 2022.

Wang, Z., Z. Zhang, C. Lee, et al. Learning to prompt for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 139-149. IEEE, 2022.

Raffel, C., N. Shazeer, A. Roberts, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020.

Scholak, T., N. Schucher, D. Bahdanau. PICARD: parsing incrementally for constrained autoregressive decoding from language models. In M. Moens, X. Huang, L. Specia, S. W. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 9895-9901. Association for Computational Linguistics, 2021.

Lester, B., R. Al-Rfou, N. Constant. The power of scale for parameter-efficient prompt tuning. In M. Moens, X. Huang, L. Specia, S. W. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 3045-3059. Association for Computational Linguistics, 2021.

Shazeer, N., M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In J. G. Dy, A. Krause, eds., Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , vol. 80 of Proceedings of Machine Learning Research , pages 4603-4611. PMLR, 2018.

Zheng, Y., H. Wang, B. Dong, et al. HIE-SQL: history information enhanced network for context-dependent text-to-sql semantic parsing. In S. Muresan, P. Nakov, A. Villavicencio, eds., Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 2997-3007. Association for Computational Linguistics, 2022.

Dai, D., Y. Sun, L. Dong, et al. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. CoRR , abs/2212.10559, 2022.

Liu, J., D. Shen, Y . Zhang, et al. What makes good in-context examples for gpt-3? In E. Agirre, M. Apidianaki, I. Vulic, eds., Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO@ACL 2022, Dublin, Ireland and Online, May 27, 2022 , pages 100-114. Association for Computational Linguistics, 2022.

Cer, D., M. Diab, E. Agirre, et al. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) , pages 1-14. Association for Computational Linguistics, Vancouver, Canada, 2017.

Fu, Y., H. Peng, L. Ou, et al. Specializing smaller language models towards multi-step reasoning. CoRR , abs/2301.12726, 2023.

Wang, H., W. Xiong, M. Yu, et al. Sentence embedding alignment for lifelong relation extraction. In J. Burstein, C. Doran, T. Solorio, eds., Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 796-806. Association for Computational Linguistics, 2019.

Kirkpatrick, J., R. Pascanu, N. C. Rabinowitz, et al. Overcoming catastrophic forgetting in neural networks. CoRR , abs/1612.00796, 2016.

Serrà, J., D. Suris, M. Miron, et al. Overcoming catastrophic forgetting with hard attention to the task. In J. G. Dy, A. Krause, eds., Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , vol. 80 of Proceedings of Machine Learning Research , pages 4555-4564. PMLR, 2018.

Han, X., Y. Dai, T. Gao, et al. Continual relation learning via episodic memory activation and reconsolidation. In D. Jurafsky, J. Chai, N. Schluter, J. R. Tetreault, eds., Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 6429-6440. Association for Computational Linguistics, 2020.

Mi, F., L. Chen, M. Zhao, et al. Continual learning for natural language generation in taskoriented dialog systems. In T. Cohn, Y. He, Y. Liu, eds., Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 , vol. EMNLP 2020 of Findings of ACL , pages 3461-3474. Association for Computational Linguistics, 2020.

Yu, T., C. Wu, X. V. Lin, et al. Grappa: Grammar-augmented pre-training for table semantic parsing. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Guo, J., Z. Zhan, Y. Gao, et al. Towards complex text-to-sql in cross-domain database with intermediate representation. In A. Korhonen, D. R. Traum, L. Màrquez, eds., Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pages 4524-4535. Association for Computational Linguistics, 2019.

Brown, T. B., B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin, eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual . 2020.

OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023.

Pasupat, P., P. Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers , pages 1470-1480. The Association for Computer Linguistics, 2015.

Chen, W., H. Zha, Z. Chen, et al. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In T. Cohn, Y. He, Y. Liu, eds., Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 , vol. EMNLP 2020 of Findings of ACL , pages 1026-1036. Association for Computational Linguistics, 2020.

Hwang, W., J. Yim, S. Park, et al. A comprehensive exploration on wikisql with table-aware word contextualization. CoRR , abs/1902.01069, 2019.

Xu, K., Y. Wang, Y. Wang, et al. Sead: End-to-end text-to-sql generation with schema-aware denoising. In M. Carpuat, M. de Marneffe, I. V. M. Ruíz, eds., Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022 , pages 1845-1853. Association for Computational Linguistics, 2022.

Wang, B., R. Shin, X. Liu, et al. RAT-SQL: relation-aware schema encoding and linking for text-to-sql parsers. In D. Jurafsky, J. Chai, N. Schluter, J. R. Tetreault, eds., Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 7567-7578. Association for Computational Linguistics, 2020.

Cao, R., L. Chen, Z. Chen, et al. LGESQL: line graph enhanced text-to-sql model with mixed local and non-local relations. In C. Zong, F. Xia, W. Li, R. Navigli, eds., Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , pages 2541-2555. Association for Computational Linguistics, 2021.

Yu, T., R. Zhang, M. Yasunaga, et al. Sparc: Cross-domain semantic parsing in context. In A. Korhonen, D. R. Traum, L. Màrquez, eds., Proceedings of the 57th Conference of the

Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pages 4511-4523. Association for Computational Linguistics, 2019.

Zhang, R., T. Yu, H. Er, et al. Editing-based SQL query generation for cross-domain contextdependent questions. In K. Inui, J. Jiang, V. Ng, X. Wan, eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pages 5337-5348. Association for Computational Linguistics, 2019.

Qin, C., S. R. Joty. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022.

Gao, Q., C. Zhao, Y. Sun, et al. A unified continual learning framework with general parameterefficient tuning. CoRR , abs/2303.10070, 2023.

A Hyperparameter Details

The detailed settings of the hyper-parameters used in our experiments are presented in Table 5.

B Tokenizer Alignment: GPT and T5

When calling OpenAI's API to utilize text-davinci-003 (hereinafter referred to as GPT), we can get a predicted SQL query p 0: L made up of L GPT tokens and a sequence of output distributions s 1: L where each s i ∈ s is the probabilities of the most probable tokens at each decoding step, corresponding to p i . Our objective is to convert the GPT output distributions to the the format of T5 tokenizer and get the target distributions t where each t i ∈ t contains the probabilities of T5 tokens, making it trainable for the student models. To achieve this goal, we apply the algorithm shown in Algorithm 1.

First, we evaluate whether GPT produces the correct SQL by comparing p with the gold SQL g . If not, we expand g as a sequence of one-hot distributions for student's training. Otherwise, we tokenize the GPT predicted SQL query using the T5 tokenizer and get a sequence of one-hot distributions q 1: K made up of K T5 tokens with probability 1. And our target is to align s and q . Here we apply an one-to-one strategy. If there exists an one-to-one mapping of a T5 token q i and a GPT token p j , we use the GPT distribution s j as the T5 distribution. In other cases where a mismatch exists, we simply take the one-hot distribution q i as the label.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author presents a novel approach that combines PEFT and ICT to train a continual table semantic parser. The proposed solution is based on a teacher-student framework. Two task streams are developed from WikiSQL and Spider semantic parsing benchmarks. Experiments using C3 and baselines are conducted using these task streams. The results indicate that their method outperforms existing competitors, achieving state-of-the-art performance across multiple metrics.\n\nSTRENGTHS:\n- The fuse of ICT and PEFT continual framework is novel and effective on the proposed continual table semantic parsing task.\n- The description of the framework and method is clearly written.\n\nWEAKNESSES:\n- The authors did not clearly state whether the Problem Formulation in Section 2.2 has been studied previously. \n- The Continual table semantic parsing problem is not well motivated. For example, the authors claimed `After training on a new task, the performance of the parser on the previous task may plummet attributed to parameter updates.`, but there is no empirical evidence of this claim as a motivating example. \n- The authors stated the \"invisibility of past demonstrations\" for ICT, which seems unrealistic even under data privacy concerns. ICT typically only requires a few examples, which can be obtained from public databases or synthetic/obfuscated database content.\n- The authors can have additional baseline: Train on a reasonable size of public text2sql data with a mixture of domains and then zero-shot on the Stream datasets. The public data will not have privacy concerns. This will show the accuracy gap and rationale behind the continual training. It seems line 196 is similar to this, what is the training set size?\n\nQUESTIONS:\n- Is the Problem Formulation in Section 2.2 novel? Have other papers examined and studied this problem? If not, the problem formulation and its motivation should be part of the paper's contribution. \n- In Table 1, what is the PEFT results on T5-large? \n- In Figure 1, what are the blue arrows going into PLM in the ICT portion?  \n- In Figure 1, why are past demonstrations invisible? This seems to be an unrealistic setting, even under data privacy concerns. We can certainly have a diverse pool of demonstrations from public databases or synthetic/obfuscated database content. \n- In Figure 3, what is the x-axis label?\n\nLIMITATIONS:\nNo negative societal impacts", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

comments: SUMMARY:
The paper introduces a method combining parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) to address the issues of overfitting and catastrophic forgetting in training a continual table semantic parser with limited training examples. Through a task-adaptive PEFT framework and a teacher-student setup that utilizes ICT, the method demonstrates enhanced performance in comparison to established baselines, as validated by experiments on two benchmarks.

STRENGTHS:
1. This paper is overall good written and easy to follow.
2. This paper proposes a method that fuses PEFT with ICT to resolve the overfitting and catastrophic forgetting problem. 
3. In addition to PEFT + ICT, the authors also propose a teacher-student framework that distills ICT results on teacher to a student model.

WEAKNESSES:
My main concern about this paper is its technical novelty.
1. Combining PEFT and ICT (demonstration) is not new. e.g., [Gao et al.](https://arxiv.org/abs/2012.15723)
2. Distilling soft prompts, on the other hand, is not new either. https://arxiv.org/abs/2212.10670 https://arxiv.org/abs/2304.08467
3. I found it unconvincing to exclude results from [7]. C3 also uses demonstration retrieval, which indicates the same pool of supervised data is used. 
4. Following 3, the claim that the method is few-shot is inaccurate. Since the examplars are selected from a pool examples, comparing to methods that only use a few examples is unfair.

QUESTIONS:
### Questions
1. Why the authors choose tabular semantic parsing? Is this method generalizable to other tasks (e.g., general NLU/NLG tasks)?
2. Why use different backbones for other methods in Table 1? How do those methods perform on T5?

### Typos
Ln. 23: "finance [3], and ?"

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method for training a continual table semantic parser, which aims to translate natural language into SQL based on task-specific tables with limited training examples. The proposed method integrates parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) to overcome the challenges of overfitting and catastrophic forgetting. The paper presents a task-adaptive PEFT framework and a teacher-student framework-based solution. Experimental evaluations demonstrate the superiority of the proposed method over existing baselines.\n\nSTRENGTHS:\n- The paper addresses an important problem in the field of table semantic parsing, namely the challenge of training a parser on a sequence of tasks with limited supervision.\n- The proposed method integrates two existing techniques, PEFT and ICT, to overcome overfitting and catastrophic forgetting, respectively. This combination of methodologies is neat and straightforward.\n- This paper provides a well-structured review of relevant literature, discussing prior work on table semantic parsing and related topics.\n- The analysis is comprehensive and gives more insight on the future research.\n\nWEAKNESSES:\n- The baseline comparison appears somewhat unfair. While I appreciate the inclusion of multiple baselines by this paper, I noticed that most of them were implemented on GRAPPA to highlight their inferior performance, whereas this paper's proposed method was applied to T5 (base and large). Additionally, there is a lack of reported MD performance for both PEFT and C3, which should be the key metric to reflect the forgetting degree. This inconsistency in reporting the experimental setup leaves me somewhat confused about the evaluation process and the methodology employed in this paper.\n- The results in Table 1 show that PEFT is dominant for catastrophic forgetting, while C3 is less dominant. in-context tuning does not play a prominent role here. And PEFT is already well-known for its advantage to avoid catastrophic forgetting, with the limitation that it cannot learn very different tasks fast.\n\nQUESTIONS:\n- I notice \"Demo 2\" appears twice in Figure 1. Is it a typo?\n- I noticed that the paper utilizes the first dataset to initialize the model parameters for the entire model. This approach raises a natural concern: does it imply that the proposed method is only applicable to scenarios where the logical form is similar, but not when the logical form differs completely?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper proposes a new continual learning method C3 for table semantic parsing which combines parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT). The C3 framework contains a teacher network (ICT) that extracts contextual information from demonstrations and a student network (PEFT) that learns the teacher's output distribution. The ICT aims to enhance the ability of few-shot learning and PEFT aims to reduce catastrophic forgetting. The author construct two stream datasets from WikiSQL and Spider to evaluate their proposed method. C3 outperforms few-shot learning and continual learning baselines on these stream datasets.

STRENGTHS:
1. The paper combines parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) with a teacher-student framework to leverage the unique advantages of each approach.  The proposed methods are effective in the continual table semantic parsing task. 
2. The proposed framework is model-agnostic and seems generalizable to other continual learning tasks, further enhancing its value and applicability.

WEAKNESSES:
I don't see significant weakness but please address my concerns in Questions.

QUESTIONS:
1. In Table 3, when employing a T5-base model as the student model, it is observed that T5-large outperforms GPT-3 as the teacher model. However, interestingly, when the student model is upgraded to T5-large, the choice of teacher model seems less important. I am curious about the performance implications if the teacher model is completely removed when utilizing T5-large as the student model. I don't see this ablation study in Table 2.

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07