Parameterizing Context: Unleashing the Power of Parameter-Efficient Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing
Authors: Yongrui Chen
Shenyu Zhang
Guilin Qi
Xinnan Guo
Venue: NeurIPS 2023 poster (2023)
Keywords: semantic parsing
continual learning
few-shot learning
TL;DR: Our method integrates parameter-efficient fine-tuning and in-context tuning in a teacher-student framework, thereby advancing continual table semantic parsing.

Abstract: Continual table semantic parsing aims to train a parser on a sequence of tasks, where each task requires the parser to translate natural language into SQL based on task-specific tables but only offers limited training examples. 
Conventional methods tend to suffer from overfitting with limited supervision, as well as catastrophic forgetting due to parameter updates.
Despite recent advancements that partially alleviate these issues through semi-supervised data augmentation and retention of a few past examples, the performance is still limited by the volume of unsupervised data and stored examples.
To overcome these challenges, this paper introduces a novel method integrating parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) for training a continual table semantic parser. Initially, we present a task-adaptive PEFT framework capable of fully circumventing catastrophic forgetting, which is achieved by freezing the pre-trained model backbone and fine-tuning small-scale prompts. 
Building on this, we propose a teacher-student framework-based solution. The teacher addresses the few-shot problem using ICT, which procures contextual information by demonstrating a few training examples. In turn, the student leverages the proposed PEFT framework to learn from the teacher's output distribution, and subsequently compresses and saves the contextual information to the prompts, eliminating the need to store any training examples.
Experimental evaluations on two benchmarks affirm the superiority of our method over prevalent few-shot and continual learning baselines across various metrics.

Parameterizing Context: Unleashing the Power of Parameter-Efficint Fine-Tuning and In-Context Tuning for Continual Table Semantic Parsing
Yongrui C2 Shenyu Zhang Guilin Qi!21 Xinnan Guo 1
1 "School of Computer Science and Engineering, Southeast University, Nanjing 211189, China Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China [yrchen, shenyuzhang, @qiu edu. en, guoximan07720166..com
Abstract
Continual table semantic parsing aims to train parser sequence of tasks, where each task requires the parser to translate natural language into SQL based on task- specific tables but only offers limited training examples. Conventional methods tend to suffer from overfitting with limited supervision, as well as catastrophic forgetting due to parameter updates. Despite recent advancements that partially alleviate these issues through semi-supervised data augmentation and retention of a few past examples, the performance is still limited by the volume of unsupervised data and stored examples. To overcome these challenges, this paper introduces a novel method integrating parameter-efficien fine-tuning (PEFT) and in-context tuning (ICT) for training a continual table semantic parser. Initially, we present task-adaptive PEFT framework capable of fully circumventing catastrophic forgetting, which is achieved by freezing the pre- pre-trained model backbone and fine-tuning small-scale prompts. Building on this, we propose teacher-studee framework based solution. The teacher addresses the few-shot problem using ICT, which procures contextual information by demonstrating few training examples. In turn, the student leverages the proposed PEFT framework to learn from the teacher's output distribution, then compresses and saves the contextual information to the prompts subsequently, eliminating the need to store any training examples. Experimental evaluations on two benchmarks affirm the superiority of our method over prevalent feo-shot and continual learning baselines across various metrics.
Introduction
Tabular data is a crucial source of information for human decision makers in various domains, including healthcare 1. retail (2] and finance 3 Although structured query language (SQL) provides an efficient means of accessing this data, natural language interface would make it more accessible to non-technical users. Consequently, there has been growing interest in Table Semantic Parsing (TSP), which involves mapping natural language queries over tabular data to formal programs.
TSP have covered many scenarios, most of which assume the datasets static. In contrast, recent work 7 highlights the fact that new tables are often constantly emerging in response changing cireunstances, leading
Current research 5 6
continual stream of TSP tasks. With this comes two new challenges: a) shot problem [8] 9 10]: For new task against unseen tables, the annotated data available in a short period is typically limited, causing the parser to be prone over-fitting. b)
Equua contributors.
I Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

PEFT
PLM
PLM
PLM
Overfitting
Prompt
Input
Promot
Input
Task
Task 2
Task
Flights
Sew-Shot
Geagraphy
Few. Shot
Movies
Few-Shot
Demo 1
Input
Demo 7
Input
Demo
Catastrophic Forgetting
ICT
PLM
PLM
PLM
Figure 1: Processes for handling TSP task stream using PEFT (upper) and ICT (bottom). and represent tuning and freezing parameters, respectively. X means that the previous data is not visible.
After training on a new task, the performance of the parser on the previous task may plummet attributed parameter updates. While 77) mitigates these challenges using semi-supervised learning and replaying past examples, however, its performance is still limited by the amount of unsupervised data and replayed examples. More realistically, in some scenarios involving data privacy and timeliness, such as patient cases and transaction records, both unsupervised data and past examples may not be available the following tasks.
catastrophic forgetting 7
Fortunately, recent research yields new ideas to tackle the issues. a) in-context tuning (ICT)
14] enhances models' few-shot learning capability using demonstrations containing few training ex- amples. b) parameter effective fine-funing (PEFT) [15 16, 17] fundamentally eliminates catastrophic forgetting by avoiding parameter updates of the pre-trained model (PLM) during the training stage and instead tuning a group of parameters of bearable size individually for each task. Nevertheless, the two solutions can merely handle the corresponding challenge while still suffering from the other. As is illustrated in Figure T for ICT, the invisibility of past demonstrations causes catastrophic forgetting when the model experiences a new task; for PEFT, though proposed improve generalization, it still suffers from severe overfitting as only few training examples are available for each new task.
In this paper, propose new method to simultancously tackle both of these challenges by integrat- ing PEFT with ICT. We start with a adsk-adee parameter-efficient continual TSP framework, which freezes the backbone PLM and only tunes the additional prompt embeddings for each distinct task. Unlike existing methods [6] 17, our backbone PLM undergoes fine-tuning on the initial task to facil- Compresse
itate adaptation to the TSP task format. Based on this foundation, we propose Continual table semantic parser (C3). C3 employs a teacher-student framework, comprising two parsers that symbolize the TEACHER and the STUDENT, respectively. For each task, the TEACHER utilizes ICT to glean contextual information from semantically akin demonstrations extracted from the training set, thereby mitigating the risk of overfitting on the few-shot data. Subsequently, the STUDENT adopts our proposed parameter-effice and learns the output distribution of the TEACHER for each training example. This method ensures that the contextual information gleaned by the TEACHER can be preserved and compacted into parameterized form, which can be reloaded later even when the demonstrations of past tasks are no longer available. Extensive experiments performed on two TSP task streams substantiate the effectiveness of our method, which surpasses all compared baselines, achieving state-of-the-ar results across multiple metrics. In summary, the contributions of this paper include:
framework to fine-tune the prompt embeddings
We introduce task-adaptive, parameter-efffiient continual framework for table semantic parsing. can completely avert catastrophic forgetting by tuning only 0.05% of the original PLM parameters.
We propose to utilize the teacher-student framework fuse PEFT and ICT that the table semantic parser can benefit from both their advantages in solving few-shot data and catastrophic forgetting. To the best of our knowledge, this is the first time that the two technologies have been integrated into the scenario of the task streams.
We create two task streams derived from two mainstream table semantic parsing benchmarks, and carry out extensive experiments using these task streams. The results demonstrate that our method surpasses existing competitors, achieving state-of-the-art performance across multiple metries.

---

2 Preliminaries
2.1 Table Semantic Parsing
Formally, given a natural language question Q and schema S = (C, T) of multiple tables, the goal TSP generate a SQL query y Fo 2, S), where Fg denotes neural semantic parser with parameters 0. T (41,....4 .(71) and C = (eI.......cc) are the sets of table names and column names, respectively, where each column 6₁ C belongs to only one table 1 E T.
In this paper, we focus on TSP in this paper for the following reasons: a) TSP important for enhancing various fields through its role Natural Language Interfaces to Databases (NLIDB). b) TSP is a challenging task requiring comprehensive understanding and mapping of queries to structured tables, presenting few-shot learning challenge due to the difficulty in obtaining supervised data. c) TSP provides objective evaluation metrics, as minor errors in SQL queries can lead to significant outcome discrepancies, thereby offering true reflection of our method's 's performance.
2.2 Problem Formulation
In this paper, assume that Fo is no longer limited fixed training and testing data, but has to face a continual stream of TSP tasks, each corresponding to a different set of tables. Formally, Fo is
sequentially trained stream of K tasks (D", D ..... DK- Each task D' consists of training set Drainn a validation set Di and a test set Dissi Each example
y), where X' = (2, S),
the input and y denotes the target SQL. For VD', VDi, if ( j. then ( ) n S = 0. where S(D)) denotes the total set of corresponding table schemas in D'. Here S also serves
task identifier for each e, enabling efficient indexing of the corresponding task. Our ultimate objective is train Fo that can achieve good accuracy on each Disst after experiencing all K tasks sequentially.
3 Parameter-Effiieen Continual Table Semantic Parsing
We start with a vanilla parameter-efficient continual TSP framework, which trains a backbone semantic parser with task-adaptive continual prompt-tuning method.
3.1 Backbone Semantic Parser
The pre-trained Text-to-Tex Text Transfer Transformer (T5) [18 is selected as our backbone parser Fa for two primary reasons. Firstly, prior studies [66 have shown that the combination of T5 with PEFT methods can result in significant performance gains across various NI.P tasks. Secondly, T5 has demonstrated robust capabilities [19] in generating logical forms on a widely-used TSP dataset. Spider [5) In particular, each input pair X' = (Q, is flattened into plain text,
X' = t1 of'
ch, t2 cf2
QQ. 1..
where o denotes the j-th column name of the i-th table, and "!" "."."." and "I" are predefined separators. Subsequently, X" is fed to the parser Fo and the generative probability of predicted SQL 5 is D
estimated by P(Ji/2", 0) = MT P(ū X', 0) where yj V denotes the j-th token of ) and V is the vocabulary of the T5 tokenizer. X' are the hidden states of X' obtained by the T5 encoder, and Pūū X', Jez, 0) denotes the output probability of ūo from the T5 decoder.
3.2 Task Adaptive Continual Prompt Tuning
as representative of PEFT-based methods, has been widely applied in recent owing to its simplicity. Inspired by these works, we employ prompt tuning to conduct the PEFT training framework for Fa. Formally, each input X' in task D prefixed with M special prompt tokens Pk = =fir ... ... During training, only the embeddings of these tokens are updated, rather than the parameters of the entire T5 model. More specifically, the training process can be divided into the following two phases.
Prompt tuning
NLP tasks [6 17
3

---

Task (-
Task
Task
Demo
Input
Input
Demo
Input
Teacher
Teacher
Teacher
Distribution
Distribution
Distribution
Student
Dxr.
Student
DRt.
Student
Dxr.
Promet
Input
Promet
Input
Promet ++
Input
Figure 2: Illustration of the proposed C3 table semantic parser.
3.2.1 Initial Task Adaptation
Intuitively, the original T5 is not well-suited to TSP, it is not exposed any logical form during pre-training and is never tasked with predicting structured outputs. While this limitation can be readily addressed through fine-tuning, hypothesize that tuning prompts alone may not be sufficient.
Therefore, we first fine-tune T5 with Daiin to obtain a set of aask-adaptive parameters that will be frozen on DD. Di.... DK-1, Notably, here the prompt embeddings are tuned as well as 0 because we expect to acquire good initialization for both of them. In particular, for each e = (X, v) E Do train
the following loss optimized using the AdaFactor [1 algorithm,
|
O(e:0,P') 0 log
_
log P(U,) X')xy,,<jj
where [:] represents the concatenation of matrices, P RM xd denotes the tunable embeddings of pp_ and p Mxd represents its hidden states from the T5 encoder. In our experiments, p randomly initialized. After multiple iterations, the pair of 9 and p with the highest validation accuracy, denoted by (0", P"), is selected for initializing the subsequent K - 1 tasks.
3.2.2 Continual Prompt Tuning
Unlike existing methods [22], for D' <0 < K), we initialize all P with p instead pi-1 because our experimental results (detail in Section 5.3.2 reveal that the latter does not yield significant performance improvements. Furthermore, this strategy enhances the flexibility of the proposed framework by eliminating the dependency on the order of subsequent tasks, all prompts share common initialization. During training, P is updated with the gradient Vp: ΣeEDi C(e; P')
using the AdaFactor optimizer, where L(e; 6 PP) calculated by Equ. (2). We experimentally set the prompt dimension d = 512 and the prompt number M = 150. leading to total of only 0.7 million parameters. This size is negligible (0.05%) compared million parameters. Consequently, the cost of storing all P1 ..... P is usually acceptable. During inference, given a test example ejest 0 its task index i can be first identified using its table schema S. Then, can solve by loading the saved (P*, 8") without any forgetting.
the entire T5-BASE of 220
In theory, our framework can also be expanded to the tasks whose tables have not been seen in DD -K The prediction be made using simple heuristics, such averaging all P Since this scenario involves the zero-shot learning problem, we leave it for future work.
4 C3 Semantic Parser
Figure 2 depicts the illustration of our proposed Context- Hentext-Compressel Continual table semantic parser (C3). C3 is typically composed of two parsers (e.g., T5) of identical architecture (optional), namely TEACHER Fre and STUDENT Fst.. and each possesses its own individual parameters during training.
4.1
Context-Enhanced Teacher Parser
The sole purpose of Frea is to gain the few-shot learning capability and thereby in-context tuning (ICT) employed train Fun. Note that Fiz only focuses on each current task D and does not care if the past tasks are forgotten. Thus, it is sequentially trained on D ...., DK- without any continual

---

learning. During D the parameters of Fua. denoted by 0i is initialized with Bica! Afterwards, o is updated by optimizing L(z; Oees) - -, log Pūū "CC<, Own). Here C denotes the hidden states of the input text, C, which is constructed by retrieving and formatting the demonstrations.
4.1.1 Demonstration Retrieval
The key to ICT lies in identifying the suitable demonstration that effectively facilitates the learning process of the model within the appropriate context. Recent studies [23 24 have reached consensus that the closer the demonstration aligns with the example being queried, the greater the benefits of ICT. Thus, for each e, retrieve the demonstration by comparing its semantic similarity to the candidate demonstrations. To maintain overall uniformity, we select the original T5 checkpoint which was fine-tuned the Semantic Textual Similarity Benchmark (STS-B) retriever R. Specifically, for example e to be queried and a candidate demonstration e' E 'Diiiin \ {e), we first wrap them into a template T(e,c), = stsb sentence1: 2, sentence2: D where Q and 2j represent the natural language questions of and e'j, respectively. Here the target SQL y/ is not considered because it is invisible in the testing stage. Subsequently. the similarity score of ej- denoted by o (e is calculated by a(e))) = R(T(e, ef)) E [0,5]. To circumvent the noise, we set threshold 17 and eventually select the top-r e'j E Dimin \ {e} whose a(e)) 2 1) the demonstration of e. In fact, not every example E Di has sufficient demonstrations. To retain Fi 's ability in handling no-demonstration examples, we apply simple but effective context mixing strategy during training: both and X' are engaged in the training for each Dirain it has r demonstrations, otherwise only X' is engaged.
125
aff-the-shelf
4.1.2 Demonstration Formatting
r retrieved demonstrations C1,...,e, compose the input C= 2 -- where both
and || are separators, and X' is obtained by Equ. [ The decision exclude table schemas S from is motivated by one main consideration: the selected demonstration examples and the example to be queried are likely be associated with the same batch of tables due to their similarity. Therefore, the explicit information regarding schemas need not be provided repeatedly outside of X', which helps streamline the input to improve efficiency.
4.2 Context-Com Compext-Compressed Student Parser
Fi. follows the PEFT framework introduced in Section and is dedicated compressing and preserving the few-shot capability Fiz learns from the demonstration into the prompt embeddings. In this way, Fi reproduce the few-shot performance by loading the saved prompt embedding, without accessing any examples or demonstrations of previous tasks. Inspired by (26), We hypothesize that this goal could be accomplished as much possible by having F learn the F 's output distribution. More concretely, we minimize the KI. divergence between F per-step autoregressive distribution and the Fis ss denoted by DxL (P(rcs)) During training, the teacher-forcing strategy is applied to ensure that the inputs to the decoders of Fi and Fii consistent each decoding step. In addition, to guarantee the quality of supervision, we directly use the gold SQL y instead of P(Owa) if Fi 's prediction for a given input X' is incorrect.
P(Osu)) where P(0) P(y C, yır 0).
5
Experiments
5.1 Experimental Setup
5.1.1 Datasets & Evaluation Metrics
Our experiments involve two widely-used TSP datasets: WikiSQL [4] a dataset for single-table semantic parsing in which each SQL query corresponds to single table and adheres to simplified SQL syntax. Spider [5] is a more challenged dataset in which each SQL requires the JOIN operation of multiple tables and contains complicated syntax including GROUP BY even nested query.
As explicated in Section 2.2 we constructed two task streams based on the above datasets. The first one, Spider-Stream, is established by merging the Spider training and validation sets and splitting
5

---

Example Number
Example Number
600
400
le D No Diatid Bu Digs
inin
400
200
200
2
6
8
10
12
14
16
2
3
6
9
10
Figure 3: Statistics of the task splits of Spider-Stream (left) and Combined- (right).
them into 16 tasks according to the domain of the tables. It is used to reflect the impact of the changing table domains. The second one, Combined-Stream, is merged task stream encompassing 11 tasks. The first tasks are randomly selected from the Spider-Stream and encompass complex SQL syntax, whereas the last 3 tasks collected from WikiSQL and involve only simple SQL syntax. It utilized to assess the dual impact of table domains and target SQL structures.
To highlight the few-shot challenge, the majority of tasks D Dirain have [Di < 500. For each method, we evaluated its performance in three random task orders, to eliminate any potential influence of specific tasks. Considering that in real applications, models are usually fully trained in certain domains before facing the task stream, we combined the first N (6 for Spider- Strram- and for Combined-Streeed -tream))) tasks of cach task order into single task. This practice enables all methods to have full adaptation to the TSP task format. In addition, for Combind-Stteane to aggravate the gap between adjacent tasks, for any task order, we always make the tasks of the even index (0, ...) from Spider and the tasks of the odd index (1, 3.... from WikiSQL.
Following 27
we adopted four metrics: 1) Task Accuracy: TA = K E ai 2) K-1
Example Accuracy: EA " ( 3) Initial Accuracy: IA
K Di=D ai,; 4) Memory Decay: Σ
MD Rー Ei= 04 - arii where dt,j denotes the accuracy on Diest after the training of Di TA and EA reflect the overall performance at the task level and example level, respectively. MD measures the extent of forgetting. IA evaluates the performance when the parser is trained on new task for the first time, considering no forgetting.
5.1.2 Compared Methods & Implementation Details
Our compared methods be categorized into three groups: a) FINE- TUNING, which trains the parser sequentially without any strategies; b) Few-Shot Learning baselines, which contain MAML [
and ICT [3] c) Continual Learning baselines, which consist of EWC [28). HAT 29), EMR (2].
EMAR 30). APPER (31], and TR [T]]. To increase the competitiveness of these baselines, we
utilized pre- trainra GRAPPA-LARGE (32] for TSP conjunction with SemQL [33), an intermediate
representation for SQL. To ensure a fairer comparison, we conducted experiments applying the best-performing methods on GRAPPA, EMR and EMAR, to T5-BASE as complementary baselines. Additionally, we compared our methods with SFNet 77 which utilizes extra unsupervised data. Note that we only apply ? on Combined-Strean for the reason that Combined -tream randomly selects portion of samples from the original Spider and WikiSQL datasets which enables the rest to be used as unsupervised data to fulfill the scenario requirements. We also set an upper boundary method
MULTI-TANE For each task Dí, it trains Fo with D (0:1)
Our method ran validation sets (See Appendix for details): a) prompt length M is set to 150; b) demonstration number and threshold v are set to and 4.0, respectively; c) the batch size is 12 and the learning rates set to 0.3 and 1 x 10-1 for prompt-tuning and fine-tuning, respectively; d) early stopping is performed after 1000 epochs and the patience is set to 10 evaluations, which is performed every 50 epochs. e)
NVIDIA RTX 4090 GPU and the hyperparameters were decided with the
T5-LARGE is always employed as Fiea All of our data and codes are publicly available"
5.2 Overall Results
Table presents the overall performance of the methods. The results indicate that most of the compared methods exhibit relatively lower effectiveness and more severe forgetting on Combined- Stream compared to Spider-Stream, even though their backbone models have been pre-trained for the

---

Table 1: Experimental results for comparison with baselines in 3 random task orders. Means and standard variations are reported. The absence of standard deviation for PEFT and C3 due to the fact that their performance is order-independent. indicates using the replayed memory of size 15 and + indicates using additional unsupervised data.
Spider-Stream
Combined-Stream
Backbone
Method
TA (%)
EA (%)
MD (%)
TA (%)
EA (%)
MD (%)
FINE-TUNISG TUNING
56.91.0
54.61.0
11.
37.61.8
43.9b.9
22..12.2
MAML g
52.21.3
49.11.5
-
31.31.3
37.21.4
- -43.8..
ICT R3
57.01.4
54.322
-17.121
37.94.0
43.91.
-37.44.4
GRAPPA
-L.ARGE
EWC
57.5g.3
55.12.4
-17.73.9
37.01.9
44.1o.9
-38.42.2
(340M)
29
57.82.9
54.83.4
-17.03.3
38.54.1
45.02a
-37.05.5
EMR 27
65.2p.2
62.90.c
-9.4..
60.9u.7
58.61.8
-10.31.8
EMAR
30
62.81.2
60.81.2
-10.5..1
63.118
60.80.9
-7.726
APPER
31
57.91.6
55.81.7
-17 -17.22.6
37.124
44.00.6
- 33.32.9
TR
1
57.91.2
55.11.4
-1..8..
59.71.0
56.31.1
-11.90.7
SENET
60.70.9
57.01.9
6.01.2
EMR
P7
60.3p.9
56.6n.1
13.40.2
62.6o.4
60.0,.4
6.50.6
EMAR
30
57.20.4
52.71.0
16.7u.s
62.10.2
58.11.1
6.5u.3
T5-DASE
PEFT
65.7
64.5
0.0
63.8
66.2
0.0
(220M)
C3
67.5p.3
66.5p.
0.0p.o
66.30.2
67.6g.2
0.0p.0
MULTI-TASE
76.3p.s
76.21.o
3.2p.7
70.00.9
71.10.7
1.70.2
T5-LARGE
PEFT
69.8
67.4
0.0
67.3
70.0
0.0
(770M)
C3
71.
6o.7
0.0g.0
68.3p.5
70.60.4
0.0g.0
FINE- TUNING MAMI
70
MD (3)
-10
EWC
60
10
EMARccom:
50
EMAR aM
(%
(9))))
60
MDD (3)
PEPT
10
Ch
Chaaaa
20
MULTI-TASK
2
3
5
2
3
Figure 4: TA (%), EA (%), and MD (%) till the seen tasks of Spider- Streer--Staammmmm (upper) and Combined- Stream (bottom) after learning on each task. Only the means are reported in random task orders.
tabular data. Surprisingly, our proposed PEFT framework outperforms all baseline models despite the absence of any explicit provision to address the few-shot problem, mainly due to its ability to retain previously learned knowledge. Moreover, our proposed C3 further improves the overall performance (4.9% 5. 5.1% in terms of TA) and achieves state-of-thed the--rr results for both datasets. Of particular note is that even with a smaller backbone model (T5-BASE 220M), C3 outperforms all compared methods based on GRAPPA-LARCE (340M). Remarkably, it manages to approach the upper bound of on Commbined-Straamm
performance (MULTI-TASE
Compared to ICT, MAML performs poorly on both datasets, especially in terms of MD, even less than FINE-TUNING. This finding may point out that it is more profound and irreversible for parameter updates. The replay-based EMR and EMAR outperform other continual learning baselines in terms of all metrics, which suggests that retaining past instances may be more direct and effective for mitigating forgetting, particularly on Combined-Streed in many cases involving data privacy, access to past examples is disabled, rendering these methods limited. Conversely, our proposed C3 not only cireumvents data privacy concerns and has a broader range of applications, but also achieves superior results without using any past examples.
where the task varies more. Unfortunately,
7

---

Table 2: Experimental results of ablation studies in the same task order. used is T5-LARGE, since only TEACHER remains when STUDENT is removed.
indicates that the backbone
Spider-Stream
Combined-Stream
Backhone
Method
TA (%)
EA (%)
MD (%)
TA (%)
EA (%)
MD (%)
FINE TUNING
53.3
50.8
20.7
59.1
58.2
-9.1
C3
67.7
66.7
0.0
66.4
67.7
0.0
T5-BASE
w/o TEACHER
65.7
64.5
0.0
63.8
66.2
0.0
w/o STUDENT
64.9
62.8
11.66
59.6
59.6
14.9
w/o In- In-Context Tuning
65.6
64.3
0.0
65.3
66.9
0.0
w/o Task Adaptation
19.4
20.6
0.0
35.8
25.7
0.0
Continual Initialization
67.0
65.4
0.0
66.0
67.4
0.0
FINE-TUNING
60.3
56.3
-17.3
65.3
67.2
T5-LARGE
C3
70.7
68.9
0.0
69.0
71.2
0.0
Table 3: Performance of C3 using GPT as the TEACHER parser.
Spider-Stream
Conbined-Stream
STUDENT
TEACHER
TA (%)
EA (%)
TA (%)
EA (%)
1ext-davinci-003
66.3
64.8
65.5
66.8
TS-BASE
T5-LARGE
67.50.
66.5o.2
66.3o.2
67.60.2
text-davinci-003
71.3
69.6
67.6
70.0
T5-LARGE
T5-LARGE
71.10.3
69.7p.5
68.3g.5
70.60.4
5.3 Detailed Results and Analysis
5.3.1 Performance Till the Seen Tasks
Figure 4 displays the methods' performance till the seen tasks on both datasets. Our proposed C3- LARGE (blue) consistently exhibits the highest performance across all evaluated time steps. Moreover, this performance advantage becomes increasingly pronounced as the number of tasks in the continual learning regime grows. Interestingly, the EA metrics on the Combined-Streant for compared methods always show oscillation. The performance troughs on the even-indexed tasks suggest a dramatic forgetting of the harder Spider tasks after the model is trained on an easier WikiSQL task.
5.3.2 Ablation Study
We compared the performance of the proposed C3 with the following settings:
w/o TEACHER: removed Fic to verify its contribution;
w/o STUDENT:
we removed Fsu and only employed Fiee (T5-LARGE) to perform ICT, to evaluate the contribution of
w/o In-Context Tuning: used FINE-TUNING train Fua to assess the necessity of ICT.
w/o Task Adaptation: we used PROMPT- TUNING to train Fsu from D
W Continual Initialization: we initialize the prompt embeddings P with pi-1 instead of P
Table 2 shows the TA (%). EA (%), and MD (%) of different settings. Our C3 equipped with all components performs best in terms of all metrics. Dramatic performance degradation demonstrates that the adaptation of the PLM the task format critical to the effectiveness of prompt tuning. Removing Fiea results in an approximate 2% drop in terms of both TA and EA, which proves the TEAcHer's few-shot learning capability contributes the entire model. With the exclusion of STUDENT, the TA exhibits more pronounced decrease on Combined-Streant (-6.8% compared to Spider-Stream (-2.8% because Combined-Streee Strream involves challenges both domain changes and SQL structure changes. The performance drops (-2.3% & -0.7%) brought by abandoning ICT reveal the necessity contextual information. Its smaller contribution on Combined-Streee Streamm is probably due to the relative simplicity of the WikiSQL tasks, which does not require demonstration make

---

78
00
20
75
Figure 5: Left two: performance of C3 with different prompt lengths M and PLM sizes; Right two: IA (%) of the TEACHER parser with different thresholds 1) and demonstration numbers r, where indicates the absence of the context mixing strategy (detailed in Section 4.1.1).
correct prediction. In contrast to [16] 's assertion, our experiments did not show any improvement with Continual Initialization. One possible reason the large gap between different tasks.
5.3.3 Impact of Large Language Models
Considering the amazing potential of Large Language Models (LLMs), such as GPT-3
34 and
GPT-4 [35, in the NLP community recently, we also evaluate the performance of C3 by replacing Fuz from T5-LARGE with the text-davinci-003,, an LLM of the GPT family that excels in textual tasks. Here we did not choose code-davinci-002 which specifically handles code tasks, and the more widely used gpt-3,5-turbo, because the API of the former is deprecated and the latter can not provide output distribution for each decoding step. Referring to [26]. we align the GPT tokenizer with the T5 tokenizer in order to calculate DXL (details in Appendix). Here we let text-davinci-O01 perform inference based on the extracted demonstration without tuning its parameters, C3 in this scenario is independent of the task order. The experimental results are shown in Table Despite not having undergone any fine-tuning, using GPT has achieved close even better results compared to using T5-LARGE. This proves, to some extent, that our proposed method is not limited to a specific architecture (encoder-decoder decoder-only) and size (million-level vs. billion-level) of PL.M
5.3.4 Impact of Different Prompt Lengths & PLM Sizes
We varied the backbone PLM in (T5-SMALL, (ASBB and prompt length M in (1, 20, 75. 150) for our proposed C3 on the two task streams. The experimental results are shown in Figure 5] which indicates that: When fixing the prompt length, increasing the backbone PLM size improves the TA; When the PLM size is fixed, increasing the prompt length improves the overall performance in general. Moreover, our findings indicate that extending the prompt length from 20 to 75 enhances TA more significantly than increasing it from 75 to 150 on both task streams.
5.3.5 Impact of Different Demonstration Numbers & Thresholds
We also varied the demonstration number r in (1, 2, 3) and threshold in (2.0, 3.0, 4.0) for the TEACHER Fre our proposed C3. 5 shows IA (%). We observed that removing the context mixing strategy leads to severe performance degradation, proving our hypothesis that there exists many examples lack of sufficient demonstrations. When using context mixing, the performance is not significantly affected by the threshold and number of demonstrations, but trade-ofi off between performance and efficiency achieved with a threshold of 4.0 and using only one demonstration.
5.3.6 Performance of TEACHER Parser with Always- Visible Demonstrations
While in many works involving in-context learning there exists an always-visible demonstration pool (e.g., public or synthetic datasets) from which we can select demonstration for all tasks, resources for TSP tasks are quite limited due high expense of annotation. Therefore, C3 applies a practical strategy which samples demonstrations from every incoming task. However, to study the above sampling strategy, we conducted an additional experiment assuming that D D the training set for task O, is an always-visible public demonstration pool from which all demonstrations for subsequent tasks are sampled. Table shows IA (%) of a T5-LARGE based teacher parser. From the results, when sampling the demonstration from D the performance of the teacher parser shows significant drop on both datasets. This reflects conclusions in 23 24 which state that performance gains maximized only when in-context learning utilizes demonstrations similar to the test examples.

---

Table 4: IA (%) of TEACHER Parser with Always-Visible Demonstrations.
TEACHER
Spider-Stream
Combined-Strean
Demonstrations from D
76.3
70.6
C3
78.2
72.4
For domain-specific databases, there is no guarantee that similar demonstrations will be found using public data resources. Our results prove that that the quality of the demonstration has great impact on the performance of ICT.
6 Related Work
Table Semantic Parsing In recent years, there has been a notable increase in interest concerning joint textual-tabular data understanding problems [6 37,, especially in table semantic parsing (TSP)
This trend can be attributed to the fact that many real-world datasets contain combination of structured and unstructured data, and comprehensive understanding of such data requires the ability to analyze and interpret both forms effectively. The related methods can be divided into three directions according to the application scenarios, namely single table
38 [0 39], multi-
table [5] 33
and conversational tasks 42 43 (2)]
Parameter Efficient Continual Learning Prompt tuning was first systematically studied by [20]. who demonstrated that fine-tuning language models on small number of task task-specific input prompts could result in substantial performance improvements with much lower computational costs compared fine-tuning. Afterward, several works 4 [6 455 applied to the continual learning scenario and achieved promising results for mitigating catastrophic forgetting. Unlike them, we leverage in-context tuning to compensate for the shortcomings of prompt tuning on few-shot tasks.
Few-shot In-Context Learning Recently, in-context learning has emerged as a promising approach to improve the performance of large language models (LLMs) such GPT-3, particularly in scenarios where only limited training data is available [34] However, LLMs are often computationally expensive to train and inference, which limits their practical use in many real-world applications. To address this limitation, recent studies [12 [33 [4] focus on extending the capabilities of in-context learning to medium- or small-scale pre-trained language models (PLMs) by tuning them with the demonstrations. Inspired by these works, we propose to utilize ICT in parameter-efficient framework as means acquire contextual information.
7 Conclusion & Future Work
In this paper, we present few-shot continual learning method that integrates parameter-efficien efficcien fine-tuning (PEFT) and in-context tuning (ICT) to handle the task stream in table semantic parsing. The teacher model is designed extract contextual information from sampled demonstrations for few-shot learning, while the student model is tasked with learning the teacher's output distribution. It employs parameter-efficient prompt-tuning catastrophic forgetting. Through extensive experimentation, we' ve demonstrated that our method surpasses all compared baselines. In our future research, we aim investigate strategies to ensure the model's zero-shot predictive capabilities, while simultancously preventing catastrophic forgetting. Theoretically, our proposed method generalizable and has the potential be adapted to other NLP tasks because it is not limited to specific backbone models and does not involve task-specific modules. We leave the evaluation of C3 on other tasks our future work.
retain its capabilities and to entirely eradicate
Acknowledgements and Disclosure of Funding
This work is partially supported by the Natural Science Foundation of China (Grant No. U21A20488). We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculation in this paper.
10

---

References
[1] Wang, P., T. Shi, C. K. Reddy. Text-to-sql generation for question answering on electronic medical records. In Y. Huang, King. Liu, M. van Steen, eds., WWW '20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 350 361. ACM/ / IW3C2, 2020.
[2] Yuan, Y. Analysis and design of human resource management model of retail enterprises under the background of big data. In M. A. Jan, F. Khan, eds., Application of Big Data, Blockchain, and Internet of Things for Education Informatization, pages 31-37. Springer International Publishing, Cham, 2021.
[3] Stockinger, K., N. Bundi, J. Heitz, et al. Scalable architecture for big data financial analytics: user-defined functions vs. SQL. J. Big Data, 6:46, 2019.
[4] Zhong, V., C. Xiong, R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00100 2017.
[5] Yu, T., R. Zhang, K. Yang, et al. Spider: A large-scale labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In E. Riloff, D. Chiang, J. Hockenmaier, J. Tsujii, eds., Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, pages 3911-3921, Association for Computational Linguistics, 2018.
[6] Yu, T., R. Zhang, H. Er, et al. Cosql: A conversational text-to-sql challenge towards cross- domain natural language interfaces databases. In K. Inui, J. Jiang, V. Ng. X. Wan, eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pages 1962 1979. Association for Computational Linguistics, 2019.
[7] Chen, Y., X. Guo, T. Wu, et al. Learn from yesterday: A supervisee continual learning method for supervision-
selt task streams. CoRR, abs/2211. 1.11221,,11226, 2022.
[8] Guo, X., Y. Chen, G. Qi, et al. Improving few-shot text-to-sql with meta self-training via column specificity. In L.. D. Raedt, ed., Proceedings of the Thirty-Firsi International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 4150-4156. ijcai.org, 2022.
[9] Chang. S., P. Liu, Y. Tang, al. Zero-shot text-to-sql learning with auxiliary task. In The Thirty- Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7488-7495, AAAI Press, 2020.
[10] Chen, Y., X. Guo, C. Wang, et al. Leveraging table content for zero-shot text-to-sql with meta-learning In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 3992-4000, AAAI Press, 2021.
[11] Li, Z., Qu, G. Haffari. Total recall: customized continual learning method for neural semantic parsers, In M. Moens, X. Huang, L. Specia, S. W. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3816-3831. Association
for Computational Linguistics, 2021.
[12] Min, S., M. Lewis, L. Zettlemoyer, et al. Metaicl: Learning to learn in context. In M. Carpuat, M. de Marneffe, I. V. M. Ruíz, eds., Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 2791-2809. Association for Computational Linguisties, 2022.
[13) Chen, Y., R. Zhong, S. Zha, et al. Meta-learning via language model in-context tuning. In S. Muresan, P. Nakov, A. Villavicencio, eds., Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 719-730. Association for Computational Linguistics, 2022.
11

---

[14] Hu, Y. C. Lee, T. Xie, et al. In-context learning for few-shot dialogue state tracking, In Y. Goldberg, Z. Kozareva, Y. Zhang, eds., Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2627-2643 Association for Computational Linguistics, 2022.
[15] Lester, B., R. AI-Rfou, N. Constant. The power of scale for parameter-efficiet prompt tuning. In M. Moens, X. Huang, L. Specia, S. W. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045-3059. Association for Computational Linguistics, 2021.
[16] Zhu, Q. B. Li, Mi, al. Continual prompt tuning for dialog state tracking. In S. Muresan, P. Nakov, A. Villavicencio, eds., Proceedings of the 60th Annual Meeting of the Association for Computationnl Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Iveland, May 22-27, 2022, pages 1124-1137 Association for Computational Linguistics, 2022.
[17] Wang, Z., Z. Zhang, C. Lee, et al. Learning to prompt for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 139-149. IEFE, 2022.
[18) Raffel, C., N. Shazeer, A. Roberts, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 10:4::677 2020.
[19] Scholak, T., N. Schucher, D. Bahdanau. PICARD: parsing incrementally for constrained auto- regressive decoding from language models. In M. Moens, X. Huang. L. Specia, S. W. Yih, eds.. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punta Cana, Dominican Republic, 7-11 November, 2021, pages
9895 9901. Association for Computational Linguistics, 2021.
[20] Lester, B., R. AI-Rfou, N. Constant. The power of scale for parameter-efficient prompt tuning. In M. Moens, X. Huang, L. Specia, S. W. Yih, eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3045-3059. Association for Computational
Linguistics, 2021.
[21] Shazeer, N., M. Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In J. G. Dy, A. Krause, eds., Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, vol. 80 of Proceedings of Machine Learning Research, pages 4603-4611, PMLR, 2018.
[22] Zheng, Y., H. Wang. B. Dong, et al. HIE-SQL: history information enhanced network for context-dependent text-to-sql semantic parsing. In S. Muresan, P. Nakov, A. Villavicencio, eds., Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2997-3007. Association for Computational Linguistics, 2022.
[23] Dai, D., Y. Sun, L. Dong, et al. Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers. CoRR, abs/2212.10553 2022.
[24] Liu, J., D. Shen, Y. Zhang, et al. What makes good in-context examples for gpt-3? In E. Agirre, M. Apidianaki, I. Vulic, eds., Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, Deel.O@AAC 2022, Dublin, Ireland and Online, May 27, 2022, pages (00-114. Association for Computational Linguistics, 2022.
[25] Cer, D., M. Diab, E. Agirre, et al. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2011), pages 1-14. Association for Computational Linguistics, Vancouver, Canada, 2017.
[26] Fu, Y., H. Peng. L. Ou et al. Specializing smaller language models towards multi-step reasoning. CoRR, abs/2301.117226 2023.
[27] Wang, H., W. Xiong, M. Yu, et al. Sentence embedding alignment for lifelong relation extraction. In J. Burstein, C. Doran, T. Solorio, eds., Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 706. Association for Computational Linguistics, 2019.
12

---

[28] Kirkpatrick, J., R. Pascanu, N. C. Rabinowitz, et al. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.007796 2016.
[29] Serrå, J., D. Suris, M. Miron, et al. Overcoming catastrophic forgetting with hard attention to the task. In J. G. Dy, A. Krause, eds.. Proceedings of the 35th International Conference on Machine Learning, ICML 2018. Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, vol. 80 of Proceedings of Machine Learning Research, pages 4555-4564 PMLR, 2018.
[30] Han, X., Y. Dai, T. Gao, et al. Continual relation learning via episodic memory activation and reconsolidation. In D. Jurafsky, Chai, N. Schluter, J. R. Tetreault, eds., Proceedings of the 58th Annual Meeting of the Association for Computational Lingwistics, ACL 2020, Online, July 5-10, 2020, pages 6429-644 6440. Association for Computational Linguistics, 2020.
[31] Mi, F., L. Chen, M. Zhao, et al. Continual learning for natural language generation in task- oriented dialog systems. In T. Cohn, Y. He, Y. Liu, eds., Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, vol. EMNLP 2020 of Findings of ACL, pages 3461-3474. Association for Computational Linguistics, 2020.
[32] Yu, T. C. Wu, X. V. Lin, et al. Grappa: Grammar-augmenter pre-training for table semantic parsing. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[33] Guo, J. Z. Zhan, Y. Gao, et al. Towards complex text-to-sql in cross-domain database with intermediate representation. In A. Korhonen, D. R. Traum, L.. Mårquez, eds., Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019. Volume 1: Long Papers, pages 4524-4535. Association for Computational Linguistics, 2019.
(34] Brown, T. B., B. Mann, N. Ryder, et al. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, H. Lin, eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. 2020.
[35] OpenAL. GPT-4 technical report. CoRR, abs/2303.08774 2023.
[36] Pasupat, P., P. Liang. Compositional semantic parsing semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 1: Long Papers, pages 1470-1480. The Association for Computer Linguistics, 2015.
[37] Chen, W., H. Zha, Z. Chen, al. Hybridqa: A dataset of multi- question answering over tabular and textual data. In T. Cohn, Y. He, Y. Liu, eds., Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, vol. EMNLP 2020 of Findings of ACI,, pages 1026-1036. Association for Computational Linguistics, 2020.
[38] Hwang, W., J. Yim, S. Park, et al. A comprehensive exploration on wikisql with table-aware word contextualization. CoRR, abs/1902.01066 2019.
[39] Xu, K., Y. Wang, Y. Wang, et al. Sead: End-to-end text-to-sql generation with schema-aware denoising. In M. Carpuat, M. de Marneffe, I. V. M. Ruíz, eds., Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1845-1853, Association for Computational Linguistics, 2022.
[40] Wang, B., R. Shin, X. Liu, et al. RAT-SQL: aelat schema encoding and linking for text-to-sql parsers. In D. Jurafsky, J. Chai, N. Schluter. J. R. Tetreault, eds., Proceedings the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7567-7578. Association for Computational Linguistics, 2020.
[41] Cao, R., L. Chen, Z. Chen, et al. LGESQL: line graph enhanced text-to-sql model with mixed local and non-local relations. In C. Zong, F. Xia, W. Li, R. Navigli, eds., Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 2541-2555. Association for Computational Linguistics, 2021.
[42] Yu, T., R. Zhang, M. Yasunaga, et al. Spare: Cross-domain semantic parsing context. In
A. Korhonen, D. R. Traum, L. Marquez, eds., Proceedings of the 57th Conference of the
13

---

Association for Computational Linguistics, ACL 2019, Florence. Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4511-4523 Association for Computational Linguistics, 2019.
[43] Zhang, R., T. Yu, H. Er, al. Editing-based SQL query generation for cross-domain context- dependent questions. In K. Inui, J. Jiang, V. Ng, X. Wan, eds., Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5337-5348. Association for Computational Linguistics, 2019.
[44] Qin, C., S. R. Joty. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T3. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[45] Gao, Q. C. Zhao, Y. Sun, et al. A unified continual learning framework with general parameter- efficient tuning. CoRR, abs/2303. 10070, 2023.
14

---

A Hyperparameter Details
The detailed settings of the hyper-parameters used in our experiments are presented in Table
Table Hyper-pane Bayper-parameters settings.
Method
Symbol
Value
Description
150
prompt tokens
demonstrations
threshold
12
batch size
0.3
learning rate for PROMPT-TUNING
heam size
C3
10
learning rate for FINE-TUNING
evaluation done every epochs PROMPT-TUNING
50
T5-BASE T5-SMALL
evaluation is done every # epochs for PROMPI-TUNING of TS-LARGE
evaluation is done every # epochs for FINE-TUNING of
T5-BASE T5-SMALL
evaluation done every epochs FINE-TUNING T5-LARGE # of maximum epochs for PROMPT-TUNING
15000
300
# of maximum epochs for FINE-TUNING
512
maximum input tokens
stop training when the specified worsens evaluations
batch size
learning rate
memory size replayed examples in EMR, EMAR.
15
TR and EWC
beam size # of maximum epochs
55
Other Basclines
maximum epochs for second iterations EMAR
N-way MAML
K-shot MAML
25
# of meta-task in MAML
300
maximum of input tokens
# of demonstrations
1.0
regulation weight EWC
B Tokenizer Alignment: GPT and T5
When calling OpenAl's API to utilize text-davinci-003 (hereinafter referred to as GPT), we can get a predicted SQL query Po.r made up of L GPT tokens and sequence of output distributions S1.7 where each 8; s is the probabilities of the most probable tokens at each decoding step, corresponding to p;. Our objective is to convert the GPT output distributions to the the format of T5 tokenizer and get the target distributions where each t, E contains the probabilities of T5 tokens, making it trainable for the student models. To achieve this goal, we apply the algorithm shown in Algorithm
First, we evaluate whether GPT produces the correct SQL by comparing p with the gold SQL If not, we expand g as a sequence of one-hot distributions for student's training. Otherwise, we tokenize the GPT predicted SQL query using the T5 tokenizer and get sequence of one-hot distributions qi:k made up of T5 tokens with probability 1. And our target is to align s and q. Here we apply an one-to-one strategy. If there exists one-to-one mapping of T5 token q; and GPT token Pj. we use the GPT distribution s, as the T5 distribution. In other cases where a mismatch exists, we simply take the one-hot distribution qi the label.
15

---

Algorithm Tokenizer alignment GPT and T5
Require: GPT predicted SQL query P1.L. GPT output distributions $1.L. gold SQL query g. T5 Tokenizer T.
initialize t [=
function FINDONETOONEMAPPINGGG p,
Exists an one-to-one mapping of q; and P, then
return Sj
else
return None
end
end function if is equivalent to then
qı:k T. TT.TOKENZE(() for i := to K do
FINDONNTTONNEMPPPINGG p. not None then if EAPPENDDENDOOTTTOONEE.PPINGG P, s))
clse t.APPEND(q:)
end if end for else t = g
end return t
16


Reviews:
[{"review_id": "YmctoT9zXM", "reviewer": "Reviewer_Bddu", "summary": "The paper proposes a new continual learning method C3 for table semantic parsing which combines parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT). The C3 framework contains a teacher network (ICT) that extracts contextual information from demonstrations and a student network (PEFT) that learns the teacher's output distribution. The ICT aims to enhance the ability of few-shot learning and PEFT aims to reduce catastrophic forgetting. The author construct two stream datasets from WikiSQL and Spider to evaluate their proposed method. C3 outperforms few-shot learning and continual learning baselines on these stream datasets.", "questions": "1. In Table 3, when employing a T5-base model as the student model, it is observed that T5-large outperforms GPT-3 as the teacher model. However, interestingly, when the student model is upgraded to T5-large, the choice of teacher model seems less important. I am curious about the performance implications if the teacher model is completely removed when utilizing T5-large as the student model. I don't see this ablation study in Table 2.", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper combines parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) with a teacher-student framework to leverage the unique advantages of each approach.  The proposed methods are effective in the continual table semantic parsing task. \n2. The proposed framework is model-agnostic and seems generalizable to other continual learning tasks, further enhancing its value and applicability.", "weaknesses": "I don't see significant weakness but please address my concerns in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to you for providing us with valuable feedback and suggestions for our paper. We will provide explanations and clarifications for each question.\n\n#### For Question 1:\nWe hypothesize that when the teacher model and student model share the same architecture (T5), the greater the difference in scale between the two, the more pronounced the distillation effect is. \n\nWe explored the performance of C3 when removing the teacher model and using only T5-Large as the student model. The experimental results are shown in the following table.\n\n| Method               |            | Spider-Stream |            | |       Combined-Stream      |            |\n| -------------------- | :--------: | :-----------: | :--------: | :-------------: | :--------: | :--------: |\n|                      | **TA (%)** |  **EA (%)**   | **MD (%)** |   **TA (%)**    | **EA (%)** | **MD (%)** |\n| T5-Large + PEFT      |    69.7    |     67.4      |     0      |      67.3       |    70.0    |     0      |\n| T5-Large + C3 (Ours) |    70.7    |     68.9      |     0      |      69.0       |    71.2    |     0      |\n\nWhen using T5-Large as a backbone, removing the teacher model still results in a performance degradation compared to the entire C3.\nIf this paper is accepted, we'll add these results to the camera-ready version."}, {"review_id": "p5bCUWfSVT", "reviewer": "Reviewer_vuh7", "summary": "This paper introduces a novel method for training a continual table semantic parser, which aims to translate natural language into SQL based on task-specific tables with limited training examples. The proposed method integrates parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) to overcome the challenges of overfitting and catastrophic forgetting. The paper presents a task-adaptive PEFT framework and a teacher-student framework-based solution. Experimental evaluations demonstrate the superiority of the proposed method over existing baselines.", "questions": "- I notice \"Demo 2\" appears twice in Figure 1. Is it a typo?\n- I noticed that the paper utilizes the first dataset to initialize the model parameters for the entire model. This approach raises a natural concern: does it imply that the proposed method is only applicable to scenarios where the logical form is similar, but not when the logical form differs completely? ", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses an important problem in the field of table semantic parsing, namely the challenge of training a parser on a sequence of tasks with limited supervision.\n- The proposed method integrates two existing techniques, PEFT and ICT, to overcome overfitting and catastrophic forgetting, respectively. This combination of methodologies is neat and straightforward.\n- This paper provides a well-structured review of relevant literature, discussing prior work on table semantic parsing and related topics.\n- The analysis is comprehensive and gives more insight on the future research.\n", "weaknesses": "- The baseline comparison appears somewhat unfair. While I appreciate the inclusion of multiple baselines by this paper, I noticed that most of them were implemented on GRAPPA to highlight their inferior performance, whereas this paper's proposed method was applied to T5 (base and large). Additionally, there is a lack of reported MD performance for both PEFT and C3, which should be the key metric to reflect the forgetting degree. This inconsistency in reporting the experimental setup leaves me somewhat confused about the evaluation process and the methodology employed in this paper.\n- The results in Table 1 show that PEFT is dominant for catastrophic forgetting, while C3 is less dominant. in-context tuning does not play a prominent role here. And PEFT is already well-known for its advantage to avoid catastrophic forgetting, with the limitation that it cannot learn very different tasks fast.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to you for providing us with valuable feedback and suggestions for our paper. We will provide explanations and clarifications for each weakness and question.\n\n#### For Weakness 1:\nWe chose GRAPPA as the backbone PLM for baselines for two main reasons:\n1. Unlike T5, GRAPPA is pre-trained specifically for the TSP task (including Spider and WikiSQL), so we intuitively thought it would have stronger performance in our scenario.\n2. Since some of the baselines we used predominantly applied PLMs with encoder-only architectures in the original papers, e.g., [8] (RoBERTa), [11] (BERT), we also followed them to use an encoder-only PLM (GRAPPA) in order to maintain the consistency with the original papers.\n\nIn fact, by comparing the fine-tuning results in $\\rm{Table} \\ 1$ and $\\rm{Table} \\ 2$, it can be seen that GRAPPA performs better on Spider-stream. On the contrary, on Combined-stream, T5 has considerably less forgetfulness compared to Grappa's. We hypothesize that this is because T5's text-to-text transfer learning capability makes it less sensitive to significant differences in the logical form across tasks of Spider and WikiSQL.\n\nTo ensure a fairer comparison, we added the experiments that apply the best-performing EMR and EMAR (in $\\rm{Table} \\ 1$ of our paper) to T5 as the new baselines. The experimental results are shown as follows:\n\n\n| Method              |            | Spider-Stream |            |            | Combined-Stream |            |\n| ------------------- | :--------: | :-----------: | :--------: | :--------: | :-------------: | :--------: |\n|                     | **TA (%)** |  **EA (%)**   | **MD (%)** | **TA (%)** |   **EA (%)**    | **MD (%)** |\n| T5-Base + EMR       |    59.1    |     56.7      |   -13.4    |    62.5    |      62.4       |    -5.7    |\n| T5-Base + EMAR      |    57.8    |     53.6      |   -15.7    |    62.0    |      59.7       |    -6.5    |\n| T5-Base + C3 (Ours) |    67.7    |     66.7      |     0      |    66.4    |      67.7       |     0      |\n\n\nFrom the results, C3 still has a significant improvement compared to the strong T5 baselines on both datasets. If this paper is accepted, we'll add these results to the camera-ready version.\n\nNote that the reason we did not add the MD performance of PEFT and C3 in $\\rm{Table} \\ 1$ and $\\rm{Table} \\ 2$ is because they have no forgetting, i.e., MD=0. Freezing PLM parameters ensures that they are not updated. Simply loading the checkpoint (soft prompt) for each task allows for an exact replication of the performance on previous tasks. We initially thought we should just omit it here because these cells will be constant values (zero). We regret that this omission misled you, and we will make a special note of it in the caption in subsequent version.\n\n#### For Weakness 2:\nIn our scenario, catastrophic forgetting is more challenging compared to the few-shot problem. We add ICT's teacher model to PEFT precisely to enhance its lack of fast learning on new tasks. Although ICT's contribution to the overall performance is not as significant as PEFT, it is true that C3 achieves about 2% improvement over PEFT on both benchmarks in the case of multiple runs.\n\n#### For Question 1:\nSorry, this is a typo, it should actually be `Demo 3`.\n\n#### For Question 2:\nIn fact, we built Combined-Stream based on such a similar motivation as you mentioned. In Combined-Stream, the first task is based on Spider, while the second task is based on WikiSQL, and then alternates in turn. Spider has a complex SQL structure with syntax such as `JOIN`, `Nested Query`, `GROUP BY`, etc., whereas WikiSQL has only simple single-table SQL without complex syntax. To some extent, this setting can reflect the impact of the initial task on subsequent tasks. From the following three plots in $\\rm{Figure} \\ 4$, the performance oscillations in the baseline due to this stronger impact, which is effectively mitigated by our proposed method."}, {"review_id": "GkqBtgw8p2", "reviewer": "Reviewer_yC62", "summary": "The paper introduces a method combining parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) to address the issues of overfitting and catastrophic forgetting in training a continual table semantic parser with limited training examples. Through a task-adaptive PEFT framework and a teacher-student setup that utilizes ICT, the method demonstrates enhanced performance in comparison to established baselines, as validated by experiments on two benchmarks.", "questions": "### Questions\n1. Why the authors choose tabular semantic parsing? Is this method generalizable to other tasks (e.g., general NLU/NLG tasks)?\n2. Why use different backbones for other methods in Table 1? How do those methods perform on T5?\n\n### Typos\nLn. 23: \"finance [3], and ?\"", "limitations": "No", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is overall good written and easy to follow.\n2. This paper proposes a method that fuses PEFT with ICT to resolve the overfitting and catastrophic forgetting problem. \n3. In addition to PEFT + ICT, the authors also propose a teacher-student framework that distills ICT results on teacher to a student model.", "weaknesses": "My main concern about this paper is its technical novelty.\n1. Combining PEFT and ICT (demonstration) is not new. e.g., [Gao et al.](https://arxiv.org/abs/2012.15723)\n2. Distilling soft prompts, on the other hand, is not new either. https://arxiv.org/abs/2212.10670 https://arxiv.org/abs/2304.08467\n3. I found it unconvincing to exclude results from [7]. C3 also uses demonstration retrieval, which indicates the same pool of supervised data is used. \n4. Following 3, the claim that the method is few-shot is inaccurate. Since the examplars are selected from a pool examples, comparing to methods that only use a few examples is unfair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to you for providing us with valuable feedback and suggestions for our paper. We will provide explanations and clarifications for each weakness and question.\n\n#### For Weakness 1:\nOur method is fundamentally different from that of [Gao et al.] for the following two reasons:\n1. [Gao et al.] is mainly based on hard prompt + ICT, which still requires fine-tuning the whole model rather than a subset of parameters during the training process, so it cannot be considered as a PEFT approach. In contrast, our method only updates soft prompt while freezing the whole PLM during the training of the teacher model, which reflects the parameter efficiency.\n2. [Gao et al.] focuses on how to improve the few-shot learning capability of smaller pre-trained models on static datasets, whereas we focus on a dynamic task-stream scenario, exploring how to avoid catastrophic forgetting while guaranteeing few-shot learning capability.\n\n* Gao, Tianyu, Adam Fisch, and Danqi Chen. \"Making pre-trained language models better few-shot learners.\" arXiv preprint arXiv:2012.15723 (2020).\n\n#### For Weakness 2:\n1. Although [Huang et al.] also proposed in-conext learning distillation, their student model is essentially different from ours in terms of motivation. Their student model focuses only on the few-shot learning capability and still employs in-context learning for prediction without soft prompts. In contrast, our student model needs to handle dynamic task-stream scenarios. The invisibility of the past task context forces C3 to use PEFT to inject in-context learning capabilities into the soft prompts, avoiding catastrophic forgetting while improving few-shot learning.\n\n2. The goal of [Mu et al.] is also not to solve the few-shot or continual learning problem, but to compress the prompt to improve LLM inference efficiency and save storage space. This is completely different from the dynamic task steam scenarios we've focused on.\n\n*  Huang, Yukun, et al. \"In-context Learning Distillation: Transferring Few-shot Learning Ability of Pre-trained Language Models.\" arXiv preprint arXiv:2212.10670 (2022). \n* Mu, Jesse, Xiang Lisa Li, and Noah Goodman. \"Learning to compress prompts with gist tokens.\" arXiv preprint arXiv:2304.08467 (2023). \n\n#### For Weakness 3:\nWe did not compare with [7] because [7] defines a slightly different scenario than the one we are concerned with. They first assume that each task in the stream has corresponding unlabeled training set $ D^i_\\rm{unsup}$, which contains only NLQs without gold SQL queries. Then, their method SFNet performs semi-supervised learning of the labeled training set $D^i_\\rm{train}$ in combination with $D^i_\\rm{unsup}$, whereas our method uses only $D^i_\\rm{train}$. To ensure that all methods use the same training data, we did not compare with [7].\n\nHowever, to make the comparison more convincing, we added an experiment running [7] on Combined-Stream. The results are shown in the table below. \n\n| **Method**          | TA (%) | EA (%) | MD (%) |\n| ------------------- | :----: | :----: | :----: |\n| SFNet [7]           |  61.9  |  59.6  |  -3.2  |\n| T5-Base + C3 (Ours) |  67.7  |  66.7  |   0    |\n\nThe reason why we only use Combined-stream and not on Spider-stream here is that Combined-stream only randomly selects a portion of samples from the original Spider and WikiSQL datasets, and the rest can be used as unsupervised data to fulfill the scenario requirements of [7].\n\nDespite utilizing additional unsupervised data for semi-supervised learning, [7] still does not perform as well as our proposed C3. Our method achieves better results under weaker assumptions.\n\n#### For Weakness 4:\nYou may have some misunderstanding of our method. For $i$-th task, all the demonstrations of our C3 are sampled from the training set $D^i_\\rm{train}$ (line 153), so the training set of C3 is still $D^i_\\rm{train}$, which is consistent with all compared methods. So comparisons are fair.\nOur few-shot learning scenario setup follows existing work [8,9]. We adopt the standard $N$-way $K$-shot definition: \"way\" denotes the database and \"shot\" denotes the training example corresponding to the database.\n* For Spider-stream/Combined-Stream, there were an average of 337.8/292.7 training examples per task and 10.1/52.5 databases, i.e., 10-way 34-shot / 53-way 6-shot.\n\nWe will include this setting to the future release.\n\n\n#### For Question 1:\nYes. Theoretically, our proposed method is generalizable and has the potential to be adapted to other NLP tasks because it is not limited to specific backbone models and does not involve task-specific modules.\n\nWe chose TSP for several reasons:\n1. Databases are one of the most widely used information carriers, playing a fundamental role in various domains. As a key technology for natural language interfaces to databases, the study of TSP is of great significance for the enhancement of many fields. In addition, the natural continuous updating of databases leads to the task stream.\n2. TSP is a difficult task. Unlike general NLU tasks (NER or RE), it requires the model to understand all aspects of the natural language query, including the entities (column names, table names) and the overall logical framework, and complete the mapping with the corresponding structured tables. Accordingly, the supervised data for this task is more difficult to obtain, resulting in the few-shot challenge.\n3. Unlike NLG tasks (like summarization, machine translation), TSP is more objective in evaluating the result metrics. Any small error in the SQL will lead to completely incorrect results. We believe that this task can more rigorously reflect the true performance of our method.\n\nIn future work, we will choose more different types of NLP tasks to evaluate our method.\n\n#### For Question 2:\nDue to space limitations, please see response to Weakness #1 mentioned by reviewer vuh7 for details."}, {"review_id": "0dhi3sknVn", "reviewer": "Reviewer_zd3d", "summary": "The author presents a novel approach that combines PEFT and ICT to train a continual table semantic parser. The proposed solution is based on a teacher-student framework. Two task streams are developed from WikiSQL and Spider semantic parsing benchmarks. Experiments using C3 and baselines are conducted using these task streams. The results indicate that their method outperforms existing competitors, achieving state-of-the-art performance across multiple metrics.\n", "questions": "- Is the Problem Formulation in Section 2.2 novel? Have other papers examined and studied this problem? If not, the problem formulation and its motivation should be part of the paper's contribution. \n- In Table 1, what is the PEFT results on T5-large? \n- In Figure 1, what are the blue arrows going into PLM in the ICT portion?  \n- In Figure 1, why are past demonstrations invisible? This seems to be an unrealistic setting, even under data privacy concerns. We can certainly have a diverse pool of demonstrations from public databases or synthetic/obfuscated database content. \n- In Figure 3, what is the x-axis label?\n", "limitations": "No negative societal impacts\n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The fuse of ICT and PEFT continual framework is novel and effective on the proposed continual table semantic parsing task.\n- The description of the framework and method is clearly written.\n", "weaknesses": "- The authors did not clearly state whether the Problem Formulation in Section 2.2 has been studied previously. \n- The Continual table semantic parsing problem is not well motivated. For example, the authors claimed `After training on a new task, the performance of the parser on the previous task may plummet attributed to parameter updates.`, but there is no empirical evidence of this claim as a motivating example. \n- The authors stated the \"invisibility of past demonstrations\" for ICT, which seems unrealistic even under data privacy concerns. ICT typically only requires a few examples, which can be obtained from public databases or synthetic/obfuscated database content.\n- The authors can have additional baseline: Train on a reasonable size of public text2sql data with a mixture of domains and then zero-shot on the Stream datasets. The public data will not have privacy concerns. This will show the accuracy gap and rationale behind the continual training. It seems line 196 is similar to this, what is the training set size? \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to you for providing us with valuable feedback and suggestions for our paper. We will provide explanations and clarifications for each weakness and question.\n\n#### For Weakness 1:\nBefore us, the problem formulation in $\\rm{Section} \\ 2.2$ was proposed by [7], so we did not include this as one of our contributions. However, their scenario is also slightly different from ours. They guarantee a portion of unsupervised data for each task on the stream, while this guarantee is removed in our scenario, making it more challenging.\n\n#### For Weakness 2:\nFor this \"the parser's performance on the previous task may drop drastically due to parameter updates\" claim, we have actually shown it experimentally. Specifically, in $\\rm{Figure} \\ 4$, the Fine-Tuned parser exposes a significant performance degradation of about 20% in terms of TA (Task Accuracy) and EA (Example Accuracy) on both datasets as the task increases. The dramatic drop in MD (Memory Decay) metrics also proves that the model really has catastrophic forgetting about previous tasks.\n\n#### For Weakness 3:\n\nPublic or synthetic/obfuscated databases are indeed a possible way to get demos. However, obtaining a valid demonstration may not be easy for two reasons:\n1. First, a demonstration is an NLQ-SQL pair, and database content alone still makes it expensive to construct a large scale of natural language that matches human expressions, especially for some small domains.\n2. Second, recent studies [23, 24] have shown that only in-context learning using demonstrations similar to the test samples can maximize performance gains. Unfortunately, for domain-specific databases, there is no guarantee that similar demonstrations will be found using public data resources. To verify this, we added an experiment: We assume that $D^0_\\rm{train}$, the training set of task_0, is an always-visible public demonstration pool from which all demos for subsequent tasks are sampled. The following table shows the average accuracy of the Teacher Parser when it encounters each task at the first time.\n\n| **Method**                         | Spider-Stream (%) | Combined-Stream (%) |\n| ---------------------------------- | :---------------: | :-----------------: |\n| Teacher Parser (Demos from task_0) |       76.3        |        70.6         |\n| C3 Teacher Parser (Ours)           |       78.2        |        72.4         |\n\nFrom the results, when sampling the demonstration from $D^0_\\rm{train}$, the performance of the Teacher parser shows a significant drop on both datasets. This again proves that the quality of the demonstration has a great impact on the performance of ICT.\n\n\n#### For Weakness 4:\nYes, our setup at line 196 is trying to simulate this scenario. We randomly choose k (6 for Spider-stream and 5 for Combine-stream) tasks to merge as an initial task (task_0) for a mixed domain. For the Spider-stream, the size of the training set for task_0 is 2082; For the Combined-stream, the size of the training set for task_0 is 1373.\n\n\n#### For Question 1:\nDue to space limitations, please see response to Weakness #1 for details.\n\n#### For Question 2:\nWe added a experiment to evaluate the performance of PEFT when using T5-Large. The experimental results are shown in the following table.\n\n\n| **Method**           | TA (%) | EA (%) | MD (%) | TA (%) | EA (%) | MD (%) |\n| -------------------- | :----: | :----: | :----: | :----: | :----: | :----: |\n| T5-Large + PEFT      |  69.7  |  67.4  |   0    |  67.3  |  70.0  |   0    |\n| T5-Large + C3 (Ours) |  70.7  |  68.9  |   0    |  69.0  |  71.2  |   0    |\n\n\nWhen using T5-Large as a backbone, our C3 still performs better than PEFT.\nIf this paper is accepted, we'll add these results to the camera-ready version.\n\n#### For Question 3:\nHere the blue arrow indicates that the model should maintain performance on the inputs from the previous task even when it encounters subsequent tasks. To avoid misunderstandings, we will add an explanation in caption in subsequent releases.\n\n#### For Question 4:\nDue to space limitations, please see response to Weakness #3 for details.\n\n#### For Question 5:\nThe x-axis represents the ID of the segmented task. Note that this is just the initial slice of the task (independent of the order), and as we mentioned before, when the methods are actually run, a random $k$ of them will constitute the initial task. The remaining $n - k$ tasks are shuffled as the task 2 to $n - k + 1$."}]

Author rebuttal:

Meta-review: This paper proposes a method that combines parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) to address the issues of catastrophic forgetting and overfitting for the table-based semantic parsing task. The teacher-student framework leverages ICT to extract contextual information from demonstrations and distills it to a student network that learns the teacher’s output distribution. The paper received mixed reviews from the 4 reviewers and the authors provided detailed answers, addressing most of the reviewers’ comments and questions. After reading the reviews, rebuttal, and discussions, I decided to recommend this paper an ACCEPT. In general, this paper presents an interesting and novel approach to table semantic parsing. Although the novelty of this paper (i.e., combining PEFT and ICT for table-based semantic parsing) was challenged by reviewer yC62, the authors have clearly explained the key differences between this work and previous work including Gao et al., 2020, Huang et al., 2022 and Mu et al., 2023. I highly recommend the authors to make these comparisons and explanations clearer in the revised version. Besides, as reviewer yC62 said, the proposed method can be seen as a general framework for most NLP tasks, instead of for table-based semantic paring only. So the authors should make it clear why table-based semantic paring is focused on here (as you did in the rebuttal).
Decision: Accept (poster)