SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our compared methods can be categorized into three groups: a) FINE-TUNING, which trains the parser sequentially without any strategies; b) Few-Shot Learning baselines, which contain MAML [8] and ICT [13]; c) Continual Learning baselines, which consist of EWC [28], HAT [29], EMR [27], EMAR [30], APPER [31], and TR [11]. To increase the competitiveness of these baselines, we utilized pre-trained GRAPPA-LARGE [32] for TSP in conjunction with SemQL [33], an intermediate representation for SQL. To ensure a fairer comparison, we conducted experiments applying the best-performing methods on GRAPPA, EMR and EMAR, to T5-BASE as complementary baselines. Additionally, we compared our methods with SFNet [7] which utilizes extra unsupervised data. Note that we only apply [7] on Combined-Stream for the reason that Combined-Stream randomly selects a portion of samples from the original Spider and WikiSQL datasets which enables the rest to be used as unsupervised data to fulfill the scenario requirements. We also set an upper boundary method MULTI-TASK. For each task  $\mathcal{D}^i$ , it trains  $\mathcal{F}_{\theta}$  with  $\mathcal{D}_{\mathrm{train}}^{(0:i)}$ .
Our method ran on one NVIDIA RTX 4090 GPU and the hyperparameters were decided with the validation sets (See Appendix for details): a) prompt length  $M$  is set to 150; b) demonstration number  $r$  and threshold  $\eta$  are set to 1 and 4.0, respectively; c) the batch size is 12 and the learning rates are set to 0.3 and  $1 \times 10^{-4}$  for prompt-tuning and fine-tuning, respectively; d) early stopping is performed after 1000 epochs and the patience is set to 10 evaluations, which is performed every 50 epochs. e) T5-LARGE is always employed as  $\mathcal{F}_{\mathrm{tea}}$ . All of our data and codes are publicly available<sup>1</sup>.
Table 1 presents the overall performance of the methods. The results indicate that most of the compared methods exhibit relatively lower effectiveness and more severe forgetting on Combined-Stream compared to Spider-Stream, even though their backbone models have been pre-trained for the
<https://github.com/Bahuia/C3>
Table 1: Experimental results for comparison with baselines in 3 random task orders. Means and standard variations are reported. The absence of standard deviation for PEFT and C3 is due to the fact that their performance is order-independent.  $\spadesuit$  indicates using the replayed memory of size 15 and  $\clubsuit$  indicates using additional unsupervised data.
Backbone	Method	Spider-Stream	Combined-Stream
TA (%)	EA (%)	MD (%)	TA (%)	EA (%)	MD (%)
GRAPPA-LARGE(340M)	FINE-TUNING	56.91.0	54.61.0	-18.81.5	37.61.8	43.90.9	-39.12.2
MAML [8]	52.21.3	49.11.5	-19.52.2	31.31.3	37.21.4	-43.81.5
ICT [13]	57.01.4	54.32.2	-17.12.1	37.94.0	43.91.8	-37.44.4
EWC [28]	57.53.3	55.12.4	-17.73.9	37.01.9	44.10.9	-38.42.2
HAT [29]	57.82.9	54.83.4	-17.03.3	38.54.1	45.02.0	-37.65.5
EMR♠ [27]	65.20.2	62.90.6	-9.40.8	60.90.7	58.61.8	-10.31.8
EMAR♠ [30]	62.81.2	60.81.2	-10.51.1	63.11.8	60.80.9	-7.72.6
APPER [31]	57.91.6	55.81.7	-17.22.6	37.12.4	44.06.6	-38.32.9
TR♠ [11]	57.91.2	55.11.4	-15.81.5	59.71.0	56.31.1	-11.90.7
SFNET♠ [7]	-	-	-	60.70.9	57.01.9	-6.01.2
T5-BASE(220M)	EMR♠ [27]	60.30.9	56.60.1	-13.40.2	62.60.4	60.01.4	-6.50.6
EMAR♠ [30]	57.20.4	52.71.0	-16.70.8	62.10.2	58.11.1	-6.50.3
PEFT	65.7	64.5	0.0	63.8	66.2	0.0
C3	67.50.3	66.50.2	0.00.0	66.30.2	67.60.2	0.00.0
MULTI-TASK	76.30.5	76.21.0	3.20.7	70.00.9	71.10.7	1.70.2
T5-LARGE(770M)	PEFT	69.8	67.4	0.0	67.3	70.0	0.0
C3	71.10.3	69.70.5	0.00.0	68.30.5	70.60.4	0.00.0
Figure 4: TA (\%), EA (\%), and MD (\%) till the seen tasks of Spider-Stream (upper) and Combined-Stream (bottom) after learning on each task. Only the means are reported in 3 random task orders.
tabular data. Surprisingly, our proposed PEFT framework outperforms all baseline models despite the absence of any explicit provision to address the few-shot problem, mainly due to its ability to retain previously learned knowledge. Moreover, our proposed C3 further improves the overall performance (4.9% & 5.1% in terms of TA) and achieves state-of-the-art results for both datasets. Of particular note is that even with a smaller backbone model (T5-BASE, 220M), C3 outperforms all compared methods based on GRAPPA-LARGE (340M). Remarkably, it manages to approach the upper bound of performance (MULTI-TASK) on Combined-Stream.
Compared to ICT, MAML performs poorly on both datasets, especially in terms of MD, even less than FINE-TUNING. This finding may point out that it is more profound and irreversible for parameter updates. The replay-based EMR and EMAR outperform other continual learning baselines in terms of all metrics, which suggests that retaining past instances may be more direct and effective for mitigating forgetting, particularly on Combined-Stream where the task varies more. Unfortunately, in many cases involving data privacy, access to past examples is disabled, rendering these methods limited. Conversely, our proposed C3 not only circumvents data privacy concerns and has a broader range of applications, but also achieves superior results without using any past examples.
Table 2: Experimental results of ablation studies in the same task order.  $\clubsuit$  indicates that the backbone used is T5-LARGE, since only TEACHER remains when STUDENT is removed.
Backbone	Method	Spider-Stream	Combined-Stream
TA (%)	EA (%)	MD (%)	TA (%)	EA (%)	MD (%)
T5-BASE	FINE-TUNING	53.3	50.8	-20.7	59.1	58.2	-9.1
C3	67.7	66.7	0.0	66.4	67.7	0.0
w/o TEACHER	65.7	64.5	0.0	63.8	66.2	0.0
w/o STUDENT	64.9	62.8	-14.6	59.6	59.6	-14.9
w/o In-Context Tuning	65.6	64.3	0.0	65.3	66.9	0.0
w/o Task Adaptation	19.4	20.6	0.0	35.8	25.7	0.0
w Continual Initialization	67.0	65.4	0.0	66.0	67.4	0.0
T5-LARGE	FINE-TUNING	60.5	56.3	-17.3	65.3	67.2	-8.8
C3	70.7	68.9	0.0	69.0	71.2	0.0
Table 3: Performance of C3 using GPT as the TEACHER parser.
STUDENT	TEACHER	Spider-Stream	Combined-Stream
TA (%)	EA (%)	TA (%)	EA (%)
T5-BASE	text-davinci-003	66.3	64.8	65.5	66.8
T5-LARGE	67.50.3	66.50.2	66.30.2	67.60.2
T5-LARGE	text-davinci-003	71.3	69.6	67.6	70.0
T5-LARGE	71.10.3	69.70.5	68.30.5	70.60.4
5.3 Detailed Results and Analysis
5.3.1 Performance Till the Seen Tasks
Figure 4 displays the methods' performance till the seen tasks on both datasets. Our proposed C3-LARGE (blue) consistently exhibits the highest performance across all evaluated time steps. Moreover, this performance advantage becomes increasingly pronounced as the number of tasks in the continual learning regime grows. Interestingly, the EA metrics on the Combined-Stream for compared methods always show oscillation. The performance troughs on the even-indexed tasks suggest a dramatic forgetting of the harder Spider tasks after the model is trained on an easier WikiSQL task.
5.3.2 Ablation Study
We compared the performance of the proposed C3 with the following settings:
- w/o TEACHER: we removed  $\mathcal{F}_{\mathrm{tea}}$  to verify its contribution;
- w/o STUDENT: we removed  $\mathcal{F}_{\mathrm{stu}}$  and only employed  $\mathcal{F}_{\mathrm{tea}}$  (T5-LARGE) to perform ICT, to evaluate the contribution of PROMPT-TUNING;
- w/o In-Context Tuning: we used FINE-TUNING to train  $\mathcal{F}_{\mathrm{tea}}$  to assess the necessity of ICT.
- w/o Task Adaptation: we used PROMPT-TUNING to train  $\mathcal{F}_{\mathrm{stu}}$  from  $\mathcal{D}^0$ .
- w Continual Initialization: we initialize the prompt embeddings  $\mathbf{P}^i$  with  $\mathbf{P}^{i - 1}$  instead of  $\mathbf{P}^*$ .
Table 2 shows the TA (\%), EA (\%), and MD (\%) of different settings. Our C3 equipped with all components performs best in terms of all metrics. Dramatic performance degradation demonstrates that the adaptation of the PLM to the task format is critical to the effectiveness of prompt tuning. Removing  $\mathcal{F}_{\mathrm{tea}}$  results in an approximate  $2\%$  drop in terms of both TA and EA, which proves the TEACHER's few-shot learning capability contributes to the entire model. With the exclusion of STUDENT, the TA exhibits a more pronounced decrease on Combined-Stream  $(-6.8\%)$  compared to Spider-Stream  $(-2.8\%)$  because Combined-Stream involves challenges of both domain changes and SQL structure changes. The performance drops  $(-2.3\%$  &  $-0.7\%)$  brought by abandoning ICT reveal the necessity of contextual information. Its smaller contribution on Combined-Stream is probably due to the relative simplicity of the WikiSQL tasks, which does not require a demonstration to make
Figure 5: Left two: performance of C3 with different prompt lengths  $M$  and PLM sizes; Right two: IA (\%) of the TEACHER parser with different thresholds  $\eta$  and demonstration numbers  $r$ , where * indicates the absence of the context mixing strategy (detailed in Section 4.1.1).
a correct prediction. In contrast to [16]'s assertion, our experiments did not show any improvement with Continual Initialization. One possible reason is the large gap between different tasks.
5.3.3 Impact of Large Language Models
Considering the amazing potential of Large Language Models (LLMs), such as GPT-3 [34] and GPT-4 [35], in the NLP community recently, we also evaluate the performance of C3 by replacing  $\mathcal{F}_{\mathrm{tea}}$  from T5-LARGE with the text-davinci-003, an LLM of the GPT family that excels in textual tasks. Here we did not choose code-davinci-002, which specifically handles code tasks, and the more widely used gpt-3.5-turbo, because the API of the former is deprecated and the latter can not provide output distribution for each decoding step. Referring to [26], we align the GPT tokenizer with the T5 tokenizer in order to calculate  $D_{\mathrm{KL}}$  (details in Appendix). Here we let text-davinci-003 perform inference based on the extracted demonstration without tuning its parameters, so C3 in this scenario is independent of the task order. The experimental results are shown in Table 3. Despite not having undergone any fine-tuning, using GPT has achieved close or even better results compared to using T5-LARGE. This proves, to some extent, that our proposed method is not limited to a specific architecture (encoder-decoder vs. decoder-only) and size (million-level vs. billion-level) of PLM.
5.3.4 Impact of Different Prompt Lengths & PLM Sizes
We varied the backbone PLM in {T5-SMALL, T5-BASE} and prompt length  $M$  in {1, 5, 20, 75, 150} for our proposed C3 on the two task streams. The experimental results are shown in Figure 5, which indicates that: When fixing the prompt length, increasing the backbone PLM size improves the TA; When the PLM size is fixed, increasing the prompt length improves the overall performance in general. Moreover, our findings indicate that extending the prompt length from 20 to 75 enhances TA more significantly than increasing it from 75 to 150 on both task streams.
5.3.5 Impact of Different Demonstration Numbers & Thresholds
We also varied the demonstration number  $r$  in  $\{1, 2, 3\}$  and threshold  $\eta$  in  $\{2.0, 3.0, 4.0\}$  for the TEACHER  $\mathcal{F}_{\mathrm{tea}}$  in our proposed C3. Figure 5 shows IA (\%). We observed that removing the context mixing strategy leads to severe performance degradation, proving our hypothesis that there exists many examples of sufficient demonstrations. When using context mixing, the performance is not significantly affected by the threshold and number of demonstrations, but a trade-off between performance and efficiency is achieved with a threshold of 4.0 and using only one demonstration.
5.3.6 Performance of TEACHER Parser with Always-Visible Demonstrations
While in many works involving in-context learning there exists an always-visible demonstration pool (e.g., public or synthetic datasets) from which we can select demonstration for all tasks, resources for TSP tasks are quite limited due to high expense of annotation. Therefore, C3 applies a practical strategy which samples demonstrations from every incoming task. However, to study the above sampling strategy, we conducted an additional experiment assuming that  $\mathcal{D}_{\mathrm{train}}^0$ , the training set for task 0, is an always-visible public demonstration pool from which all demonstrations for subsequent tasks are sampled. Table 4 shows IA (\%) of a T5-LARGE based teacher parser. From the results, when sampling the demonstration from  $\mathcal{D}_{\mathrm{train}}^0$ , the performance of the teacher parser shows a significant drop on both datasets. This reflects the conclusions in [23, 24], which state that performance gains are maximized only when in-context learning utilizes demonstrations similar to the test examples.
Table 4: IA (%) of TEACHER Parser with Always-Visible Demonstrations.
TEACHER	Spider-Stream	Combined-Stream
Demonstrations from D0train	76.3	70.6
C3	78.2	72.4
For domain-specific databases, there is no guarantee that similar demonstrations will be found using public data resources. Our results prove that the quality of the demonstration has a great impact on the performance of ICT.
Table Semantic Parsing In recent years, there has been a notable increase in interest concerning joint textual-tabular data understanding problems [36, 37], especially in table semantic parsing (TSP) [4, 5]. This trend can be attributed to the fact that many real-world datasets contain a combination of structured and unstructured data, and a comprehensive understanding of such data requires the ability to analyze and interpret both forms effectively. The related methods can be divided into three directions according to the application scenarios, namely single table [4, 38, 10, 39], multtable [5, 33, 40, 41] and conversational tasks [42, 6, 43, 22].
Parameter Efficient Continual Learning Prompt tuning was first systematically studied by [20], who demonstrated that fine-tuning language models on a small number of task-specific input prompts could result in substantial performance improvements with much lower computational costs compared to full-model fine-tuning. Afterward, several works [44, 17, 16, 45] applied it to the continual learning scenario and achieved promising results for mitigating catastrophic forgetting. Unlike them, we leverage in-context tuning to compensate for the shortcomings of prompt tuning on few-shot tasks.
Few-shot In-Context Learning Recently, in-context learning has emerged as a promising approach to improve the performance of large language models (LLMs) such as GPT-3, particularly in scenarios where only limited training data is available [34]. However, LLMs are often computationally expensive to train and inference, which limits their practical use in many real-world applications. To address this limitation, recent studies [12, 13, 14] focus on extending the capabilities of in-context learning to medium- or small-scale pre-trained language models (PLMs) by tuning them with the demonstrations. Inspired by these works, we propose to utilize ICT in a parameter-efficient framework as a means to acquire contextual information.
In this paper, we present a few-shot continual learning method that integrates parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) to handle the task stream in table semantic parsing. The teacher model is designed to extract contextual information from sampled demonstrations for few-shot learning, while the student model is tasked with learning the teacher's output distribution. It employs parameter-efficient prompt-tuning to retain its capabilities and to entirely eradicate catastrophic forgetting. Through extensive experimentation, we've demonstrated that our method surpasses all compared baselines. In our future research, we aim to investigate strategies to ensure the model's zero-shot predictive capabilities, while simultaneously preventing catastrophic forgetting. Theoretically, our proposed method is generalizable and has the potential to be adapted to other NLP tasks because it is not limited to specific backbone models and does not involve task-specific modules. We leave the evaluation of C3 on other tasks to our future work.
This work is partially supported by the Natural Science Foundation of China (Grant No. U21A20488). We thank the Big Data Computing Center of Southeast University for providing the facility support on the numerical calculation in this paper.
Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 4511-4523. Association for Computational Linguistics, 2019.
[43] Zhang, R., T. Yu, H. Er, et al. Editing-based SQL query generation for cross-domain context-dependent questions. In K. Inui, J. Jiang, V. Ng, X. Wan, eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5337-5348. Association for Computational Linguistics, 2019.
[44] Qin, C., S. R. Joty. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[45] Gao, Q., C. Zhao, Y. Sun, et al. A unified continual learning framework with general parameter-efficient tuning. CoRR, abs/2303.10070, 2023.
A Hyperparameter Details
The detailed settings of the hyper-parameters used in our experiments are presented in Table 5.
Table 5: Hyper-parameters settings.
Method	Symbol	Value	Description
C3	M	150	# of prompt tokens
r	1	# of demonstrations
η	4.0	threshold
-	12	batch size
-	0.3	learning rate for PROMPT-TUNING
-	4	beam size
-	1 × 10-4	learning rate for FINE-TUNING
-	50	evaluation is done every # epochs for PROMPT-TUNING of T5-BASE or T5-SMALL
-	25	evaluation is done every # epochs for PROMPT-TUNING of T5-LARGE
-	10	evaluation is done every # epochs for FINE-TUNING of T5-BASE or T5-SMALL
-	5	evaluation is done every # epochs for FINE-TUNING of T5-LARGE
-	15000	# of maximum epochs for PROMPT-TUNING
-	300	# of maximum epochs for FINE-TUNING
-	512	maximum # of input tokens
-	10	stop training when the specified metric worsens for # evaluations
Other Baselines	-	32	batch size
-	2 × 10-4	learning rate
-	15	memory size of replayed examples in EMR, EMAR, TR and EWC
-	5	beam size
-	55	# of maximum epochs
-	1	# of maximum epochs for the second iterations in EMAR
-	2	N-way of MAML
-	1	K-shot of MAML
-	25	# of meta-task in MAML
-	300	maximum # of input tokens
-	1	# of demonstrations
-	1.0	regulation weight in EWC
B Tokenizer Alignment: GPT and T5
When calling OpenAI's API to utilize text-davinci-003 (hereinafter referred to as GPT), we can get a predicted SQL query  $\mathbf{p}_{0:L}$  made up of  $L$  GPT tokens and a sequence of output distributions  $\mathbf{s}_{1:L}$  where each  $\mathbf{s}_i\in \mathbf{s}$  is the probabilities of the most probable tokens at each decoding step, corresponding to  $\mathbf{p}_i$ . Our objective is to convert the GPT output distributions to the format of T5 tokenizer and get the target distributions  $\mathbf{t}$  where each  $\mathbf{t}_i\in \mathbf{t}$  contains the probabilities of T5 tokens, making it trainable for the student models. To achieve this goal, we apply the algorithm shown in Algorithm 1.
First, we evaluate whether GPT produces the correct SQL by comparing  $\mathbf{p}$  with the gold SQL  $\mathbf{g}$ . If not, we expand  $\mathbf{g}$  as a sequence of one-hot distributions for student's training. Otherwise, we tokenize the GPT predicted SQL query using the T5 tokenizer and get a sequence of one-hot distributions  $\mathbf{q}_{1:K}$  made up of  $K$  T5 tokens with probability 1. And our target is to align  $\mathbf{s}$  and  $\mathbf{q}$ . Here we apply an one-to-one strategy. If there exists an one-to-one mapping of a T5 token  $\mathbf{q}_i$  and a GPT token  $\mathbf{p}_j$ , we use the GPT distribution  $\mathbf{s}_j$  as the T5 distribution. In other cases where a mismatch exists, we simply take the one-hot distribution  $\mathbf{q}_i$  as the label.
Algorithm 1 Tokenizer alignment of GPT and T5
Require: GPT predicted SQL query  $\mathbf{p}_{1:L}$ , GPT output distributions  $s_{1:L}$ , gold SQL query g, T5 Tokenizer T. initialize t = [] function FINDONETOONEMAPPING(qi, p, s) ifExists an one-to-one mapping of qi and pj then return sj else return None end if end function if p is equivalent to g then  $\mathbf{q}_{1:K} = \mathbf{T}.\mathrm{TOKENIZE}(\mathbf{p})$  for  $i := 1$  to  $K$  do if FINDONETOONEMAPPING(qi, p, s) is not None then t.APPEND(FINDONETOONEMAPPING(qi, p, s)) else t.APPEND(qi) end if end for else t = g end if return t