SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yongrui Chen $^{1,2,*}$  Shenyu Zhang $^{1,2,*}$  Guilin Qi $^{1,2,\dagger}$  Xinnan Guo $^{1,2}$
$^{1}$ School of Computer Science and Engineering, Southeast University, Nanjing 211189, China  
 $^{2}$ Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China {yrchen, shenyuzhang, gqi}@seu.edu.cn, guoxinnan0727@163.com
Continual table semantic parsing aims to train a parser on a sequence of tasks, where each task requires the parser to translate natural language into SQL based on task-specific tables but only offers limited training examples. Conventional methods tend to suffer from overfitting with limited supervision, as well as catastrophic forgetting due to parameter updates. Despite recent advancements that partially alleviate these issues through semi-supervised data augmentation and retention of a few past examples, the performance is still limited by the volume of unsupervised data and stored examples. To overcome these challenges, this paper introduces a novel method integrating parameter-efficient fine-tuning (PEFT) and in-context tuning (ICT) for training a continual table semantic parser. Initially, we present a task-adaptive PEFT framework capable of fully circumventing catastrophic forgetting, which is achieved by freezing the pre-trained model backbone and fine-tuning small-scale prompts. Building on this, we propose a teacher-student framework-based solution. The teacher addresses the few-shot problem using ICT, which procures contextual information by demonstrating a few training examples. In turn, the student leverages the proposed PEFT framework to learn from the teacher's output distribution, then compresses and saves the contextual information to the prompts subsequently, eliminating the need to store any training examples. Experimental evaluations on two benchmarks affirm the superiority of our method over prevalent few-shot and continual learning baselines across various metrics.
Tabular data is a crucial source of information for human decision-makers in various domains, including healthcare [1], retail [2] and finance [3]. Although structured query language (SQL) provides an efficient means of accessing this data, a natural language interface would make it more accessible to non-technical users. Consequently, there has been a growing interest in Table Semantic Parsing (TSP), which involves mapping natural language queries over tabular data to formal programs.
Current research [4, 5, 6] on TSP have covered many scenarios, most of which assume the datasets are static. In contrast, recent work [7] highlights the fact that new tables are often constantly emerging in response to changing circumstances, leading to a continual stream of TSP tasks. With this comes two new challenges: a) few-shot problem [8, 9, 10]: For a new task against unseen tables, the annotated data available in a short period is typically limited, causing the parser to be prone to over-fitting. b)
Figure 1: Processes for handling TSP task stream using PEFT (upper) and ICT (bottom).  $\diamond$  and  $\star$  represent tuning and freezing parameters, respectively.  $\times$  means that the previous data is not visible.
catastrophic forgetting [11, 7]: After training on a new task, the performance of the parser on the previous task may plummet attributed to parameter updates. While [7] mitigates these challenges using semi-supervised learning and replaying past examples, however, its performance is still limited by the amount of unsupervised data and replayed examples. More realistically, in some scenarios involving data privacy and timeliness, such as patient cases and transaction records, both unsupervised data and past examples may not be available in the following tasks.
Fortunately, recent research yields new ideas to tackle the issues. a) in-context tuning (ICT) [12, 13, 14] enhances models' few-shot learning capability using demonstrations containing a few training examples. b) parameter effective fine-tuning (PEFT) [15, 16, 17] fundamentally eliminates catastrophic forgetting by avoiding parameter updates of the pre-trained model (PLM) during the training stage and instead tuning a group of parameters of bearable size individually for each task. Nevertheless, the two solutions can merely handle the corresponding challenge while still suffering from the other. As is illustrated in Figure 1: for ICT, the invisibility of past demonstrations causes catastrophic forgetting when the model experiences a new task; for PEFT, though proposed to improve generalization, it still suffers from severe overfitting as only a few training examples are available for each new task.
In this paper, we propose a new method to simultaneously tackle both of these challenges by integrating PEFT with ICT. We start with a task-adaptive parameter-efficient continual TSP framework, which freezes the backbone PLM and only tunes the additional prompt embeddings for each distinct task. Unlike existing methods [16, 17], our backbone PLM undergoes fine-tuning on the initial task to facilitate adaptation to the TSP task format. Based on this foundation, we propose a Context-Compressed Continual table semantic parser (C3). C3 employs a teacher-student framework, comprising two parsers that symbolize the TEACHER and the STUDENT, respectively. For each task, the TEACHER utilizes ICT to glean contextual information from semantically akin demonstrations extracted from the training set, thereby mitigating the risk of overfitting on the few-shot data. Subsequently, the STUDENT adopts our proposed parameter-efficient framework to fine-tune the prompt embeddings and learns the output distribution of the TEACHER for each training example. This method ensures that the contextual information gleaned by the TEACHER can be preserved and compacted into a parameterized form, which can be reloaded later even when the demonstrations of past tasks are no longer available. Extensive experiments performed on two TSP task streams substantiate the effectiveness of our method, which surpasses all compared baselines, achieving state-of-the-art results across multiple metrics. In summary, the contributions of this paper include:
- We introduce a task-adaptive, parameter-efficient continual framework for table semantic parsing. It can completely avert catastrophic forgetting by tuning only  $0.05\%$  of the original PLM parameters.
- We propose to utilize the teacher-student framework to fuse PEFT and ICT so that the table semantic parser can benefit from both their advantages in solving few-shot data and catastrophic forgetting. To the best of our knowledge, this is the first time that the two technologies have been integrated into the scenario of the task streams.
- We create two task streams derived from two mainstream table semantic parsing benchmarks, and carry out extensive experiments using these task streams. The results demonstrate that our method surpasses existing competitors, achieving state-of-the-art performance across multiple metrics.
2.1 Table Semantic Parsing
Formally, given a natural language question  $\mathcal{Q}$  and a schema  $S = (\mathcal{C},\mathcal{T})$  of multiple tables, the goal of TSP is to generate a SQL query  $\mathcal{Y} = \mathcal{F}_{\theta}(\mathcal{Q},\mathcal{S})$ , where  $\mathcal{F}_{\theta}$  denotes a neural semantic parser with parameters  $\theta$ .  $\mathcal{T} = \{t_1,\dots ,t_{|\mathcal{T}|}\}$  and  $\mathcal{C} = \{c_1,\dots ,c_{|\mathcal{C}|}\}$  are the sets of table names and column names, respectively, where each column  $c_{i}\in \mathcal{C}$  belongs to only one table  $t_j\in \mathcal{T}$ .
In this paper, we focus on TSP in this paper for the following reasons: a) TSP is important for enhancing various fields through its role in Natural Language Interfaces to Databases (NLIDB). b) TSP is a challenging task requiring comprehensive understanding and mapping of queries to structured tables, presenting a few-shot learning challenge due to the difficulty in obtaining supervised data. c) TSP provides objective evaluation metrics, as minor errors in SQL queries can lead to significant outcome discrepancies, thereby offering a true reflection of our method's performance.
2.2 Problem Formulation
In this paper, we assume that  $\mathcal{F}_{\theta}$  is no longer limited to fixed training and testing data, but has to face a continual stream of TSP tasks, each corresponding to a different set of tables. Formally,  $\mathcal{F}_{\theta}$  is sequentially trained on a stream of  $K$  tasks  $\{\mathcal{D}^0,\mathcal{D}^1,\ldots ,\mathcal{D}^{K - 1}\}$ . Each task  $\mathcal{D}^i$  consists of a training set  $\mathcal{D}_{\mathrm{train}}^i$ , a validation set  $\mathcal{D}_{\mathrm{valid}}^i$ , and a test set  $\mathcal{D}_{\mathrm{test}}^i$ . Each example  $e = (\mathcal{X},\mathcal{Y})$ , where  $\mathcal{X} = (\mathcal{Q},\mathcal{S})$  is the input and  $\mathcal{V}$  denotes the target SQL. For  $\forall \mathcal{D}^i,\forall \mathcal{D}^j$ , if  $i\neq j$ , then  $\mathcal{S}(\mathcal{D}^i)\cap \mathcal{S}(\mathcal{D}^j) = \emptyset$ , where  $\mathcal{S}(\mathcal{D}^i)$  denotes the total set of corresponding table schemas in  $\mathcal{D}^i$ . Here  $\mathcal{S}$  also serves as a task identifier for each  $e$ , enabling efficient indexing of the corresponding task. Our ultimate objective is to train a  $\mathcal{F}_{\theta}$  that can achieve good accuracy on each  $\mathcal{D}_{\mathrm{test}}^i$  after experiencing all  $K$  tasks sequentially.
3 Parameter-Efficient Continual Table Semantic Parsing
We start with a vanilla parameter-efficient continual TSP framework, which trains a backbone semantic parser with a task-adaptive continual prompt-tuning method.
3.1 Backbone Semantic Parser
The pre-trained Text-to-Text Transfer Transformer (T5) [18] is selected as our backbone parser  $\mathcal{F}_{\theta}$  for two primary reasons. Firstly, prior studies [16] have shown that the combination of T5 with PEFT methods can result in significant performance gains across various NLP tasks. Secondly, T5 has demonstrated robust capabilities [19] in generating logical forms on a widely-used TSP dataset Spider [5]. In particular, each input pair  $\mathcal{X} = (\mathcal{Q},\mathcal{S})$  is flattened into a plain text,
$$
\mathcal {X} ^ {\prime} = t _ {1}: c _ {1} ^ {t _ {1}}, c _ {2} ^ {t _ {1}}, \dots , c _ {m _ {1}} ^ {t _ {1}}; t _ {2}: c _ {1} ^ {t _ {2}}, c _ {2} ^ {t _ {2}}, \dots , c _ {m _ {2}} ^ {t _ {2}}; \dots | \mathcal {Q}, \tag {1}
$$
where  $c_{j}^{t_{i}}$  denotes the  $j$ -th column name of the  $i$ -th table, and " $: ", " , "$ , and " $|"$  are predefined separators. Subsequently,  $\mathcal{X}'$  is fed to the parser  $\mathcal{F}_{\theta}$  and the generative probability of a predicted SQL  $\tilde{\mathcal{V}}$  is estimated by  $P(\tilde{\mathcal{Y}}|\mathcal{X}',\theta) = \prod_{j=1}^{|\tilde{\mathcal{Y}}|}P(\tilde{y}_j|\mathbf{X}',\tilde{y}_{<j},\theta)$ , where  $y_{j}\in \mathcal{V}$  denotes the  $j$ -th token of  $\tilde{\mathcal{V}}$  and  $\mathcal{V}$  is the vocabulary of the T5 tokenizer.  $\mathbf{X}'$  are the hidden states of  $\mathcal{X}'$  obtained by the T5 encoder, and  $P(\tilde{y}_j|\mathbf{X}',\tilde{y}_{<j},\theta)$  denotes the output probability of  $\tilde{y}_j$  from the T5 decoder.
3.2 Task Adaptive Continual Prompt Tuning
Prompt tuning [20], as a representative of PEFT-based methods, has been widely applied in recent NLP tasks [16, 17] owing to its simplicity. Inspired by these works, we employ prompt tuning to conduct the PEFT training framework for  $\mathcal{F}_{\theta}$ . Formally, each input  $\tilde{\mathcal{X}}$  in task  $\mathcal{D}^k$  is prefixed with  $M$  special prompt tokens  $\mathcal{P}^k = p_1^k p_2^k \ldots p_M^k$ . During training, only the embeddings of these tokens are updated, rather than the parameters of the entire T5 model. More specifically, the training process can be divided into the following two phases.
Figure 2: Illustration of the proposed C3 table semantic parser.
3.2.1 Initial Task Adaptation
Intuitively, the original T5 is not well-suited to TSP, as it is not exposed to any logical form during pre-training and is never tasked with predicting structured outputs. While this limitation can be readily addressed through fine-tuning, we hypothesize that tuning prompts alone may not be sufficient.
Therefore, we first fine-tune T5 with  $\mathcal{D}_{\mathrm{train}}^0$  to obtain a set of task-adaptive parameters  $\theta$  that will be frozen on  $\mathcal{D}^1,\ldots ,\mathcal{D}^{K - 1}$ . Notably, here the prompt embeddings are tuned as well as  $\theta$  because we expect to acquire a good initialization for both of them. In particular, for each  $e = (\mathcal{X},\mathcal{Y})\in \mathcal{D}_{\mathrm{train}}^0$ , the following loss is optimized using the AdaFactor [21] algorithm,
$$
\mathcal {L} (e; \theta , \mathbf {P} ^ {0}) = - \log P (\mathcal {Y} | \mathcal {X} ^ {\prime}, \theta) = - \sum_ {j = 1} ^ {| \mathcal {Y} |} \log P \left(y _ {j} \mid \left[ \bar {\mathbf {P}} ^ {0}; \mathbf {X} ^ {\prime} \right], y _ {<   j}, \theta\right), \tag {2}
$$
where  $[\cdot ]$  represents the concatenation of matrices,  $\mathbf{P}^0\in \mathbb{R}^{M\times d}$  denotes the tunable embeddings of  $\mathcal{P}^0$ , and  $\bar{\mathbf{P}}^0\in \mathbb{R}^{M\times d}$  represents its hidden states from the T5 encoder. In our experiments,  $\mathbf{P}^0$  is randomly initialized. After multiple iterations, the pair of  $\theta$  and  $\mathbf{P}^0$  with the highest validation accuracy, denoted by  $(\theta^{*},\mathbf{P}^{*})$ , is selected for initializing the subsequent  $K - 1$  tasks.
3.2.2 Continual Prompt Tuning
Unlike existing methods [22], for  $\mathcal{D}^i$ $(0 < i < K)$ , we initialize all  $\mathbf{P}^i$  with  $\mathbf{P}^*$  instead of  $\mathbf{P}^{i-1}$  because our experimental results (detail in Section 5.3.2) reveal that the latter does not yield significant performance improvements. Furthermore, this strategy enhances the flexibility of the proposed framework by eliminating the dependency on the order of subsequent tasks, as all prompts share a common initialization. During training,  $\mathbf{P}^i$  is updated with the gradient  $\nabla_{\mathbf{P}^i}\sum_{e\in \mathcal{D}_{\mathrm{train}}^i}\mathcal{L}(e;\theta^*,\mathbf{P}^i)$  using the AdaFactor optimizer, where  $\mathcal{L}(e;\theta^*,\mathbf{P}^i)$  is calculated by Equ. (2). We experimentally set the prompt dimension  $d = 512$  and the prompt number  $M = 150$ , leading to a total of only 0.7 million parameters. This size is negligible (0.05%) compared to the entire T5-BASE of 220 million parameters. Consequently, the cost of storing all  $\mathbf{P}^1,\ldots,\mathbf{P}^{K-1}$  is usually acceptable. During inference, given a test example  $e_{\mathrm{test}}^i\in \mathcal{D}_{\mathrm{test}}^{0:K-1}$ , its task index  $i$  can be first identified using its table schema  $S$ . Then,  $\mathcal{F}$  can solve it by loading the saved  $(\mathbf{P}^i,\theta^*)$  without any forgetting.
In theory, our framework can also be expanded to the tasks whose tables have not been seen in  $\mathcal{D}^0,\ldots ,\mathcal{D}^{K - 1}$ . The prediction can be made using simple heuristics, such as averaging all  $\mathbf{P}^i$ . Since this scenario involves the zero-shot learning problem, we leave it for future work.
4 C3 Semantic Parser
Figure 2 depicts the illustration of our proposed Context-Compressed Continual table semantic parser (C3). C3 is typically composed of two parsers (e.g., T5) of identical architecture (optional), namely TEACHER  $\mathcal{F}_{\mathrm{tea}}$  and STUDENT  $\mathcal{F}_{\mathrm{stu}}$ , and each possesses its own individual parameters during training.
4.1 Context-Enhanced Teacher Parser
The sole purpose of  $\mathcal{F}_{\mathrm{tea}}$  is to gain the few-shot learning capability and thereby in-context tuning (ICT) is employed to train  $\mathcal{F}_{\mathrm{tea}}$ . Note that  $\mathcal{F}_{\mathrm{tea}}$  only focuses on each current task  $\mathcal{D}^i$  and does not care if the past tasks are forgotten. Thus, it is sequentially trained on  $\mathcal{D}^0, \ldots, \mathcal{D}^{K-1}$  without any continual
learning. During  $\mathcal{D}^i$ , the parameters of  $\mathcal{F}_{\mathrm{tea}}$ , denoted by  $\theta_{\mathrm{tea}}^i$ , is initialized with  $\theta_{\mathrm{tea}}^{i - 1}$ . Afterwards,  $\theta_{\mathrm{tea}}^i$  is updated by optimizing  $\mathcal{L}(x;\theta_{\mathrm{tea}}) = -\sum_{j = 1}^{|\tilde{\mathcal{Y}} |}\log P(\tilde{y}_j|\mathbf{C},y_{< j},\theta_{\mathrm{tea}})$ . Here  $\mathbf{C}$  denotes the hidden states of the input text,  $\mathcal{C}$ , which is constructed by retrieving and formatting the demonstrations.
4.1.1 Demonstration Retrieval
The key to ICT lies in identifying the suitable demonstration that effectively facilitates the learning process of the model within the appropriate context. Recent studies [23, 24] have reached a consensus that the closer the demonstration aligns with the example being queried, the greater the benefits of ICT. Thus, for each  $e$ , we retrieve the demonstration by comparing its semantic similarity to the candidate demonstrations. To maintain overall uniformity, we select the original T5 checkpoint which was fine-tuned on the Semantic Textual Similarity Benchmark (STS-B) [25], as an off-the-shelf retriever  $\mathcal{R}$ . Specifically, for example  $e$  to be queried and a candidate demonstration  $e_j' \in \mathcal{D}_{\text{train}}^i \setminus \{e\}$ , we first wrap them into a template  $T(e, e_j') = \text{stsb sentence1}: \mathcal{Q}$ , sentence2:  $\mathcal{Q}_j'$ , where  $\mathcal{Q}$  and  $\mathcal{Q}_j'$  represent the natural language questions of  $e$  and  $e_j'$ , respectively. Here the target SQL  $\mathcal{Y}$  is not considered because it is invisible in the testing stage. Subsequently, the similarity score of  $e_j'$ , denoted by  $\sigma(e_j')$ , is calculated by  $\sigma(e_j') = \mathcal{R}(T(e, e_j')) \in [0, 5]$ . To circumvent the noise, we set a threshold  $\eta$  and eventually select the top- $r$ $e_j' \in \mathcal{D}_{\text{train}}^i \setminus \{e\}$  whose  $\sigma(e_j') \geq \eta$  as the demonstration of  $e$ . In fact, not every example  $e \in \mathcal{D}_{\text{train}}^i$  has sufficient demonstrations. To retain  $\mathcal{F}_{\text{tea}}^i$ 's ability in handling no-demonstration examples, we apply a simple but effective context mixing strategy during training: both  $\mathcal{C}$  and  $\mathcal{X}'$  are engaged in the training for each  $e \in \mathcal{D}_{\text{train}}^i$  if it has  $r$  demonstrations, otherwise only  $\mathcal{X}'$  is engaged.
4.1.2 Demonstration Formatting
$r$  retrieved demonstrations  $e_1, \ldots, e_r$  compose the input  $\mathcal{C} = \mathcal{Q}_1|\mathcal{Y}_1| \ldots |\mathcal{Q}_r|\mathcal{Y}_r||\mathcal{X}'$ , where both  $|$  and  $||$  are separators, and  $\mathcal{X}'$  is obtained by Equ. (1). The decision to exclude table schemas  $\mathcal{S}$  from  $\mathcal{C}$  is motivated by one main consideration: the selected demonstration examples and the example  $e$  to be queried are likely to be associated with the same batch of tables due to their similarity. Therefore, the explicit information regarding schemas need not be provided repeatedly outside of  $\mathcal{X}'$ , which helps to streamline the input to improve efficiency.
4.2 Context-Compressed Student Parser
$\mathcal{F}_{\mathrm{stu}}^i$  follows the PEFT framework introduced in Section 3 and is dedicated to compressing and preserving the few-shot capability  $\mathcal{F}_{\mathrm{tea}}^i$  learns from the demonstration into the prompt embeddings. In this way,  $\mathcal{F}_{\mathrm{stu}}^i$  can reproduce the few-shot performance by loading the saved prompt embedding, without accessing any examples or demonstrations of previous tasks. Inspired by [26], We hypothesize that this goal could be accomplished as much as possible by having  $\mathcal{F}_{\mathrm{stu}}^i$  learn the  $\mathcal{F}_{\mathrm{tea}}^i$  's output distribution. More concretely, we minimize the KL divergence between  $\mathcal{F}_{\mathrm{stu}}^i$  's per-step autoregressive distribution and the  $\mathcal{F}_{\mathrm{tea}}^i$  's, denoted by  $D_{\mathrm{KL}}(P(\theta_{\mathrm{tea}}) \parallel P(\theta_{\mathrm{stu}}))$  where  $P(\theta) = P(y_j | \mathbf{C}, y_{<j}, \theta)$ . During training, the teacher-forcing strategy is applied to ensure that the inputs to the decoders of  $\mathcal{F}_{\mathrm{tea}}^i$  and  $\mathcal{F}_{\mathrm{stu}}^i$  are consistent at each decoding step. In addition, to guarantee the quality of supervision, we directly use the gold SQL  $\mathcal{V}$  instead of  $P(\theta_{\mathrm{tea}})$  if  $\mathcal{F}_{\mathrm{tea}}^i$  's prediction for a given input  $\mathcal{X}$  is incorrect.
5.1.1 Datasets & Evaluation Metrics
Our experiments involve two widely-used TSP datasets: WikiSQL [4] is a dataset for single-table semantic parsing in which each SQL query corresponds to a single table and adheres to a simplified SQL syntax. Spider [5] is a more challenged dataset in which each SQL requires the JOIN operation of multiple tables and contains complicated syntax including GROUP BY even nested query.
As explicated in Section 2.2, we constructed two task streams based on the above datasets. The first one, Spider-Stream, is established by merging the Spider training and validation sets and splitting
Figure 3: Statistics of the task splits of Spider-Stream (left) and Combined-Stream (right).
them into 16 tasks according to the domain of the tables. It is used to reflect the impact of the changing table domains. The second one, Combined-Stream, is a merged task stream encompassing 11 tasks. The first 8 tasks are randomly selected from the Spider-Stream and encompass complex SQL syntax, whereas the last 3 tasks are collected from WikiSQL and involve only simple SQL syntax. It is utilized to assess the dual impact of table domains and target SQL structures.
To highlight the few-shot challenge, the majority of tasks  $\mathcal{D}_{\mathrm{train}}^i$  have  $|\mathcal{D}_{\mathrm{train}}^i| < 500$ . For each method, we evaluated its performance in three random task orders, to eliminate any potential influence of specific tasks. Considering that in real applications, models are usually fully trained in certain domains before facing the task stream, we combined the first  $N$  (6 for Spider-Stream and 5 for Combined-Stream) tasks of each task order into a single task. This practice enables all methods to have full adaptation to the TSP task format. In addition, for Combind-Stream, to aggravate the gap between adjacent tasks, for any task order, we always make the tasks of the even index  $(0,2,\ldots)$  from Spider and the tasks of the odd index  $(1,3,\ldots)$  from WikiSQL.
Following [27, 11], we adopted four metrics: 1) Task Accuracy:  $\mathrm{TA} = \frac{1}{K}\sum_{i=0}^{K-1}a_{i,K-1}$ ; 2) Example Accuracy:  $\mathrm{EA} = a_{\mathcal{D}_{\mathrm{test}}^{(0;K-1)}}$ ; 3) Initial Accuracy:  $\mathrm{IA} = \frac{1}{K}\sum_{i=0}^{K-1}a_{i,i}$ ; 4) Memory Decay:  $\mathrm{MD} = \frac{1}{K-1}\sum_{i=0}^{K-2}a_{i,K-1} - a_{i,i}$ ; where  $a_{i,j}$  denotes the accuracy on  $\mathcal{D}_{\mathrm{test}}^i$  after the training of  $\mathcal{D}^j$ . TA and EA reflect the overall performance at the task level and example level, respectively. MD measures the extent of forgetting. IA evaluates the performance when the parser is trained on a new task for the first time, considering no forgetting.
5.1.2 Compared Methods & Implementation Details