Long-Tailed Question Answering in an Open World
Year: 2023

Abstract:
Real-world data often have an open long-tailed distribution, and building a unified QA model supporting various tasks is vital for practical QA applications. However, it is non-trivial to extend previous QA approaches since they either require access to seen tasks of adequate samples or do not explicitly model samples from unseen tasks. In this paper, we define Open Long-Tailed QA (OLTQA) as learning from long-tailed distributed data and optimizing performance over seen and unseen QA tasks. We propose an OLTQA model that encourages knowledge sharing between head, tail and unseen tasks, and explicitly mines knowledge from a large pre-trained language model (LM). Specifically, we organize our model through a pool of fine-grained components and dynamically combine these components for an input to facilitate knowledge sharing. A retrieve-then-rerank frame is further introduced to select in-context examples, which guild the LM to generate text that express knowledge for QA tasks. Moreover, a twostage training approach is introduced to pretrain the framework by knowledge distillation (KD) from the LM and then jointly train the frame and a QA model through an adaptive mutual KD method. On a large-scale OLTQA dataset we curate from 43 existing QA datasets, our model consistently outperforms the stateof-the-art. We release the code and data at https://github.com/AlibabaResearch/ DAMO-ConvAI/tree/main/oltqa.

heading: 1 Introduction
text: Real-world data often have a long-tailed and openended distribution (Liu et al., 2019b). As a cornerstone for AI applications (Yang et al., 2019), Question Answering (QA) is widely investigated to tackle various QA tasks involving diverse formats
∗ Work done while the author was interning at Alibaba. † Equal contribution. ‡ Corresponding author.
and domains (Khashabi et al., 2020b; Zhong et al., 2022a). The frequency distribution of QA tasks in our daily life is long-tailed (Reed, 2001), with a few head tasks of adequate samples and many more tail tasks of limited samples, and we continuously encounter new tasks that are not seen during training in an open world.
We formally study Open Long-Tailed QA (OLTQA) emerging in natural data settings. A practical QA system shall learn from long-tailed distributed data, i.e., a few head tasks and many tail tasks, and it is expected to perform well over a balanced test set which include head, tail, and unseen tasks.
OLTQA must handle not only few-shot learning for tail tasks in the closed world (Shu et al., 2017), but also zero-shot learning for unseen tasks in an open world (Scheirer et al., 2012) with one unified model. A major challenge for OLTQA is the lack of knowledge required for the language understanding and reasoning abilities of QA tasks, especially under such low resource conditions (Yan et al., 2020). Therefore, it is important for an OLTQA model to share knowledge between head, tail, and unseen QA tasks (Zaremoodi et al., 2018), and mine knowledge from external resources (Liu et al., 2022b).
However, it is non-trivial to directly extend previous methods to the OLTQA setting. Specifically, an effective implementation of knowledge sharing is the multi-task learning (MTL) approach (Liu et al., 2019a; Raffel et al., 2020), in which task-specific components are maintained to preserve learned knowledge (Aghajanyan et al., 2021; Karimi Mahabadi et al., 2021). As we constantly encounter new tasks in practice, it is challenging to directly apply MTL methods since they do not explicitly model samples from unseen tasks.
Another challenge is the absence of samples from unseen tasks in the training process, which leads to poor prior knowledge about unseen tasks. Fortunately, a large pre-trained language model
6362
(LM) embeds broad-coverage knowledge that can help a variety of tasks (Rubin et al., 2022). One key ingredient in LM knowledge mining is to select demonstrative in-context examples, which guild the LM to generate text that express knowledge for downstream tasks (Liu et al., 2022a). However, few studies have explored selecting in-context examples to directly optimize QA performance in the OLTQA setting.
In this study, we propose an OLTQA model to address challenges mentioned above for the OLTQA setting. Specifically, to encourage knowledge sharing between head and tail tasks while acknowledging the emergence of unseen tasks, we organize our model at the instance-level and use a dynamic architecture for each input (Wiwatcharakoses and Berrar, 2020), i.e., a pool of fine-grained components are maintained and dynamically combined in each forward pass based on the input (Wang et al., 2021). This scheme tackles unseen tasks, since the learned knowledge is distributed into different model components (Trauble et al., 2022).
We further mine knowledge from a large pretrained LM. Concretely, we employ a retrieve-thenrerank frame (Ren et al., 2021) to select demonstrative in-context examples for a test instance, which guide the LM to decode the output (Brown et al., 2020). The LM outputs are viewed as hints for QA tasks (Zhang and Wan, 2022) and leveraged for improving QA performance. The retrieve-thenrerank frame consists of an efficient retriever and an effective re-ranker (Zamani et al., 2022), which is optimized by a two-stage training approach. The first stage pre-trains the retrieve-then-rerank framework by knowledge distillation from a pre-trained LM (Izacard et al., 2022). The second stage jointly train the above framework and an encoder-decoder QA model through adaptive mutual knowledge distillation (Xie and Du, 2022) to allow information exchange between each other. Our key contributions are summarized as follows:
• We formally define the OLTQA task, which learns from natural long-tail distributed data and optimizes the performance over seen and unseen tasks. We curate a large OLTQA dataset according to a long-tail distribution from 43 existing representative QA datasets.
• We propose an OLTQA model, consisting of knowledge sharing and knowledge mining components to address challenges of OLTQA.
An instance-level knowledge sharing mechanism is introduced, and a retrieve-then-rerank frame is employed to mine knowledge from a large pre-trained LM through a novel twostage knowledge distillation training process.
• Our extensive experimentation on the OLTQA dataset demonstrates that our model consistently outperforms the state-of-the-art.

heading: 2 Related Work
text: Question Answering (QA) is important for advanced AI applications (Yang et al., 2019). Recent approaches try to build unified QA models by casting different QA tasks into a unified text-totext format (McCann et al., 2019; Khashabi et al., 2020b; Zhong et al., 2022a). Some works try to improve QA performance under the low-resource conditions (Yan et al., 2020; Van et al., 2021; Bai et al., 2022). Some approaches also attempt to solve the open-domain QA problem, aiming at answering general domain questions through an extensive collection of documents (Voorhees et al., 1999; Chen et al., 2017; Singh et al., 2021; Cheng et al., 2021). These approaches do not learn from natural long-tail distributed data.
Long-Tailed Learning focuses on long-tail distributed data (Liu et al., 2019b). Recent approaches for long-tailed learning include rebalancing (Zhang et al., 2021), information augmentation (He et al., 2021), and module improvement (Cui et al., 2021). In this study, we attempt to build a QA model from long-tail distributed data by knowledge sharing and knowledge mining.
Knowledge Mining from external resources is essential for building robust QA models (Pan et al., 2019). Wikipedia and knowledge bases are used to improve QA performance (Bi et al., 2019; Banerjee et al., 2019). Large pre-trained LMs store rich knowledge, which is used to solve various tasks via conditioned generation (Petroni et al., 2019). Recent approaches build prompt retrievers to select in-context examples from a training set to optimize LM generation performance (Rubin et al., 2022). However, these approaches cannot directly optimize our OLTQA model. In this study, we jointly train a retrieve-then-rerank framework and a QA model to enhance QA performance.
Knowledge distillation (KD) is often employed to learn a student model using the knowledge distilled from a teacher model by enforcing the agreement of outputs between the two models (Hin-
ton et al., 2015). Mutual KD helps a group of models mutually generate knowledge to train each other (Zhao and Han, 2021). Our OLTQA model jointly trains the retrieve-then-rerank frame and the QA model through adaptive mutual KD, encouraging them to collaborate with each other (Xie and Du, 2022).

heading: 3 Method

heading: 3.1 Problem Setup
text: In this study, we aim to learn from n QA tasks {T1, · · · , Tn}, in which training sets follow a longtailed Zipf distribution with power value α, i.e., a few head tasks of adequate samples and many tail tasks of limited samples. Each sample of Ti is a tuple of a context c, a question q, and an answer a: ⟨c, q,a⟩. Our QA model F is built to predict a based on c and q. We also consider a more challenging setting in an open world, i.e., model F needs to predict answers for unseen tasks. Therefore, we collect another ñ unseen tasks {Tn+1, · · · , Tn+ñ} that are only used for testing.

heading: 3.2 Overview
text: Our model tackles the open long-tailed QA problem by training a prompt-enhanced encoderdecoder QA model F on long-tailed distributed data. There are mainly two challenges to be addressed: (1) How to alleviate the low-resource problem and share knowledge between head, tail, and unseen tasks; (2) How to mine knowledge from external resources. These two issues are tackled with two key ingredients in our model (see Figure 1): 1. An instance-level knowledge sharing method (Section 3.3); 2. A knowledge mining method from a pre-trained language model (Section 3.4).
We follow previous approaches to serialize the context c, question q, and answer a into text sequences (Khashabi et al., 2020b; Zhong et al., 2022b). For each training sample ⟨c, q,a⟩, we first construct a prompt P based on c and q, and then the encoder takes in the concatenation of P , c, and q and the decoder predicts a, i.e., p(a|[P; c; q]), where [; ] denotes the sequence concatenation operation. Specifically, P is a concatenation of two kinds of prompts, i.e., a meta prompt Pm and a knowledge prompt Pk. To capture fine-grained knowledge distributed in each input sample, we maintain s meta prompts {P im}si=1 and dynamically combine these prompts based on c and q to obtain Pm (Wang et al., 2021). We associate a key
vector kim for each meta prompt P im, respectively. A fixed query function h is built to map c and q to a query vector x = h(c, q). h is initialized by a fixed pre-trained LM and not tuned in the training phase. Pm can be determined by retrieving the most similar key vectors kim using x. Note that Pm is a soft prompt, i.e., a sequence of trainable embeddings that is randomly initialized and optimized when training QA model F (Liu et al., 2021).
We also mine knowledge from a large pre-trained LM g to construct knowledge prompt Pk. Liu et al. (2022a) showed that the efficacy of output generated by an LM could vary widely depending on the choice of in-context examples. In this study, we introduce a retrieve-then-rerank framework ⟨R1, R2⟩ (Ren et al., 2021) to select incontext examples from a training set Dtr, consisting of a retriever R1 and a re-ranker R2 (Zamani et al., 2022). The retriever R1 is implemented as an efficient dual-encoder (Xiong et al., 2021). The re-ranker R2 is built as a more effective crossencoder (Luan et al., 2021). For a test instance ⟨c, q⟩, we mine knowledge following three steps: 1. R1 retrieves a subset of l candidate examples {ei = ⟨ci, qi,ai⟩}li=1 from training set Dtr; 2. LM g produces a text hi for each example ei by conditional generation pg(hi|[ei; c; q]), which can serve as a hint for the test instance; 3. R2 further select top l̃ hints {hi}l̃i=1 to obtain the knowledge prompt Pk (l̃ ≪ l), in which the scoring function measures the similarity between ⟨c, q⟩ and ⟨ei,hi⟩. Note that Pk is a hard prompt (Jiang et al., 2020), which is a concatenation of texts in {hi}l̃i=1.

heading: 3.3 Instance-level Knowledge Sharing
text: To facilitate knowledge sharing between head, tail, and unseen tasks at the instance level, we maintain a pool of prompts and optimize key vectors assigned to these prompts. Specifically, for each input ⟨c, q⟩, we select s̃ prompt keys that are closest to the query vector x = h(c, q) and concatenate these s̃ associated meta prompts to obtain Pm. Intuitively, the knowledge associated with the input sample is distributed in these s̃ meta prompts.
When learning meta prompt keys, we assume the distribution of these keys should balance diversity and locality. Concretely, meta prompts are expected to distribute to the whole vector space so that every meta prompt can be involved in the training process, while similar prompt keys are grouped into clusters so that the knowledge of each sample
can be better shared. We propose the following loss to enforce the above two properties:
Lm = E ⟨c,q,a⟩∈Dtr
( ∑
i∈S(x) max(0, ||kim,x|| − η)
+ ∑
i,j∈S(x) max(0, γ − ||kim,kjm||)/s̃2),
(1) where the operator ||·, ·|| determines the distance between two input vectors (here we use cosine distance), Dtr is the training set of all seen tasks, S(x) is the index set of s̃ selected meta prompt keys that are closest to x, η and γ are scalar hyperparameters to control the distance margin. Specifically, the first term in the above equation pulls these selected meta prompt keys around the query vector. The second term pushes these keys away from each other to occupy the whole vector space.

heading: 3.4 Pre-trained LM Knowledge Mining
text: To further enhance QA performance, we also mine knowledge from a large pre-trained LM g. We employ a retrieve-then-rerank framework ⟨R1, R2⟩ to retrieve in-context examples from a training set Dtr and further select hints for the test instance that are generated by LM g. We propose a two-stage knowledge distillation method to jointly train the framework ⟨R1, R2⟩ and QA model F .
Stage I. We pre-train R1 and R2 by knowledge distillation from a pre-trained LM g, inspired by
Rubin et al. (2022). We first construct a set of c candidate examples {ei = ⟨ci, qi,ai⟩}ci=1 for a traning instance ⟨c, q,a⟩ with BM25 (Robertson et al., 2009) . Then, we score each candidate example ei and calculate a distribution of candidate examples by applying the Softmax operator over the resulting scores, based on scoring functions of LM g, R1, and R2, respectively. Specifically, the distribution for the LM g scoring function is:
plm(ek) = exp(log(pg(a|[ek; c; q])))∑c i=1 exp(log(pg(a|[ei; c; q]))) ,
where pg(a|[ek; c; q]) is the score for candidate ek, which is the probability under LM g of output sequence conditioned on the candidate example and the training instance. In a similar manner, we calculate distributions pr1 and pr2 based on scoring functions of R1 and R2, respectively. We optimize R1 and R2 by minimizing KL-divergence of plm from pr1 and pr2 (Izacard et al., 2022):
Llm = E ⟨c,q,a⟩∈Dtr (KL(⊣ [plm]∥pr1)
+ KL(⊣ [plm]∥pr2)), (2)
where ⊣ [·] is a stopgrad operator that sets the gradient of its operand to zero.
Stage II. We jointly train ⟨R1, R2⟩ and the QA model F . For each training sample ⟨c, q,a⟩, we
first construct prompt Pm and Pk, and then optimize the encoder-decoder QA model F together with Pm using the following loss:
Lf = E ⟨c,q,a⟩∈Dtr (− log pF (a|[Pm;Pk; c; q])). (3)
To allow information exchange and encourage agreement between ⟨R1, R2⟩ and QA model F , mutual knowledge distillation is introduced to refine R1, R2, and F by knowledge distillation from each other (Zhao and Han, 2021). However, in this case, a worse-performing model is allowed to generate knowledge to train a better-performing model, which may lead to collective failures (Xie and Du, 2022). Therefore, we propose an adaptive mutual knowledge distillation method to allow a model to generate knowledge for training another model only if it performs better.
Therefore, we evaluate the performance of R1, R2, and F on a validation set Dval before mutual knowledge distillation. Specifically, we select top l̃ hints {hi}l̃i=1 from the c candidate examples {ei}ci=1 of a validation instance ⟨c, q,a⟩ based on scoring functions of R1, R2, F , and then obtain knowledge prompt Pr1k , Pr2k and P f k , respectively. The scoring function of QA model F is pF (a|[Pm;hi; c; q]), where hi is a hint for example ei and acts as a pseudo knowledge prompt. We evaluate R1, R2, and F as follows:
vi = E ⟨c,q,a⟩∈Dval log pF (a|[Pm;P ik; c; q]), (4)
where i ∈ {r1, r2, f} denotes a specific model. Lastly, we optimize the adaptive mutual knowledge distillation loss as follows:
Lmkd = E ⟨c,q,a⟩∈Dtr
∑
i,j∈{r1,r2,f} KL(⊣ [pi]∥pj) · I(vi > vj),
(5)
where pf is the distribution of candidate examples based on the scoring function of QA model F .
The whole training process of our model is summarized in Algorithm 1.

heading: 4 Experiments

heading: 4.1 Datasets
text: We curate an open long-tailed question answering benchmark from 43 existing representative QA datasets (Khashabi et al., 2022) covering four QA formats (Extractive QA, Abstractive QA, Multiplechoice QA, and Yes/No QA). See Appendix A for
Algorithm 1: The training process Input: Training data Dtr, validation data
Dval. Output: QA model F , meta prompts
{P im}si=1, prompt keys {kim}si=1, framework ⟨R1, R2⟩.
more details of the datasets. We regard each dataset as an individual QA task and reserve ñ = 22 as unseen tasks. Our model is trained on the rest of n = 21 seen tasks while tested on all 43 tasks. We down-sample the training sets of all seen tasks following a Zipf distribution with power value α = 2.0 to construct the training data for our model. Figure 2 shows the training data statistics.

heading: 4.2 Metrics
text: The evaluation metric of each above task follows Khashabi et al. (2022) (see more details in Appendix A). We calculate the average performances over 21 seen tasks (Aseen) and 22 unseen tasks (Aunseen) to evaluate the QA performance. We also calculate the average scores over a subset of seen tasks with m largest training sets (Head@m) and n smallest training sets (Tail@n) to evaluate the performance of head and tail tasks, respectively.

heading: 4.3 Implementation Details
text: We use T5-base (Raffel et al., 2020) to initialize the QA model F . For knowledge sharing, we maintain totally s = 30 meta prompts, and set the length of each meta prompt to 10. We adopt a fixed T5-base encoder with an average pooling layer to generate the query vector. For each instance, we select s̃ = 5 meta prompts to construct Pm. We set η = 0.15 and γ = 0.3 in Eq. 1. For knowledge mining, we use a dual-encoder as retriever, and a cross-encoder as re-ranker. Encoders in the retriever and the re-ranker are all initialized with Bert-base-uncased (Devlin et al., 2019). We use GLM-10B (Du et al., 2022) with 10B parameters as pre-trained LM g. For each instance, the retriever first selects l = 64 examples from the training dataset, and the re-ranker selects l̃ = 4 examples to construct Pk. All hyper-parameters are tuned according to the average score on the validation set. All results reported in our paper are averages of 3 runs with different random seeds. We use the AdamW (Loshchilov and Hutter, 2017) optimizer with a learning rate of 1e-4 and batch size of 32. Our model is trained for five epochs. All experiments are performed on 8 A100 GPUs. See Appendix D for more implementation details.

heading: 4.4 Baselines
text: We use the following competitive baselines: 1. UnifiedQA: (Khashabi et al., 2020b) casts different QA tasks into a unified text-to-text format and builds a single model for all QA tasks; 2. ProQA: (Zhong et al., 2022a) uses structural prompts to train a unified QA model with a QA-centric pre-training; 3. Muppet: (Aghajanyan et al., 2021) maintains taskspecific heads and learns QA tasks through multitask learning; 4. Hyperformer++: (Karimi Mahabadi et al., 2021) uses a hyper-network to generate task-specific adapters for multi-task learning; 5. EPR: (Rubin et al., 2022) propose an efficient method to retrieve in-context examples for a test instance and use a pre-trained LM to directly decode the output based on the examples. Note that “Muppet” and “Hyperformer++” have no specific modules for unseen tasks. Thus, we select a task with the lowest perplexity across all seen tasks for an input from unseen tasks in the testing phase, following Madotto et al. (2021).

heading: 4.5 Main Results
text: Table 1 shows the result on seen tasks. Our model outperforms all competitive baselines in terms of Head@3, Tail@4, Aseen, and achieves SOTA results on all head and tail tasks. We can observe that: 1. Our model achieves an even larger performance improvement for tail tasks, i.e., abso-
lute improvement is 1.44 in Head@3 and 8.98 in Tail@4, compared to the best-performing baseline Hyperformer++. The performance gain precisely demonstrates the advantages of knowledge sharing between head and tail tasks and knowledge mining from external resources. 2. Our model also outperforms the in-context learning baseline EPR without any parameter update of the pre-trained LM. This shows that leveraging knowledge mined from a pretrained LM and directly optimizing QA tasks can lead to better QA performance. See Appendix B for more evaluation details of all 21 seen tasks.
Table 2 shows the result on unseen tasks. Our model yields the best performances on all metrics. We can also observe that: 1. Our model that shares knowledge through fine-grained components (i.e., a pool of meta prompts) and mines knowledge from an LM generally obtain higher performance. 2. EPR is on par with the other baselines trained on seen tasks. It shows that a pre-trained LM embeds a large amount of knowledge, which can help QA tasks potentially.

heading: 4.6 Ablation Studies
text: Model Main Components: Ablation studies are carried out to validate the effectiveness of each main component in our model. Specifically, the following variants are investigated: 1. w/o Pm removes the knowledge sharing component, i.e., meta prompt Pm is not used. 2. w/o Pk removes the knowledge mining component, i.e., knowledge prompt Pk is not used. Results in Table 1 and Table 2 indicate that our model outperforms all ablation variants. Specifically, we can also observe that: 1. Both knowledge sharing (see w/o Pm) and knowledge mining (see w/o Pk) components help to improve the QA performance. 2. Knowledge mining brings larger improvement compared to knowledge sharing component on both tail and unseen tasks. This further proves the importance of leveraging knowledge embedded in the pre-trained LM for the OLTQA setting. We provide examples where our model is correct and the variant without knowledge mining (i.e., w/o Pk) is incorrect, together with 4 top hints selected by the retrievethen-rerank framework in Appendix C.
Knowledge Mining Components: To evaluate design choices of retrieve-then-rerank framework ⟨R1, R2⟩ and two-stage knowledge distillation (KD) in knowledge mining, we perform ablation on alternatives: 1. BM25 Retriever uses the unsu-
pervised retriever BM25 (Robertson et al., 2009) to replace retriever R1. 2. EPR Retriever trains R1 by using a pre-trained LM as the scoring function (Rubin et al., 2022). 3. w/o Re-ranker removes the re-ranker R2, and directly uses R1 to select examples and generate hints. 4. w/o MKD removes the adaptive mutual KD loss Lmkd. 5. Static MKD removes Lmkd, and performs mutual KD based on the performance of R1, R2, and F evaluated at the very beginning of training stage two. 6. Back KD removes Lmkd, and train R1 and R2 using knowledge distilled from F (Izacard et al., 2022).
Results in Table 3 show that the knowledge mining approach used in our model performs better than all other variants. We can further observe that: 1. Retrieving in-context examples using other approaches (i.e., BM25 Retriever and EPR Retriever) degenerates the model performance by a large margin. This shows the effectiveness of the two-stage training of R1 in our model. 2. Re-ranking hints generated by an LM help to improve the QA performance (see w/o Re-ranker). 3. Removing the adaptive mutual KD loss (i.e., w/o MKD) degenerates the QA performance. This proves the effectiveness of information exchange between the two branches of our model. 4. Variants of Lmkd lead to limited QA performance (see Static MKD and Back KD). This shows the importance of performance-aware for mutual knowledge distillation.

heading: 4.7 Further Analysis
text: Effect of Pm in Different Data Distributions We also validate the effectiveness of meta prompt Pm for knowledge sharing in different data distributions. Specifically, we construct a variant of the training set (and denote it as “w/o head”) by discarding samples from head tasks, which consist of samples from 16 tail tasks. We also denote the original training set as “w/ head”. The performance of our model on these two datasets is tested with and without Pm.
Results in Table 4 show that our model benefits more from Pm with samples from head tasks. This further validates our claim that meta prompt Pm helps to facilitate knowledge sharing between head, tail, and unseen tasks.
Analysis on Pm Selection Mechanism We plot the heat map of meta prompt Pm selection frequency for each task in Figure 3. We can observe that: 1. Some hot meta prompts are shared by most tasks, which probably encode common knowledge for question answering. 2. Other meta prompts are shared by a few tasks, which might contain task-specific knowledge.
Analysis on Adaptive Mutual KD We visualize the performance of R1, R2, and QA model F on the validation set Dval which are evaluated (Eq. 4) at the beginning of each epoch during training stage two in Figure 4. We can observe that: 1. Initially, R1 and R2 are allowed to generate knowledge for training F because they are pre-trained in training stage one. After epoch one, F performs better than R1 and R2, and starts to teach student model R1 and R2 as a teacher model. 2. During training, R2 gradually outperforms R1. Overall, the relative performance of R1, R2, and QA model F compared to each other is not stable during training. Thus,
to avoid collective failures, being aware of individual performance is essential to perform mutual knowledge distillation.
Influence of Dataset Longtail-ness The longtailness of the dataset (i.e., the degree of imbalance of task distribution in training) could have an impact on the model performance. Figure 5(a) shows that as the dataset becomes more imbalanced (i.e., α of Zipf distribution increases), our model only undergoes a moderate performance drop compared to UnifiedQA. Here, the performance is evaluated on a test set from all 43 tasks. Influence of Proportion of Unseen Tasks The performance change w.r.t. proportion of unseen tasks is shown in Figure 5(b). Compared to UnifiedQA, the performance of our model changes steadily as the proportion of unseen tasks rises. The knowledge sharing and knowledge mining components of our model enhance robustness to unseen tasks.

heading: 5 Conclusion
text: We introduce the open long-tailed QA (OLTQA) task that learns from natural long-tail distributed data and optimizes the performance over seen and
unseen tasks. We propose an OLTQA model to address the challenges of OLTQA. An instance-level knowledge sharing mechanism is introduced, and a retrieve-then-rerank frame is employed to mine knowledge from a large pre-trained LM through a two-stage knowledge distillation training process. We validate our model on a curated OLTQA benchmark. Our publicly available data would enable future research that is directly transferable to realworld applications.
Limitations
We identify the major limitation of this work is its input modality. Specifically, our model only considers textual inputs, ignoring question answering tasks in vision and audio. A multi-modal question answering model under realistic open longtailed scenario is worth further exploration. Fortunately, through multi-modal pre-training models (Xu et al., 2021; Huo et al., 2021) and question answering methods (Kim et al., 2020), we can equip our model with multi-modal question answering ability. For future work, learning multi-modal question answering in an open (including out of distribution data (Lang et al., 2022, 2023a,b)) longtailed scenario still remains a challenge, and we will continue to work on it.
Ethics Statement
This work does not raise any direct ethical issues. In the proposed work, we seek to develop a method for long-tailed question answering in an open world, and we believe this work can benefit the field of question answering, with the potential to benefit other fields involving open long-tailed problem. All experiments are conducted on open datasets.

heading: A Datasets and Metrics
text: Datasets. We carry out experiments on the following datasets:
• Extractive: SQuAD 1.1 (Rajpurkar et al., 2016), SQuAD 2 (Rajpurkar et al., 2018), NewsQA (Trischler et al., 2017) Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), AdversarialQA (Bartolo et al., 2020), ReCoRD (Zhang et al., 2018),
• Abstractive: DROP (Dua et al., 2019) NarrativeQA/NarQA (Kočiský et al., 2018), the open-domain version of NaturalQuestions/NatQA (Kwiatkowski et al., 2019), QAConv (Wu et al., 2022), TweetQA (Xiong et al., 2019),
• Multiple-choice: HeadQA (Vilares and Gómez-Rodríguez, 2019), RACE-C (Liang et al., 2019), MCTest (Richardson et al., 2013), RACE (Lai et al., 2017), OpenBookQA (Mihaylov et al., 2018) ARC (Clark et al., 2018, 2016), QASC (Khot et al., 2020), CommonsenseQA/CQA (Talmor et al., 2019), Winogrande (Sakaguchi et al., 2020), MMMLU (Hendrycks et al., 2021), ReClor (Yu et al., 2020), Quail (Rogers et al., 2020), OneStopQA (Berzak et al., 2020),
MCScript (Ostermann et al., 2018), MCScript 2.0 (Ostermann et al., 2019), CosmosQA (Huang et al., 2019), ProcessBank (Berant et al., 2014), DREAM (Sun et al., 2019), PROST (Aroca-Ouellette et al., 2021), PhysicalIQA/PIQA (Bisk et al., 2020), SocialIQA/SIQA (Sap et al., 2019)
• Yes/no: BoolQ (Clark et al., 2019), BoolQNP (Khashabi et al., 2020a) the binary (yes/no) subset of MultiRC (Khashabi et al., 2018), StrategyQA (Geva et al., 2021), PubmedQA (Jin et al., 2019).
The statistics of these datasets are summarized in Table 8. Note that we follow the pre-process scheme released by Khashabi et al. (2020b) to tackle these datasets. As 22 tasks are unseen in the training phase, we only use the training and validation sets of the other 21 tasks to build our framework. Metrics. The evaluation for each task follows Khashabi et al. (2022). Specifically, for Multiple-choice tasks, we use accuracy. For Extractive tasks, we use the F1 token overlap between the answer text and golden truth. For Abstractive tasks, we use ROUGE-L for NarrativeQA, BLEU for TweetQA, and F1 for the other tasks. For Yes/no questions, we also use the F1 token overlap.

heading: B Overall Results
text: We compare our OLTQA model with competitive baselines and ablation variants on each component. The full results of our model, baselines and ablation variants under 21 seen tasks are shown in Table 5, while the results under 22 unseen tasks are shown in Table 6. Bold numbers are superior results.

heading: C Case Study
text: We provide examples from tail and unseen tasks, where our model is correct and the variant without knowledge mining (i.e., w/o Pk) is incorrect, together with top hints selected by the retrieve-thenrerank framework. Table 7 demonstrates that hints yielded by our model are related to the ground truth which effectively corrects the predicted answer.

heading: D More Implementation Details
text: We use T5-base (Raffel et al., 2020) to initialize our encoder-decoder QA model (12 layers, 768- dimensional hidden size, and 12 attention heads).
In knowledge sharing, we maintain totally s = 30 meta prompts, and set the length of each meta prompt to 10. We adopt a fixed T5-base encoder with an average pooling layer to generate the query vector. For each instance ⟨c, q,a⟩, we select s̃ = 5 meta prompts to construct Pm. For meta prompt key training, we set η = 0.15 and γ = 0.3 in Eq. 1.
In knowledge mining, we adopt GLM-10B (Du et al., 2022) with 10B parameters as a large pretrained LM. For retrieve-then-rerank example selection, R1 first retrieves l = 64 examples from all training examples, and R2 selects l̃ = 4 examples among retrieval results. The retriever R1 is implemented with two separate dense encoders EX(·) and ED(·) to map ⟨c, q⟩ and ei into vectors. The score for ei is then computed as EX([c; q])
T · ED(ei), which is the dot product of two vectors. The re-ranker R2 is a dense encoder EC combined with a linear layer fc. Concretely, EC transforms the concatenation of example ei, hint hi and input ⟨c, q⟩ into a representation, which is fed into fc to get the score, denoted as fc(EC([ei;hi; c; q])). EC ,ED and EX are all initialized with BERT base uncased (Devlin et al., 2019). In two-stage training, we leverage BM25 to select c = 512 example candidates.
All experiments are performed on 8 A100 GPUs (80GB). The batch size is set to 32. We use the AdamW (Loshchilov and Hutter, 2017) optimizer with a learning rate of 1e-4 and batch size of 32. The dataset is trained for five epochs. All hyper-parameters are tuned according to the average score on the validation set. In our experiments, We perform 3 runs by setting the random seed to {42, 43, 44} respectively. In this way, we report the average score of each method. Note that we only use the random seed 42 for tuning hyper-parameters. Our model has 551.59M tunable parameters.
To obtain the ROUGE-L score, we use the NLTK package for sentence tokenization, and python rouge-score package for evaluation. To obtain the BLEU score, we use the NLTK package for evaluation.

heading: E Results under Different Random Seeds
text: We use random seed 42 and 43 to construct another two sets of head, tail, and unseen tasks, and compare our method with the baseline UnifiedQA. As shown in Table 9, our method is robust when using different tasks as head, tail or unseen tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section4, Appendix A, Appendix D
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? Section 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 4, Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4, Appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix D

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix D
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Our publicly available data would enable future research that is directly transferable to realworld applications. Limitations
We identify the major limitation of this work is its input modality. Specifically, our model only considers textual inputs, ignoring question answering tasks in vision and audio. A multi-modal question answering model under realistic open longtailed scenario is worth further exploration. Fortunately, through multi-modal pre-training models (Xu et al. , 2021; Huo et al. , 2021) and question answering methods (Kim et al. , 2020), we can equip our model with multi-modal question answering ability. For future work, learning multi-modal question answering in an open (including out of distribution data (Lang et al. , 2022, 2023a,b)) longtailed scenario still remains a challenge, and we will continue to work on it