MetaAdapt: Domain Adaptive Few-Shot Misinformation Detection via Meta Learning
Year: 2023

Abstract:
With emerging topics (e.g., COVID-19) on social media as a source for the spreading misinformation, overcoming the distributional shifts between the original training domain (i.e., source domain) and such target domains remains a non-trivial task for misinformation detection. This presents an elusive challenge for early-stage misinformation detection, where a good amount of data and annotations from the target domain is not available for training. To address the data scarcity issue, we propose MetaAdapt, a meta learning based approach for domain adaptive few-shot misinformation detection. MetaAdapt leverages limited target examples to provide feedback and guide the knowledge transfer from the source to the target domain (i.e., learn to adapt). In particular, we train the initial model with multiple source tasks and compute their similarity scores to the meta task. Based on the similarity scores, we rescale the meta gradients to adaptively learn from the source tasks. As such, MetaAdapt can learn how to adapt the misinformation detection model and exploit the source data for improved performance in the target domain. To demonstrate the efficiency and effectiveness of our method, we perform extensive experiments to compare MetaAdapt with state-of-the-art baselines and large language models (LLMs) such as LLaMA, where MetaAdapt achieves better performance in domain adaptive few-shot misinformation detection with substantially reduced parameters on real-world datasets.

heading: 1 Introduction
text: Recently, significant progress has been made in misinformation detection due to the improvements in developing machine learning-based methods (Wu et al., 2019; Shu et al., 2020c; Wu et al., 2022b). Such methods include large language models (LLMs), which can be fine-tuned for detecting and responding to rumors on social media platforms (Jiang et al., 2022; He et al., 2023; Touvron
et al., 2023). However, misinformation on emerging topics remains an elusive challenge for existing approaches, as there exists a large domain gap between the training (i.e., source domain) and the target distribution (i.e., target domain) (Yue et al., 2022). For instance, existing models often fail to detect early-stage misinformation due to the lack of domain knowledge (see Figure 1).
With the increase of emerging topics (e.g., COVID-19) on social media as a source of misinformation, the failure to distinguish such earlystage misinformation can result in potential threats to public interest (Roozenbeek et al., 2020; Chen et al., 2022). To tackle the problem of cross-domain early misinformation detection, one possible solution is crowdsourcing, which collects domain knowledge from online resources (Medina Serrano et al., 2020; Hu et al., 2021b; Shang et al., 2022a; Kou et al., 2022b). Another alternative approach is to transfer knowledge from labeled source data to unlabeled target data with domain adaptive methods (Zhang et al., 2020; Li et al., 2021; Suprem and Pu, 2022; Shu et al., 2022; Yue et al., 2022; Zeng et al., 2022). However, the former methods use large amounts of human annotations while the latter approaches require extensive unlabeled examples. As such, existing methods are less effective for detecting cross-domain early misinformation, where neither large amounts of annotations nor tar-
5223
get domain examples can be provided for training.
Despite the insufficiency of early misinformation data, limited target examples and annotations can often be achieved at minimal costs (Kou et al., 2021, 2022a; Shang et al., 2022c). Nevertheless, previous approaches are not optimized to learn from the source data under the guidance of limited target examples (Zhang et al., 2020; Lee et al., 2021; Mosallanezhad et al., 2022; Yue et al., 2022). Such methods are often unaware of the adaptation objective and thus fail to maximize the transfer of source domain knowledge. To fully exploit existing data from different domains, we consider a cross-domain few-shot setting to adapt misinformation detection models to an unseen target domain (Motiian et al., 2017; Zhao et al., 2021; Lin et al., 2022). That is, given a source data distribution and access to limited target examples, our objective is to maximize the model performance in the target domain. An example of such application can be adapting a model from fake news detection to COVID early misinformation detection, where abundant fake news from existing datasets can be used for training the model under the guidance of limited COVID misinformation examples.
In this paper, we design MetaAdapt, a few-shot domain adaptation approach based on meta learning for early misinformation detection. Specifically, we leverage the source domain examples (i.e., source task) and train a model to obtain the task gradients. Then, we evaluate the updated model on the few-shot target examples (i.e., meta task) to derive second-order meta gradients w.r.t. the original parameters. We additionally compute the similarity between the task gradients and meta gradients to select more ‘informative’ source tasks, such that the updated model adaptively learns (from the source data) to generalize even with a small number of labeled examples. In other words, the meta model learns to reweight the source tasks with the objective of optimizing the model performance in the target domain. Therefore, the resulting model can optimally adapt to the target distribution with the provided source domain knowledge. To show the efficacy of our meta learning-based adaptation method, we focus on the early-stage misinformation of COVID-19 and demonstrate the performance of MetaAdapt on real-world datasets, where MetaAdapt can consistently outperform state-ofthe-art methods and large language models by demonstrating significant improvements.
We summarize our contributions as follows1:
1. We propose a few-shot setting for domain adaptive misinformation detection. Here, the labeled source data and limited target examples are provided for the adaptation process.
2. We propose MetaAdapt, a meta learningbased method for few-shot domain adaptive misinformation detection. Our MetaAdapt ‘learns to adapt’ to the target data distribution with limited labeled examples.
3. MetaAdapt can adaptively learn from the source tasks by rescaling the meta gradients. Specifically, we compute similarity scores between the source and meta tasks to optimize the learning from the source distribution.
4. We show the effectiveness of MetaAdapt in domain adaptive misinformation detection on multiple real-world datasets. In our experiments, MetaAdapt consistently outperforms state-of-the-art baselines and LLMs.

heading: 2 Related Work

heading: 2.1 Misinformation Detection
text: Existing misinformation detection methods can be categorized into the following: (1) content-based misinformation detection: such models are trained to perform misinformation classification upon input claims. For example, pretrained transformer models are used to extract semantic or syntactic properties to detect misinformation (Karimi and Tang, 2019; Das et al., 2021; Yue et al., 2022; Jiang et al., 2022). Moreover, multimodal input is used to learn text and image features that improve detection performance (Khattar et al., 2019; Shang et al., 2021; Santhosh et al., 2022; Shang et al., 2022b); (2) social-aware misinformation detection: user interactions can be used to evaluate online post credibility (Jin et al., 2016). Similarly, patterns on propagation paths help detect misinformation on social media platforms (Monti et al., 2019; Shu et al., 2020b). Social attributes like user dynamics enhance misinformation detection by introducing context (Shu et al., 2019). Combined with contentbased module, misinformation detection systems demonstrate improved accuracy (Mosallanezhad et al., 2022; Lin et al., 2022); (3) knowledge-based
1We adopt publicly available datasets in the experiments and release our implementation at https://github.com/Yueeeeeeee/MetaAdapt.
misinformation detection: external knowledge can be leveraged as supporting features and evidence in fact verification and misinformation detection (Vo and Lee, 2020; Liu et al., 2020; Brand et al., 2021). Knowledge graphs or crowdsourcing approaches can derive additional information for explainability in misinformation detection (Cui et al., 2020; Hu et al., 2021b; Koloski et al., 2022; Kou et al., 2022a; Shang et al., 2022a; Wu et al., 2022a). Yet existing methods focus on improving in-domain performance or explainability, few-shot misinformation detection in a cross-domain setting is not well researched. Hence, we study domain adaptive few-shot misinformation detection using contentbased language models in our work.

heading: 2.2 Domain Adaptive Learning
text: Domain adaptive learning aims to improve model generalization on an unseen domain given a labeled source domain. Such methods are primarily studied in image and text classification problems (Li et al., 2018; Kang et al., 2019; Sicilia et al., 2021). In image classification, existing methods minimize the representation discrepancy between source and target domains to learn domain-invariant features and transfer source knowledge to the target domain (Kang et al., 2019; Na et al., 2021). Similarly, domain-adversarial and energy-based methods adopt additional critique, with which domainspecific features are regularized (Sicilia et al., 2021; Xie et al., 2022). Class-aware contrastive learning is proposed for fine-grained alignment, which regularizes the inter-class and intra-class distances to achieve domain-invariant yet class-separating features (Li et al., 2018; Shen et al., 2022).
In text classification, various approaches are proposed to improve the target domain performance in cross-domain settings (Silva et al., 2021; Li et al., 2021; Ryu et al., 2022; Nan et al., 2022). For instance, domain-adversarial training is used to learn generalizable features to detect cross-domain multimodal misinformation (Wang et al., 2018; Lin et al., 2022; Shu et al., 2022). Reinforcement learning and contrastive adaptation are also adopted for fine-grained domain adaptation in misinformation detection (Mosallanezhad et al., 2022; Yue et al., 2022). Nevertheless, domain-adaptive misinformation detection is not well studied in the fewshot learning setting. Therefore, we combine both settings and develop a method tailored for fewshot domain adaptation in misinformation detec-
tion: MetaAdapt. By leveraging knowledge transfer via the proposed meta objective, our approach shows significant improvements on out-of-domain misinformation using only a few labeled examples.

heading: 2.3 Few-Shot Learning
text: Few-shot learning aims to learn a new task with a few labeled examples (Wang et al., 2020). Existing few-shot learning approaches (e.g., prototypical networks) learn class-wise features in the metric space to rapidly adapt to new tasks (Vinyals et al., 2016; Snell et al., 2017). Meta learning methods search for the optimal initial parameters for unseen few-shot tasks via second-order optimization (Finn et al., 2017; Rajeswaran et al., 2019; Zhou et al., 2021). In computer vision, few-shot domain adaptation is studied in image classification to transfer knowledge to an unseen target domain (Motiian et al., 2017; Tseng et al., 2019; Zhao et al., 2021). For language problems, meta learning is proposed to improve the few-shot performance in language modeling and misinformation detection (Sharaf et al., 2020; Han et al., 2021; Salem et al., 2021; Zhang et al., 2021; Lei et al., 2022).
To the best of our knowledge, few-shot domain adaptive misinformation detection via meta learning is not studied in current literature. Moreover, the mentioned few-shot setting can be helpful in real-world scenarios (e.g., detecting rumors on emerging topics). As such, we propose meta learning-based MetaAdapt for misinformation detection. MetaAdapt leverages limited target examples and adaptively exploits the source domain knowledge via task similarity, and thus improves the few-shot misinformation detection performance in the unseen target domain.

heading: 3 Preliminary
text: We consider the following problem setup for domain adaptive few-shot misinformation detection: labeled source data and k-shot target examples (i.e., k examples per class) are available for training. The objective of our framework is to train a misinformation detection model f that is optimized for the target domain performance using both source and few-shot target examples. Data: Our research is defined within the scope of single-source adaptive misinformation detection (i.e., we study the adaptation problem from a single source domain to the target domain). We denote Ds as the source domain and Dt as the (differ-
ent) target domain. In our setting, labeled source data and limited target examples can be used for training. The few-shot adaptation is performed in two-fold: (1) an initial model is updated upon multiple batches of sampled source data examples (i.e., source tasks) respectively; (2) the updated models are evaluated on the few-shot target examples respectively (i.e., meta task) to compute the meta loss, followed by updating the initial parameters using the derived second-order derivatives. The input data is defined as follows:
• Labeled source data: source training data Xs is provided by source domain Ds. Here, each sample (x(i)s , y (i) s ) ∈ Xs is a tuple comprising of
input text x(i)s and label y (i) s ∈ {0, 1} (i.e., false or true). During training, source data batches are sampled as different ‘source tasks’ and used to optimize the initial model.
• Few-shot target data: we assume limited access to the target domain Dt. In other words, only kshot subset X ′t from Xt is provided for training. Target samples are provided in the same label space with (x(i)t , y (i) t ) ∈ X ′t, while the size of
X ′t is constrained with k examples in each label class (i.e., 10 in our experiments). X ′t, or ‘meta task’ is used to compute the meta loss and meta gradients w.r.t. the initial parameters.
Model & Objective: The misinformation detection model is represented by a function f parameterized by θ. f takes textual statements as input and predicts the probability of input as true information, i.e., y(i) = argmax(f(θ,x(i))). For optimization, our objective is to maximize the model performance on target data Xt (Note Xt ̸= X ′t) from the target domain Dt. Mathematically, this can be formulated as the optimization problem of
minimizing the loss L of θ over target data Xt:
min θ
L(θ,Xt), (1)
where L is the loss function (i.e., cross-entropy).

heading: 4 Methodology
text: Provided with labeled source data and k-shot target examples, we first present our meta adaptation framework for domain adaptive few-shot misinformation detection. To improve the adaptation performance, we introduce a second-order meta learning algorithm MetaAdapt based on learnable learning rate and task similarity. An illustration of MetaAdapt is provided in Figure 2. Upon deployment, the adapted models achieve considerable improvements thanks to the adaptive optimization and similarity-guided meta adaptation.

heading: 4.1 Few-Shot Meta Adaptation
text: Given model f with initial parameters θ, source dataset Xs, few-shot target data X ′t and the number of tasks n in each iteration, we formulate the meta adaptation framework as a bi-level optimization problem and provide a mathematical formulation:
min θ
1
n
n∑ L(Alg(θ, Sampler(Xs)),X ′t), (2)
in which Sampler stands for the source task sampler that draws source tasks of a fixed size from Xs, Alg represents the optimization algorithm using first-order gradient descent, i.e.:
Alg(θ,X) = ϕ = θ − α∇θL(θ,X), (3)
with α representing the task learning rate and ϕ representing the learnt parameter set over X .
In Equation (2), we are interested in learning an optimal parameter set θ that minimizes the
meta loss on the few-shot target set X ′t, which can be denoted as the outer-level optimization of meta adaptation. The outer-level learning is achieved by deriving gradients w.r.t. θ based on the meta loss using task-specific parameters (i.e., Alg(θ, Sampler(Xs)) or ϕ) and few-shot examples X ′t. To obtain task-specific parameters, we sample a batch of source examples using Sampler and perform gradient descent steps on the original θ (i.e, Equation (3)), which is known as the innerlevel optimization. The inner-level optimization only requires first-order derivatives, however, to optimize the outer-level problem, it is necessary to differentiate through Alg (i.e., ϕ), which requires using second-order gradients (Finn et al., 2017).
We now take a closer look at how to compute the derivatives with chain rule in meta adaptation:
dL(Alg(θ,X),X ′t) dθ =
dAlg(θ,X) dθ
∇ϕL(Alg(θ,X),X ′t), (4)
Note that Alg(θ,X) is equivalent to ϕ. The rightside component ∇ϕL(Alg(θ,X),X ′t) refers to first-order derivatives by computing the meta loss using the few-shot examples X ′t and task-specific parameter set ϕ (L → ϕ). This step can be computed with conventional gradient descent algorithms. The left-side component dAlg(θ,X)dθ is a non-trivial step as it requires second-order derivatives (i.e., Hessian matrix) to track parameter-toparameter changes through Alg(θ,X) to θ. In our implementation, we compute the meta gradients w.r.t. θ using the meta evaluation loss similar to model agnostic meta learning (Finn et al., 2017). We also adopt adaptive learning rate α and β and cosine annealing to improve the convergence of source and meta tasks (Antoniou et al., 2018).

heading: 4.2 The Proposed MetaAdapt
text: While meta adaptation leverage source tasks to improve the target domain performance, it learns homogeneously from all source tasks without considering the informativeness of each individual task. To further exploit the source domain knowledge, we propose a task similarity-based MetaAdapt for domain adaptive few-shot misinformation detection. We first estimate the task-specific parameters with adaptive learning rates, followed by rescaling the meta loss using the task similarity scores. The proposed method selectively learns from source tasks, and thus, further improves the adaptation
Algorithm 1: MetaAdapt Algorithm 1 Input Parameter set θ, source data Xs, k-shot data
X ′t, number of iterations N , number of tasks n; 2 for iter ∈ {1, 2, ..., N} do 3 for i ∈ {1, ..., n} do 4 Sample source task from Xs; 5 Update task parameter set ϕi with Equation (3); 6 Compute meta loss and meta gradients using ϕi, X ′t as in Equation (4); 7 Calculate similarity score si with Equation (5); 8 end 9 Normalize similarity scores s with Equation (6);
10 Update original parameter θ with Equation (7); 11 end
performance. We present the training details of MetaAdapt in Algorithm 1.
We initialize the model and denote the parameters with θ. For source task i, the original parameters are updated with first-order derivatives as we sample source tasks from Xs. Specifically in each step, the task gradients can be computed with ∇θL(θ, Sampler(Xs)), as in Equation (3). After a few gradient descent steps, the parameters converge locally and we denote the updated parameters with ϕi. As we update multiple times in each source task, we denote the task gradients with ϕi − θ for simplicity. Subsequently, the meta loss L(ϕi,X ′t) is computed using ϕi and the few-shot target examples X ′t. To compute the meta gradients with backpropagation, we follow the chain rule and compute the derivatives w.r.t. the original parameter set θ. Similar to Equation (4), we use dϕi dθ ∇ϕiL(ϕi,X ′t) to denote the meta gradients.
To compute task similarity scores, we leverage task and meta gradients. The objective of computing gradient similarity is to selectively learn from the source tasks. If the task and meta gradients yield a high similarity score, the parameters are converging to the same direction in both inner- and outer-loop optimization. Thus, the source task optimization path is more ‘helpful’ to improve the meta task performance (i.e., target domain performance). Or if, on the contrary, then the source task may be less effective for improving the meta task performance. Based on this principle, we compute task similarity score si with cosine similarity:
si = CosSim(ϕi − θ, dϕi dθ ∇ϕiL(ϕi,X ′t)). (5)
In each iteration, we sample n source tasks and compute the similarity scores for each pair of task
and meta gradients. Then, the similarity scores [s1, s2, ..., sn] are transformed to a probability distribution using tempered softmax:
s = softmax([ s1 τ , s2 τ , ..., sn τ ]), (6)
where τ is the temperature hyperparameter to be selected empirically. Finally, we update the original parameters with rescaled meta gradients:
θ − β n∑
i
si · dϕi dθ ∇ϕiL(ϕi,X ′t). (7)
In summary, MetaAdapt computes task and meta gradients using sampled source tasks and few-shot target examples. Then, task similarity is computed to find more ‘informative’ source tasks, followed by tempered rescaling of the meta gradients. Finally, the updated model parameters should exploit the source domain knowledge and demonstrate improved performance on the target data distribution. The overall framework of MetaAdapt is illustrated in Figure 2. Unlike previous works (Motiian et al., 2017; Tseng et al., 2019; Zhao et al., 2021; Yue et al., 2022), we discard domain-adversarial or feature regularization methods, instead, we propose to leverage meta adaptation to guide the knowledge transfer from the source to target domain. Additionally, similarity-based gradients rescaling is designed to exploit different source tasks to achieve fine-grained adaptation performance.

heading: 5 Experiments

heading: 5.1 Settings
text: Model: Similar to (Li et al., 2021; Yue et al., 2022), we select RoBERTa as the base model to encode input examples in MetaAdapt. RoBERTa is a transformer model pretrained on a variety of NLP tasks before the COVID pandemic (Liu et al., 2019). Evaluation: To validate the proposed method, we follow (Kou et al., 2022a; Li et al., 2021; Yue et al., 2022) and split the datasets into training, validation and test sets. The few-shot target examples are selected as the first k examples in the validation set and the rest validation examples are used for validating the model. For evaluation metrics, we adopt balance accuracy (BA), accuracy (Acc.) and F1 score (F1) to evaluate the performance. See evaluation details in Appendix A. Datasets and Baselines: To examine MetaAdapt performance, we adopt multiple source and target datasets. We follow (Yue et al., 2022) and
adopt FEVER (FE) (Thorne et al., 2018), GettingReal (GR) (Risdal, 2016), GossipCop (GC) (Shu et al., 2020a), LIAR (LI) (Wang, 2017) and PHEME (PH) (Buntain and Golbeck, 2017) as the source datasets. For the target domain, we adopt CoAID (Cui and Lee, 2020), Constraint (Patwa et al., 2021) and ANTiVax (Hayawi et al., 2022). Our naïve baseline leverages few-shot target examples to fine-tune the source pretrained models. We also adopt state-of-the-art baselines from domain adaptation and few-shot learning methods for domain adaptive few-shot misinformation detection: CANMD, ACLR, ProtoNet and MAML (Finn et al., 2017; Snell et al., 2017; Lin et al., 2022; Yue et al., 2022). Additionally, we select two large language models LLaMA and Alpaca to evaluate few-shot in-context learning (ICL) and parameter-efficient fine-tuning (PEFT) performance in misinformation detection (Touvron et al., 2023; Taori et al., 2023). Implementation: We follow the preprocessing pipeline as in (Yue et al., 2022). Specifically, we translate special symbols (e.g., emojis) back into English, tokenize hashtags, mentions and URLs and remove special characters from the input. We use the 10-shot setting (i.e., k = 10), the model is trained using AdamW optimizer with 0.01 weight decay and no warm-up, where we sample 3 source tasks and perform 3 updates in inner-loop optimization. Then, we compute the meta loss and evaluate task similarity scores before rescaling the meta gradients with temperature τ . All our main experiments are repeated 3 times, we select the best model with the validation set for final evaluation. Hyperparameter selection (e.g., inner and outer learning rates, temperature etc.) and implementation details are provided in Appendix A.

heading: 5.2 Main Results
text: Supervised results: We first report supervised results on all datasets in Table 1. The upper and
lower parts of the table report the source and target dataset performance respectively. We observe the following: (1) overall, the performance on statements and posts achieves better performance than news articles. An example can be found on GossipCop (News) with 0.776 BA, compared to 0.863 on PHEME (Social network posts); (2) for disproportionate label distributions (e.g., CoAID), the BA metric reduces drastically compared to other metrics, indicating the difficulty of training fair models on unfair distributions. For instance, BA is circa 10% lower than accuracy and F1 on CoAID; (3) the RoBERTa model only achieves an average BA of 0.778 on source datasets, which suggests that transferring knowledge from the source to the target datasets can be a challenging task. Few-shot adaptation results: The few-shot crossdomain experiments (10-shot) on all source-target combinations are presented in Table 2. In the table, rows represent the source datasets while the columns represent the target datasets. We include the adaptation methods in each row, while the metrics are reported in the columns under target datasets. For MetaAdapt, we report the mean results in the table and provide the standard deviation values using the ± sign. For convenience, the best results are marked in bold and the second best results are underlined. We observe: (1) adapting to the COVID domain is a non-trivial task upon dissimilar source-target label distributions. In the example of GossipCop → CoAID, baseline methods show BA values to be slightly higher than 0.6 (despite high accuracy and F1), suggesting that the model predicts the majority class with a
much higher likelihood; (2) by learning domaininvariant feature representation, baseline methods like CANMD and ACLR can improve the adaptation results compared to the naïve baseline. For instance, CANMD achieves over 42.3% and 21.7% average improvements on BA for Constraint and ANTiVax; (3) using the meta adaptation approach, MetaAdapt significantly outperform all baselines in the BA metric. For instance, MetaAdapt outperforms the best-performing baseline MAML 7.4% in BA on ANTiVax, with similar trends to be found in accuracy and F1. Specifically for CoAID (with over 90% positive labels), improvements on BA demonstrates that MetaAdapt can learn fair features for improved detection results despite slightly worse accuracy and F1 results. In summary, the results in Table 2 show that MetaAdapt is particularly effective in adapting early misinformation detection systems using limited target examples. In contrast to the baseline models, MetaAdapt can achieve significant improvements on all metrics across source-target dataset combinations. In the case of large domain discrepancy, MetaAdapt demonstrates superior performance by exploiting the few-shot target examples with second-order dynamics and similarity-based adaptive learning. Comparison to large language models: To further demonstrate the effectiveness of MetaAdapt in domain adaptive misinformation detection, we compare our MetaAdapt with state-of-the-art large language models (LLMs). In particular, we adopt LLaMA-7B and Alpaca-7B and perform both fewshot in-context learning (i.e., ICL) and parameterefficient fine-tuning (i.e., FT) on target datasets, with results presented in Table 3. We notice: (1) de-
spite the significant increase in model parameters (from ∼0.1B to 7B), LLMs can still fail to distinguish misinformation without further tuning. For instance, LLaMA consistently scores 0.5 in BA by predicting the majority class in in-context learning. (2) Fine-tuning LLMs can significantly improve the performance in misinformation detection. With PEFT tuning, Alpaca achieves 40.7% performance improvement in BA across target datasets conditioned only on the few-shot target examples. (3) With the proposed MetaAdapt, smaller language models like RoBERTa are capable of outperforming fine-tuned large language models. On average, MetaAdapt outperforms LLaMA-PEFT and Alpaca-PEFT by 14.3% and 14.1% in the BA metric on target datasets. The results suggest that MetaAdapt is both effective and efficient in early misinformation detection by combining outof-domain knowledge and meta adaptation.
Robustness study: We also evaluate the sensitivity of MetaAdapt with respect to the number of few-shot examples. In particular, we select the number from 0 to 15 and use MetaAdapt to perform the adaptation. The results are averaged across the source datasets and reported in Table 4. Note that for 0-shot experiments, we train the models on source datasets and directly evaluate on target data. We observe the following: (1) as expected, the adaptation performance improves rapidly with increasing number of few-shot examples. (2) Surprisingly, we observe performance drops in accuracy and F1 on CoAID when increasing the few-shot number from 0 to 1. This suggests that the large domain discrepancy between both domains may reduce the effectiveness of MetaAdapt; (3) Overall,
the magnitude of improvements grows rapidly at first and then slowly plateaus as we increase the few-shot number. The largest improvements can be found from 1-shot to 5-shot setting, where the BA results improve by 18.1% on average. In sum, we observe that even limited number of target examples can improve the target domain performance with MetaAdapt. We provide additional hyperparameters sensitivity analysis in Appendix B. Ablation study: We evaluate the effectiveness of the proposed components in MetaAdapt. In particular, we remove the proposed similarity-based gradients rescaling (w/o Task Similarity), adaptive learning rate (w/o Adaptive LR) and second-order optimization method (w/o 2nd-order Grads) in order and observe the performance changes. Experiment results on target datasets are averaged across the source datasets and are reported in Table 5. For all components, we observe performance drops on BA when removed from MetaAdapt2. For example, the performance of MetaAdapt reduces by 2.0% and 2.3% in BA when we remove task similarity and adaptive LR consecutively. We further replace MetaAdapt with first-order approximation, which results in a performance drop of 3.9% in the BA metric. Overall, the results suggest that the proposed components are effective for domain adaptive few-shot misinformation detection.

heading: 6 Conclusion
text: In this paper, we explore meta learning in domain adaptive few-shot misinformation detection. We propose MetaAdapt, a meta learning-based approach for cross-domain few-shot adaptation. MetaAdapt is the first to leverage few-shot target examples for exploiting the source domain knowledge under the guidance of limited target examples. Experiment results demonstrate the effectiveness of our method by achieving promising results on multiple challenging source-target dataset combinations, where MetaAdapt can significant outperform
2In the case of CoAID, Acc and F1 are less reliable evaluation metrics due to the imbalanced distribution of labels.
the state-of-the-art baselines and large language models by a significant margin.

heading: 7 Limitations
text: Despite introducing meta learning for domain adaptive few-shot misinformation detection, we have not discussed the setting of cross-domain adaptation with multiple source datasets to further improve the performance for identifying early-stage misinformation. Due to the lack of early-stage misinformation data, we limit our choice of the target domain to COVID-19, which may hinder the generalization of the proposed method to other domains. Additionally, the proposed method does not leverage efficient approximation or first-order meta learning methods to reduce the computational costs in training MetaAdapt. As such, we plan to explore multi-source few-shot misinformation detection via efficient meta learning as future work.

heading: Acknowledgments
text: This research is supported in part by the National Science Foundation under Grant No. IIS-2202481, CHE-2105005, IIS-2008228, CNS-1845639, CNS1831669. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

heading: 4. Model-Agnostic Meta-Learning (MAML) is
text: the first to leverage second-order derivatives for few-shot learning. MAML first update model parameters upon sampled tasks, followed by computing the meta loss and derive the secondorder gradients w.r.t. the original parameters. Similarly, We adapt MAML to our meta adaptation framework with the homogeneous label space acorss tasks and the transformer encoder for misinformation detection (Finn et al., 2017).
A.3 Implementation Details For our evaluation method, we follow the previous works (Kou et al., 2022a; Li et al., 2021; Yue et al., 2022) and split the datasets into training, validation, and test sets with the ratio of 7:2:1. If the dataset provides a default split, we directly use the provided split sets. The validation sets are used for label correction in (Yue et al., 2022) and constructing few-shot examples and saving the best model in training. For our few-shot adaptation setting, we select the first k examples (10-shot as default) from the original validation set and used the remaining examples for validation. We use accuracy and F1 score for evaluation. The balanced accuracy (BA) is additionally introduced to evaluate the adaptation performance in both classes equally, BA is defined as the mean of sensitivity and specificity.
For the naïve baseline, we train the base model on the source dataset using AdamW optimizer (learning rate 1e-5) without warm-up. Then, the model is fine-tuned on the few-shot examples under the same training condition. For other baseline methods, we use the original implementation (if provided) and follow the original hyperparameter configuration. Otherwise we reimplement the baseline methods and adopt the identical training pipeline (AdamW with 1e-5 learning rate and no warm-up). Few-shot learning baselines (i.e., ProtoNet & MAML) are adapted to the domain adaptive few-shot learning framework by using the few-
shot target examples as query set. For the large language models, in-context learning is performed using the generative and perplexity-based approach based on (Wu et al., 2023), while fine-tuning is performed using low-rank adaptation as in (Hu et al., 2021a). In the MetaAdapt experiments, we adopt 3 as the number of tasks n and update the task specific model ϕ for 3 times in inner-level optimization, notice for each task we start from the original parameter set θ. To train MetaAdapt, we use a training batch size of 4 in each task and the learning rates (both α and β) are selected from [1e− 5, 2e− 5, 3e− 5]. The temperature hyperparameter is selected from [1, 0.1, 0.01] (See sensitivity analysis in Appendix B). We perform training with 500 or 1000 meta iterations and validate the model every 50 iterations, the best model is used for evaluation on the test sets.

heading: B Additional Results
text: Sensitivity Analysis of Initial Learning Rates: We study the sensitivity of α and β on the CoAID dataset, the best results (averaged across source datasets) are presented in Figure 3. Overall, we observe that the performance of MetaAdapt is insensitive to the changes of both learning rates. Interestingly, we notice that accuracy values are negatively correlated with the BA scores, suggesting that accuracy may not be an ideal metric for data distributions with disproportionate label classes. Sensitivity Analysis of Temperature Values: We also study the sensitivity of τ in gradient rescaling on CoAID, we present the best results (averaged across source datasets) in Figure 4. In short, the performance of MetaAdapt first increases and then starts to reduce, with the best results ocurring at τ = 0.01, indicating the effectiveness of similaritybased gradients rescaling. Overall, the proposed MetaAdapt is robust to the hyperparameters and outperforms the baseline methods consistently.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: 7 Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: 7 Limitations

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 5 Experiments

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 5 Experiments

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: 5 Experiments & A.1 Datasets

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 5 Experiments & A.1 Datasets

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? 5 Experiments & A.1 Datasets

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? A.1 Datasets
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5 Experiments & A.1 Datasets
C 3 Did you run computational experiments?

heading: 5 Experiments

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? 5 Experiments
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 5 Experiments & A.3 Implementation Details

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5 Experiments

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5 Experiments & A.3 Implementation Details
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: As such, we plan to explore multi-source few-shot misinformation detection via efficient meta learning as future work