MiDAS: Multi-integrated Domain Adaptive Supervision for Fake News Detection
Authors: Abhijit Suprem
Calton Pu
Venue: Submitted to ICLR 2023 (2023)
Keywords: multi-domain adaptation
lipschitz continuity
text classification
weak supervision
team-of-experts

Abstract: COVID-19 related misinformation and fake news, coined an 'infodemic', has dramatically increased over the past few years. This misinformation exhibits concept drift, where the distribution of fake news changes over time, reducing effectiveness of previously trained models for fake news detection. Given a set of fake news models trained on multiple domains, we propose an adaptive decision module to select the best-fit model for a new sample. We propose MiDAS, a multi-domain adaptative approach for fake news detection that ranks relevancy of existing models to new samples. MiDAS contains 2 components: a doman-invariant encoder, and an adaptive model selector. MiDAS integrates multiple pre-trained and fine-tuned models with their training data to create a domain-invariant representation. Then, MiDAS uses local Lipschitz smoothness of the invariant embedding space to estimate each model's relevance to a new sample. Higher ranked models provide predictions, and lower ranked models abstain. We evaluate MiDAS on generalization to drifted data with 9 fake news datasets, each obtained from different domains and modalities. MiDAS achieves new state-of-the-art performance on multi-domain adaptation for out-of-distribution fake news classification.  

Under review as a conference paper at ICLR 2023
MIDAS: MULTI-INTEGRATER DOMAIN ADAPTIVE SU- PERVISION FOR FAKE NEWS DETECTION
Anonymous authors Paper under doublc-blind review
ABSTRACT
Covid- 19 related misinformation and fake news, coined an 'infodemic', has dramat- ically increased over the past few years. This misinformation exhibits concept drift, where the distribution of fake news changes over time, reducing effectiveness of previously trained models for fake news detection. Given set fake news models trained on multiple domains, we propose an adaptive decision module to select the best-fit model for a new sample. We propose MIDAS, a multi-domain adaptative approach for fake news detection that ranks relevancy of existing models new encoder, and an adap- samples. MIDAS contains components: a pre-trained and fine-tuned models tive model selector. MIDAS integrates multiple pre-trate with their training data to create a domain-invariant representation. Then, MIDAS local Lipschitz smoothness of the invariant embedding space estimate each model's relevance to new sample. Higher ranked models provide predictions, and lower ranked models abstain. We evaluate MIDAS on generalization drifted data with 9 fake news datasets, each obtained from different domains and modalities. MIDAS achieves new state-of of-the-art performance on multi-domain adaptation for out-of-distiibution fake news classification.
INTRODUCTION
The misinformation and fake news associated with the COVID-19 pandemic, called an 'infodemic' by WHO (Enders et al., 2020). have grown dranatically, and evolved with the pandemic. Fake news has eroded institutional frust (Ognyanova et al. 2020) and have increasingly negative impacts outside social communities (Quinn et al. 2021). The challenge while they are raging, just like today's s online email spam filters, instead of offline, retrospective detection long after the campaigns have ended. We divide this challenge to detect fake news online into two parts: (1) the variety of data (both real and fake), and (2) the timeliness of data collection and processing (both real and fake). In this paper, we focus on the first (variety) part of challenge, with the timeliness (which depends on solutions to handle variety) in future work (Pu et al. 2020).
filter active fake news campaigns
The infodemic, and fake news more generally, evolves with a growing variety of ephemeral topics and content, phenomenon called real concept drift (Gama al.) 2014). However, the excellent results on single-domain classification (Chen et al. (021), have generalization difficulties when applied to cross-domain experiments (Wahle et al. [022]] Suprem & Pu 2022). benchmark study over 15 language models shows reduced cross-domain fake news detection accuracy (Wahle et al.) 2022). A generalization study in (Suprem & Pu 2022) finds significant performance deterioration when models are used on unseen, non-overlapping datasets. Intuitively, it is entirely reasonable that of-te-of-the-rt models trained on one dataset or time period will have reduced accuracy on future time periods. Real concept drift is introduced into fake news as content changes (Gama et al. [0014), camouflage (Shrestha & Spezzano (20TT) linguistic drift (Fisenstein al.) 2014), and adversarial adaptation by fake news producers when faced with debunking efforts such as CDC on the pandemic (Weinzierl et al. 2021).
To catch up with concept drift, the classification models need to be expanded to cover wide variety
of data sets (Li et al. 2021, Suprem Pu 2022; Kaliyar et al., 2021), or augmented with new knowledge true novelty such as the appearance of the Omieron variant (Pu et al.) 2020). In this paper, we assume the availability of domain-specifie authorative sources such as CDC and WHO that provide trusted up-to-date information on the pandemic.

---

Under review as a conference paper at ICLR 2023
A key challenge of such multi-domain classifiers is decision module to select the best-fit model amongst a set of existing models to classify new samples. This degree of knowledge is defined by the overlap between unlabeled sample and existing models' training datasets (Suprem & Pu (022). Intuitively, a best-fit model better captures a sample point's neighborhood in its own training data Urner & Ben-Davvd (213)) Chen et al. (2022).
MiDAS. We propose MIDAS, multi-domain adaptative approach for early fake news detection, with potential for online filtering. MIDAS integrates multiple pre- rre-trained and fine-tuned models along with their training data to create a domain-invarian representation. On this representation, MIDAS uses notion of local Lipschitz smoothness to estimate the overlap, and therefore relevancy, between a new sample and model training datasets. This overlap estimate is used to rank models on relevancy to the new sample. Then, MIDAS selects the highest ranked model to perform classification. We evaluate MIDAS on 9 fake news datasets obtained from different domains and modalities. We show new state-of of-the-art performance multi- adaptation for early fake news classification.
Contributions. Our contributions are as follows:
1. MIDAS: a framework for adaptive model selection by using sample-to-data overlap to measure model relevancy
2. Experimental results of MIDAS on 9 fake news datasets with state-offtte-art results using unsupervised domain adaptation.
2 RELATED WORK
2.1 MULTI-DOMAN ADAPTATION
Domain adaptation maps target domain into source domain. This allows classifier learned from the source domain to predict the target domain samples (Farahani et al. [202]). Some approaches focus on domain invariant representation between source and target (Huang al.) 2021). Then, new classifier can be trained on this invariant representation for both source and target samples. Domain invariance scalable to multiple source domains by fusing their latent representations with an adversarial mncoder-discriming
framework (Li et al.) (202T). For multi- domain adaptation (MDA), classifiers for each source have different weights: static weights using distance (Li et al. or per-sample weights on 12 norm (Suprem et al. 2020).
2021
2.2
LABEL CONFIDENCE
Alongside domain adaptation, weak supervision (WS) is also common for propagating labels from source domains to target domain (Ratner al. (017). Both approaches estimate labels closest to the true label of the target domain sample. This works with the assumption that the source domains labeling functions, respectively, are correlated to the true labels due expertise and domain knowledge. In each case, whether MDA or WS, domains or labeling functions need to be weighted to ensure reliance on the best-fit source. Snorkel, from (Ratner et al.) 2017), uses expert labeling functions and weighs them on conditional independence. Similarly, approaches in (Chen et al. 2020 Fu al. 2020) use coverage of expert foundation models and weigh on distance embedded sample. EEWS from (Rühling Cachay et al. 2021) directly combines source data and labeling function in estimator parametrization to generate dynamic weights for each sample. MDA approaches weigh sources with weak supervision (Li et al. 202I). distance (Suprem & Pu 2022), or team-of-experrs
(Pu et al. 2020).
PROBLEM SETUP AND STRATEGY
Let there be source data domains, with labels (X,, Y;) 1=1 E (D) Each of these source has an associated source model SM, with a total of lo SMs: { where we have access to the training data X; and weights w,. Each SM yields hidden embeddings through feature extractor backbone, or foundation model (Bommasani et al. 202T). Embeddings are projected to class probabilities with any type of classification layer/module.
2

---

Under review as a conference paper at ICLR 2023
(3)
Source training data
X1
Discriminator
Lp
X':
Unlabeled data
LR
B:
encoder
G:
Common embedding space
X
E
D
M1
Feats
0'
(D)
Decoders to reconstruct X
Label
Lo:
Discriminator loss
:
(L
Reconstruction loss
X
Feats
0::
Normalized L-scress
D
M
Label
r': Label matrix
Figure MIDAS architecture: encoder generates domain invariant representation. We perturbation to this representation. Then, fine-tuned models (M) (2 1)* process the sample and perturbations. After computing the local Lipschitz constant for each model, we can rank their L scores and select the best-fit model's label from the label matrix.
MIDAS adaptively weights the à SM predictions for some unlabeled, potentially drifted target data domain X'. We accomplish this by using local embedding smoothness of SMs as proxy for model relevance to sample x' E X" SMs are typically smooth in the embedding space (Urner & Ben-David 2013); further, smoothness is correlated with local accuracy (Chen et al. 2022).
With MIDAS, we rank each fk smoothness measure around the embedding for æ', i.e. f& (x'). Then, MIDAS can directly use the top-ranked fs. or the smoothest f& under a smoothness threshold. directly
the best-fit relevant models for z', with the remaining models abstaining. Because measuring smoothness on the embedding space, MIDAS can use already fine-tuned, state-of-the-ae the-art classifiers for each task, allowing off- shel,, alug-and-play usage. These classifiers are foundation models (Bommasani et al., 202T) that have been fine-tuned with architectural changes, learned weights, and hyperparameter tuning for their specific dataset.
There are two key challenges in MIDAS:
1. How do compare smoothness of SMs that have been trained on different domains?
2. How can we measure the smoothness itself for unlabeled samples in the embedding space of SMs?
We address (1) with an encoder that generates a domain invariant representation on {X; ) This unifies the data domains, allowing comparisons for different SMs start from the same source domain. For (2), we extend the idea of local Lipschitz smoothness from (Chen et al. 2022) and (Urner) Been-David 2013) randomized Lipschitz smoothness. randomized Lipschitz smoothness, we randomly perturb (xx), ), the domain invariant representation of æ'. Then, we compute the local lipschitz constant on these perturbations E(x') + e, with respect E(x') measure smoothness. This allows to calculate an L3 for each fu and use the local Lipschitz constant, a measure for the local smoothness, measure of relevancy.
4 MIDAS
We now describe MIDAS components and implementation details. The MIDAS architecture is shown in Figure I First, we cover the encoder-decoder framework generate the domain invariant representations of source and target datasets. Then we present the randomized Lipschitz smoothness measure to generate SM relevancy rankings.
4.1 DOMAIN INVARIANT ENCODER
To compare different fs relevancy, we require a common source domain (Li al. 2021). We achieve this with single-encoder multiple-decoders design, where we have single encoder to generate domain invariant representation from all source domains. Then we use decoders to reconstruct the invariant representation for each fx. TO train E, invariance with a min-max game, where the discriminator tries to identify the source domain of the invariant representation, and the encoder tries to fool it:
adversarial discriminator D' to enforce

---

Under review as a conference paper at ICLR 2023
min max D
Σ x (D)(())))
(I)
We use a gradient reversal layer R to convert the min-max to single-step minimization. It is the identity matrix during the forward pass, and the discriminator gradient during the backward pass, scaled by À -1. Then, the adversarial optimization becomes:
Loss
min E.R(D')
Σ E "x
(2)
To train DR. we use the masked language modeling loss from BERT and AIBERT pretraining. Parameters are initialized with AIBERT's a llleer-bbaeeuucaseee weights (Lan et al. 2019).
In summary, we train single encoder and ls decoders. The encoder projects our training data in the form of SentencePiece tokens (Kudo & Richardson 2018)) into nomain-invariant representation, trained with a domain discriminator. Each decoder then reconstructs the original input training tokens from the invariant representation. We use decoders because MIDAS is designed to work with our existing BERT and AIBERT classifiers, which expect SentencePiece token input. Decoders are trained with masked language modeling, where we randomly mask up to 15% of words or tokens in the input. Then, during prediction, unlabeled, potentially drifted sample x' from unseen distribution X' converted to domain invariant representation E(x'). Each decoder Dj reconstructs from this invariant representation the input to its corresponding SM fk.
4.2 RANDOMIZED LIPSCHITZ SMOOTINESS
With common embedding space, can now compare relevancy of each model an unlabeled, potentially drift sample x'. To present our approach, we need to introduce Lipschitz continuity.
Definition (Lipschitz Continuity). A function f R - RRm is Lipschitz continuous if, for some metric space (X,0), there exists constant such that
((((1), S L 02),
We can extend this to define Lipschitz-smooth with respect to SM predictions using Lipschitzness from (Chen al. 2022).
Definition (Lipschitznesss. An SM Lipszhitz smooth if, for some class label C,
Pr([((z)== Pr((((()) = c) La0(21,
(3)
That is, with La smoothness, the difference in f& predicted labels on and 32 is bounded by La for all æ E X. However, the local value of L3 can vary across the embedding space. Consequently. fa is smoother wherever Lę is smaller! As such, we want small L3 for samples in the same class, and large L; for samples from different classes.
With these definitions, can present our approach for finding the best-fit relevant fa for x', defined as the f& with the smoothest embedding space around x'.
Theorem 1.
Let the best-fit fx for a sample x' the SM that is smoothest around x'. We can find the best-fit fx for particular sample x', given distance threshold by solving:
((r(ff(Z')), Pr(fx(z,))) 0(x,
arg min
max 0( .. N
(4)
As Tag + 0, the embedding function approaches mode collapse, where every input point is projected to the embedding point
4

---

Under review as a conference paper at ICLR 2023
The max term estimates the L3 value for each fx by sampling N random points in e-Ball around x'. Then, we find the fü that has the smallest Lk-
A key insight is that adversarial attacks exploit non-smoothness of model's embedding space to fool classifiers, by generating a noise e such that fx + +) * fx()). This non-smoothness occurs when fü does not capture enough training data in the region around x properly; in GANs, this causes 'holes' in the latent space (Suprem et al. 2020) during image synthesis. Conversely, adversarial defenses either enforce smoothness around embedding space on potentially perturbed inputs themselves (Das et al., 2018). Similarly, GANs can enforce I-Lipschitzness to improve coverage of sample generation Qin al., 2020).
So, given several fa with different local Lı: around the embedding fa + e), lower LN indicates smoother embedding space because that SM has captured more training data in the region surrounding relative to other SMs, similar to the overlap metric calculated in (Suprem & Pu 2022) However, even with a low Lg. the classification labels yis ff (z)). obtained from the classification module fg of the kth SM, can change on perturbations around We can use probabilistic Lipschitzness to bound the probability of yı changing in the neighborhood x + e as a function of the perturbation c.
Definition (Probabilistic Lipschitzness). Let Φ R + [0, 1]. Given x', Ir ~ Px, we say that fx is ø-Lipschitz if, for all 0, there is an increasing function Φ(c) such that:
(zf( f((x,) - (0x 2.) > 0 S ((() af Px
Pr
(5)
That is, function that is Lipschitz by Definition (L--Lipschitz) satisfies Definition 3 (é-Lipschitz) with (eE) 1 i 1/L and @(E) = e < I/L.
From this, it follows that if fx satisfies the d-Lipschitz condition, then the number of samples within an e-Ball of x' that have different label from x' is bounded by die). per (Urner & Ben-David 2013). As we move further from , the probability of a label change increases. Let a' be the accuracy for fk at x'. If we know the accuracy drop at the edge of the e-Ball where labels change values, we can bound the accuracy of predictions between x' and some perturbed point v, in an c-Ball around x' to:
a 2 d' - a Φ(c)
If we calculate for SM using training examples of different labels within the margins allowed by probabilistic Lipschitzness, we find that accuracy bound depends only on choice of e. Consequently can approximate f& 's accuracy on x! if xr is within small e-Ball around x' to be at least:
((P(A(())))PP((((a.)) Pr(fa(x') yk) Φ(e)
(6)
4.3 PUTTING IT TOGETHER: MIDAS
Recall that the encoder projects all samples to a common invariant domain E {X) - G. Each SM-specific decoder D; then converts from the invariant domain to the k-th source domain D& : G + XA. With this encoder-decoders framework, we can use the invariant domain G as the common source domain for all SMs.
Per Definition 3, accuracy is best estimated in an < 1 ball. Once E is trained, we can use the class cluster centers (obtained from K-Means clustering on the embeddings) from the training data to compute a local cluster cluster-specific L for each cluster in each ft. This partitioning allows us to compute local smoothness characteristics of the embedding space, simialr to the concurrent work in (Chen) et al. 2022). We use the cluster centers because we want the strongest measure of smoothness for each k. and this occurs near the cluster center; for example (Suprem & Pu, 2022) and (Chen et al.. 2022) both use the centroid to set up accuracy thresholds. Then, can estimate local La for each fü at the cluster centers with the m nearest point to the center using Equation (3) We take the maximum L among all fü to obtain the upper bound on the local smoothness among the SMs. On this maximum, we can calculate number of neighbors to compute the local L3 for each label cluster each ft. We explore impact of in SSection 5.2)
1/max ( ). So, the only hyperparameter here is m, the
5

---

Under review as a conference paper at ICLR 2023
Oracle Acc. (Fine-Tuning)
Generalization Acc.
Dataset
Training
Testing
Decreasc
(Held-Out)
kaggle short
Patel
2021
31k
9K
0,97
0.57
kaggle long
Patel
2021
31k
9K
0.98
0.53
46%
coaid (Cui & Lee 2020)
5K
IK
0.97
0.56
42%
cov19_ Text Agarwal 2020
2.5K
0.5K
0.98
0.58
41%
cov19 _tsttttlee___ Agaawwl
2020
2.5K
0.5K
0.95
0.62
35%
rumor (Cheng al.
|2021
2021
4.5K
IK
0.83
0.54
35%
eq (Muilu et al. 2020)
12.5K
2K
0.71
0.51
28%
miscov19 19 (Memon & Carley 2020
4K
0.6K
0.68
0.50
26%
covid fake (Das al.
2021)
4K
2K
0.96
0.52
45%
We use 9 covid fake news datasets to evaluate MiDAS. Here we also present a motivating experiment with respect to concept drift and generalizability: for each dataset, we train an 'oracle' on the training data. The oracle's performance concept drift case, where models need to generalize data distributions they have not yet encountered. There significant accuracy drop due to concept drift, domain shift, label overlap
Table 1:
the test data is compared to ensemble of the other 8 datasets. The latter tests the
Now, for unlabeled sample x', first generate the domain invariant representation z'c E(x'). Then we perturb z'g to generate r points { in an e-Ball around z'g. Using z'c and
compute the local Lipschitz constant T42 for each fa using Equation (3) We have 2 possibilities:
{ura)) y 1
1. 4 2 1/e: fa that satisfy this condition abstain from providing predictions, since the accuracy
drop is unbounded.
2. 14 < <//e: fa that satisfy this condition can provide labels, because their accuracy is bounded per Equation (6)
EVALUATION
5.1 EXPERIMENTAL SETUP AND DATASETS
We implemented and evaluated MIDAS PyTorch 1.11 server rumning NVIDIA T100 GPUs. We have released our implementation code.
MIDAS Datasets. We use fake news datasets, shown in Table I Where available, we used
provided train-test splits; otherwise, we performed class-balanced 70:30 splits. We performed preliminary motivating evaluation, shown in Table I Here, we have compared an oracle case to the concept drift case. In oracle case, we train models, one on each dataset, and then evaluate this model on its corresponding dataset's 's test set. This is the case where the prediction data matches the training distribution. the concept drift case, we trained model on datasets, then evaluated on the held-out dataset. In this the prediction dataset, even though on the same topic of Covid-19 fake news detection, does not match the training distribution. We see significant accuracy drops, between 20% 50%. This matches the observations in generalizability in (Suprem & Pu 2022) and (Wahle et al. 2022).
MiDAS Evaluation. We evaluate MIDAS with held-one-out testing at the dataset level similar to the generalization studies approach in (Suprem & Pu 2022). Our results are presented in /eeciinn 5.2 To test MIDAS, we first train the encoder to learn a domain invariant representation with all but one dataset. Then evaluate MIDAS' performance on classifying the unseen, i.e. drifted, dataset. We repeat this for each dataset in Table 1 MIDAS' performance is compared to Snorkel (Ratner et al. 2017). EEWS (Rühling Cachay et al. 2021) an ensemble, and an 'oracle" fine-tuned AIBERT model trained on the held-out dataset.
Ablation Study. To further evaluate MIDAS' efficacy, also conducted ablation study in Section by varying the number training datasets, types of loss functions, masked language model trainig, and loss weights.

---

Under review as a conference paper at ICLR 2023
Label overtep befen ME55
0.25
80
r-Bal --Ball radius
m==50
L-Valve
60
0.35
0.05
100
150
100
#Noorest Neighbors
Center
#earrsst Neightors Clister Center
(a) Overlapping clus- (b) Domain invariance ters before MiDAS after MiDAS
(c) L-Values for different w
(d) c-Ball radius for different w
Figure 2: MIDAS characteristies. Figure 2a shows tSNE projection of all 9 datasets' pretrained BERT encoder's embeddings. FFigure 26 shows dataset emheddings when generated with MiDAS' encoder. The label overlaps between true news and lake news have been separated; in this case, enforcing domain invariance forces true and fake labels of all datasets to cluster. In [Figure 2c and Figure 2d] we show impact of L- L-Value and resulting calculation for different values of m. We select m 50 for remaining experiments.
Adjustment of m. For each experiment in Section 5.2] we sampled points in an e-ball around æ, where calculated using steps Section 4.3 We explore effects of adjusting the radius of this sampling ball by changing m, the number of nearest neighbors, in [Section 5.4] Specifically, we show that as the sampling/perturbattno
radius increases beyond c, MIDAS accuracy decreases. Conversely, as sampling radius is reduced, MIDAS increases accuracy while sacrificing coverage.
5.2 MIDAS EVALUATION
We now present evaluation results for MIDAS. In each experiment, we designate a single dataset as the target dataset without labels, and the remaining datasets act as source domains. In these cases, the held-one-out dataset acts as the drifted dataset, similar to the generalization experiments in (Suprem & Pu 2022). We follow the steps in SSection 4.3 test MIDAS, and compare classification accuracy to approaches: (i) an ensemble of the training models with equal weights, (ii) a Snorkel labeler (Ratner et al. 2017) that treats each model as a labeling function, (iii) an EEWS labeler (Rühling Cachay et al. 202T) that treats each model as labeling function, (iv) a KMP-model (Suprem & Pu 2022) that uses KMeans clustering with proxies to compute overlap, and (v) 'oracle' AIBERT model fine-tuned on the held-out dataset.
Figure 2 shows several characteristics of the MiDAS encoder. Figure 26 shows domain invariance in
the labels. Each point is a sample from the datasets; before applying MiDAS, there is significant label overlap because each dataset exists in separate domains. After applying MiDAS, datasets are projected to a domain invariance embedding. This forces sampes with the same label, irrespective of source domain, to cluster together and reduce the label overlap observed in (Suprem & Pu] 2022).
After training MIDAS' encoder, we need to compute the e radius using m nearest neighbors to the label cluster centers. We examine the impact of different m values for computing e in innuue 2c and Figure 2d As we increase the number of neighbors used in estimating local L, the estimate for L increases, indicating reduced smoothness the further we deviate from the cluster center. In turn, this reduces the acceptable e-ball radius to bound probability of label change, per (Umer & Ben-David 2013), since 1/L. large m would significantly reduce the size of the sampling e-ball, and perturbations would be negligible. A small m would yield a poor estimate for local I, and large sampling ball. We further explore impact of changing m directly on accuracy in [Section 5.4 Here, we fix 50 for remaining experiments, since observed the e- e-Ball radius generally stabilized around this value.
In Table 2 we show results of MIDAS compared to the 5 approaches discussed above. In all but one of our experiments, MIDAS outperforms other labeling schemes in classifying the unseen, drifted samples. On average, MIDAS sees 30% increase in accuracy compared to ensemble. Further, by using the training data itself to adaptively guide SM selection, MiDAS improves by 21 % on Snorkel, 10% EEWS and KMP.
7

---

Under review as a conference paper at ICLR 2023
Oracle Labels
access Labels
Dataset
FT-AlBert
Ensemble
Snorkel
EEWS
KMP
MiDAS
Baggle_short
0.97
0.57
0.61
0.67
0.69
0.86
kaggle_long
0.98
0.53
0.63
0.68
0.70
0.71
coaid
0.97
0,56
0.64
0.74
0.81
0.84
cov19_teet
0.98
0.58
0.59
0.68
0.75
0.79
cov19_tin title
0.95
0.62
0.69
0.75
0.61
0.81
0.83
0.54
0.59
0.62
0.45
0.67
cq miscov 19
0.71
0.51
0.54
0.56
0.68
0.50
0.52
0.56
0.45
0.54
covid fake
0.96
0.52
0.56
0.61
0.50
0.75
Table 2: MiDAS Evaluation: MiDAS outperforms generalizing unseen, drifted data points. For each held-ou dataset, given fine-tuned models trained on the remaining datasets, MiDAS able select best-fit model for each sample. Using this best-fit model, MiDAS outperforms an seiuhte ensemble by over 30%.
Dataset
MiDAS-Half
+Center
Masking
+Weighted
MIDAS Encoder Convergence
Kagele_ short kaggle_long
0.56
0.75
0.79
MIDAS-Half
0,55
0.62
0.65
0.68
0.71
Sources
coaid cov19_tcxt
0.53
0.71
0.76
0.79
0.84
Center-Loss
0.54
0.72
0.75
0.79
Masking
Weighted-Loss
cov19_ @ww99_itle title
0.58
0.71
U.76
0.78
0.81
rumor
0,52
0.54
0.55
0.55
0.57
19scov19
0.51
0.53
0.54
0.54
caked_Take
0.51
0.59
0.64
0.69
0.75
0 100 200 300 400 500 600 700
Training
Table 3: MIDAS Ablation Study. We examined impact of different design choices here. Of note is that using masked language modeling is significantly useful improving end-to-end accuracy. Further, adding a center term ensures the domain- domain-invariance representations have separable clusters, as we see in Figure 2b] Finally, with weighted loss, we give the discriminator 2x the importance of the encoder masking loss to focus domain invariance.
Figure 3: Convergence of Mi- DAS with each component in ablation study.
5.3 ABLATION STUDY
We evaluated the impact of several design and training choices for MIDAS in an ablation study in Table 3 We use a version MiDAS trained with half of the sources with the most data points for each experiment ensemble on different source datasets. Adding the remaining sources improves MiDAS' coverage and improves accuracy by 15%. We add center loss term (He al., 2018) to the encoder output to encourage clustering on the labels between multiple sources; increasing accuracy by 5%. Next, added language masking the input during the encoder-decoder training further fine-tune the encoder for the fake-news tasks, yielding a 4% improvement, Finally, we increased the weights for the discriminator loss compared encoder loss to emphasize domain invariance, yielding 5% improvement for MiDAS' accuracy on fake news detection for unseen, drifted data. We compare convergence for different experiments in in/Figure 3] the encoder converges faster in each case. Further, adding the center and weighted losses contribute to discriminator fooling and stabilizing the discriminator loss.
MIDDAS-Half) This yields near-random accuracy, since this is modified
5.4 ADJUSTMENT OF
Finally, we investigate with respect to m. which was fixed at m=50. For these experiments, we investigated increasing and decreasing m to, respectively, increase and decrease e. Increasing the neighbors increases the computed L, since we using points further from the smooth cluster center. In turn, this reduces the sampling e-ball, so the perturbations we apply will smaller, and some cases, negligible. Furthermore, threshold value of L needed accept an SM's prediction is higher (since is 1/e), so MiDAS tolerates lower smoothness for each model, and accepts predictions from more models, resulting in higher coverage and lower overall accuracy. On the other hand, using fewer
8

---

Under review as a conference paper at ICLR 2023
m-value
kaggle short Coverage
coaid Coverage
rumor
cq Coverage
F1-Score
F1 F1-Score
F1-Score
Coverage
m=1
0.97
0.03
0.97
0.01
0.86
0.05
0.81
0.03
10=10
0.91
0.35
0.92
0.29
0.82
0.48
0.74
0.34
m=20
0.87
0.65
0.85
0.73
0.73
0.63
0.61
0.59
m=50
0.86
0.86
0.84
0.89
0.67
0.82
0.57
0.78
m=75
0.73
0.91
0.78
0.94
0.62
0.89
0.53
0,85
m=100
0.69
0.95
0.73
0.98
0.59
0.97
0,51
0,91
m=150
0.57
0.99
0.56
1.00
0.54
1.00
0.43
0.95
Table 4: Impact of values: As inerease the number of nearest neighbors, get higher estimate for L, per Figure 2c] which reduces the e-Ball sampling radius and relaxes threshold for model predictions. This leads lower accuracy with higher coverage. Decreasing m, turn, increases accuracy at cost lower coverage.
neighbors means larger sampling ball and smaller threshold for acceptance. is more likely for perturbed samples to be further away, yielding higher value of L unless corresponding model is especially smooth around that point. This would, as result, reduce coverage, but increase accuracy.
We show this in Table for several values of across 4 of our datasets. Using only the nearest neighbor yields minimal coverage. As we increase the m, coverage increases significantly, and accuracy approaches ensemble accuracy. Conversely, as we reduce m, also reduce and consequently, the smoothness threshold to accept prediction. This increases accuracy, since MIDAS rejects predictions that not satisfy the threshold. However, coverage decreases as well: we show in Table 4 fewer unseen samples from the target domain can be labeled as we decrease that m can have outsized impact on accuracy as well: 'coaid' fl scores drop from 0.73 to 0.56, even though coverage increases only slightly, from 0.98 1.0 when we increase from 100 to 150. This occurs because at m = 150, MiDAS' relaxed thresholds allow poorer models to provide predictions well, redueing accuracy in the final averaged result.
We also see
5.5 LIMITATIONS AND FUTURE DIRECTIONS
We tested MiDAS in the scenario where fine-tume sine-tundddee models already exist. This constrains the MiDAS encoder, which must also train decoder match the inputs of the fine-tuned models that expect tokenized input. A more flexible approach would deploy models and MiDAS together, with each fine-tuned model directly trained with the MiDAS encoder. This would improve both training time, convergence, as well as accuracy, since each model would directly use the MiDAS generated encodings, instead of domain-specific reconstructions.
Furthermore, we selected m using empirical observations. However, there can be technically grounded approaches well, such using the high-density bands from (Suprem et al. 2020 Diang al. 2018). In these cases, the h-hh density region of each cluster can be used to estimate a good m. We leave further exploration of m as well integration of fine-tuned models into the encoder training framework to future work.
6 CONCLUSION
We have presented MIDAS, system for adaptively selecting best-fit model for of samples from drifting distributions. MIDAS uses domain-invariann embedding to estimate local smoothness for fine-tuned models around drifting samples. By using local smoothness as a proxy for accuracy and training data relevancy, MIDAS improves on generalization accuracy across 9 fake news datasets. With MIDAS, we can detect COVID-19 related fake news with over 10% accuracy improvement weak labeling approaches. We hope MIDAS will lead further exploration into the tradeoff between generalizability and fine-tuning, as well as research into mitigating generalization difficulties of pre-trained models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Isha Agarwal. Covid 19 Fake News Dataset. 6 2020. doi: figssaar. 12489293.
vl. URL https:///igshar cwte.ccoaarrtt cccm///rticclee//aassttttee/eeee
Da taset
CSV 12489293
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0722 07258, 2021.
Ben Chen, Bin Chen, Dehong Gao, Qijin Chen, Chengfu Huo, Xiaonan Meng. Weijun Ren, and Yang Zhou. Transformer-basee language model fine-tuning methods for covid-19 fake news detection. In International Workshop on Combating Hostile Posts in Regional Languages during Emergency Situation, pp. 83 92. Springer, 2021.
Mayee Chen, Daniel Fu, Frederic Sala, Sen Wu, Ravi Teja Mullapudi, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Train and you'll miss it: Interactive model iteration with weak supervision and pre-trained embeddings. arXiv preprint arXiv:2006. 15168, 2020.
Mayee F Chen, Daniel Y Fu, Dyah Adila, Michael Zhang, Frederic Sala, Kayvon Fatahalian, and Christopher Ré. Shoring up the foundations: Fusing model embeddings and weak supervision. arXiv preprint arXiv:2203. 13270,, 2022.
Mingxi Cheng, Songli Wang, Xiaofeng Yan, Tianqi Yang, Wenshuo Wang, Zehao Huang, Xiongye Xiao, Shahin Nazarian, and Paul Bogdan. A covid-19 rumor dataset. Frontiers in Psychology, 12: 1566, 2021.
Limeng Cui and Dongwon Lee. Coaid: Covid-19 healthcare misinformation dataset. arXiv preprint arXiv:2006. 200.0000600888,, 2020.
Nilaksh Das, Madhuri Shanbhogue. Shang-Tss -se Chen, Fred Hohman, Siwei Li, Li Chen, Michael Kounavis, and Duen Horng Chau. Shield: Fast, practical defense and vaccination for deep learning using jpeg compression. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 196 204, 2018.
Sourya Dipta Das, Ayan Basak, and Saikat Dutta. A heuristic-driven uncertainty based ensemble framework for fake news detection in tweets and news articles. Neurocomputing, 2021.
Jacob Eisenstein, Brendan O' Comnoor, Noah A Smith, and Eric P Xing. Diffusion of lexical change in social media. PloS one, 9[11):111311 2014. ISSN 1932-6203.
Adam M Enders, Joseph E Uscinski, Casey Klofstad, and Justin Stoler. The different forms of covid-19 misinformation and their consequences. The Harvard Kennedy School Misinformation Review, 2020.
Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in Data Science and Information Engineering, pp. 877-894, 2021.
Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper. Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In International Conference on Machine Learning, pp. 3280-320 3291.. PMLR, 2020.
Joao Gama, Indre Zliobaite, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. survey on concept drift adaptation. ACM computing surveys (CSUR), 46(4):44, 2014. ISSN 0360-0300.
Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. Triplet-center loss for multi-view 3d object retrieval. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1945-1954, 2018
Yinqiu Huang, Min Gao, Jia Wang. and Kai Shu. Dafd: Domain adaptation framework for fake news detection. In International Conference on Neural Information Processing, pp. 305-316. Springer, 2021.
10

---

Under review as a conference paper at ICLR 2023
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust classifier. Advances in neural information processing systems, 31, 2018.
Rohit Kumar Kaliyar, Anurag Goswami, and Pratik Narang. Mennet: generalizing fake news detection with multichannel convolutional neural network using novel covid-19 dataset. In 8th ACM IKDD CODS and 26th COMAD, pp. 437-437. 2021.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:/808.062: 06226, 2018.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1 19922, 2019.
Yichuan Li, Kyumin Lee, Nima Kordzadeh, Brenton Faber, Cameron Fiddes, Elaine Chen, and Kai Shu. Multi-source domain adaptation with weak supervision for early fake news detection. In 202/ IEEE International Conference on Big Data (Big Data), pp. 668-676. IEFF, 2021.
Shahan Ali Memon and Kathleen M Carley, Characterizing covid-19 misinformation communities using a novel twitter dataset. arXiv preprint arXiv:2008.0077 00791, 2020.
Ece C Mutlu, Toktam Oghaz, Jasser Jasser, Ege Tutunculer, Amirarsalan Rajabi, Aida Tayebi, Ozlem Ozmen, and Ivan Garibay, A stance data set on polarized conversations on twitter about the efficacy of hydroxychloroquine as a treatment for covid-19. Data in Brief, pp. 106401, 2020.
Katherine Ognyanova, David Lazer, Ronald E Robertson, and Christo Wilson. Misinformation in action: Fake news exposure is linked to lower trust in media, higher trust in government when your side is in power. Harvard Kennedy School Misinformation Review, 2020.
Sameer Patel. Covid-19 Fake News Dataset (Kaggle). 2021. URL httpt::////// www www.wa.gglecoom tode//thereallsmaat/EEeee
news detection/
Calton Pu, Abhijit Suprem, Rodrigo Alves Lima, Aibek Musaev, De Wang, Danesh Irani, Steve Webb, and Joao Eduardo Ferreira. Beyond artificial reality: Finding and monitoring live events from social sensors. ACM Transactions on Internet Technology (TOIT), 20(1):1-21, 2020.
Yipeng Qin, Niloy Mitra, and Peter Wonka. How does lipschitz regularization influence gan training? In European Conference Computer Vision, pp. 310-326. Springer, 2020.
Emma K Quinn, Sajjad S Fazel, and Cheryl E Peters. The instagram infodemic: cobranding of conspiracy theories, coronavirus disease 2019 and -uthorit--uesstoning beliefs. Cyberpsychology, Behavior, and Social Nerworking, 24(8):573-577, 2021.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 11, pp. 269, NIH Public Access, 2017.
Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. End-to-end weak supervision. Advances in Neural Information Processing Systems, 34, 2021.
Anu Shrestha and Francesca Spezzano. Textual characteristics of news title and body to detect fake news: a reproducibility study. In European Conference on Information Retrieval, pp. 120-133. Springer, 2021.
Abhijit Suprem and Calton Pu. Exploring generalizability of fine-tuned models for fake news detection, 2022.
Abhijit Suprem, Joy Arulraj, Calton Pu, and Joao Ferreira. Odin: Automated drift detection and recovery in video analytics. Proc. VLDB Endow., 13(12), 2020. ISSN 2150-8097. URL https //doi . www.ooo........... 1ET.11177870900
3407837
Ruth Urner and Shai Ben- Den-D. Probabilistic lipschitzness a niceness assumption for deterministic labels. In Learning Faster from Easy Data- Workshop@ @ NIPS, volume 2, pp. 1, 2013.

---

Under review as a conference paper at ICLR 2023
Jan Philip Wahle, Nischal Ashok, Terry Ruas, Norman Meuschke, Tirthankar Ghosal, and Bela Gipp. Testing the generalization neural language models for covid-19 misinformation detection. In International Conference Information, pp. 381-392. Springer, 2022.
Maxwell Weinzierl, Suellen Hopfer, and Sanda M Harabagiu, Misinformation adoption or rejection in the era of covid-19. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM), AAAI Press, 2021.
12


Reviews:
[{"review_id": "kG_HHhZTXU", "reviewer": "Reviewer_WVAw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach, MiDAS, to classify new samples in the situation where there are multiple domains and domain concepts drift over time. The proposed method has two components: a domain-invariant encoder, and an adapetive model selector. The encoder creates domain-invariant representations for data samples and the selector uses Lipschitz smoothness to estimate the best-fit model for a new sample. Experimental evaluation is performed on several fake-news datasets.\n", "summary_of_the_review": "This paper presents a solution for classifying samples when multiple source domains are present and concept drifts exist. The target application is fake news detection, however, the connection of the proposed framework with this specific application is not strong. The motivation for using Lipschitzness should be better described.\n", "strength_and_weaknesses": "Strengths:\n\n+This paper introduces the problem of concept drift in misinformation clearly. The motivation of designing a domain-adaptive approach is clear.\n\n+The proposed framework is well described with figures and details. The first part, the encoder, aims to learn domain-invariant representations for data samples. Based on the learned representations, this paper introduces the Lipschitz continuity and extends this to find the best-fit model from multiple source models. \n\n+Experiments are performed on multiple fake news datasets to test the performance of the proposed approach on classifying unseen/drifted distributions. Ablation studies are performed by changing the number of training datasets, types of loss functions, masked language model training, and loss weights.\n\nWeaknesses:\n\nFirst, this paper focuses on the application of fake news detection. However, the connection between the proposed model with fake news seems not strong. Very little analysis of fake news data distributions is provided. Datasets are not described in detail. The unique challenges in fake news detection are not introduced. If the proposed method is a general approach for any concept-drift problems, the experiments are only conducted on fake-news datasets.\n\nSecond, the motivation for using Lipschitz smoothness to estimate/find the best source model/function for an unlabeled and potentially drifting sample is not strong. The reasons for using this notion should be better elaborated.\n\nThird, some notations can be better clarified. For instance, how is function theta specified is not clearly described in the methodology section. Equation 3 seems to use absolute difference, but the reason why this can be used for estimating the smoothness of source functions remains unclear. The motivation for using probabilistic Lipschitzness can be better explained. It would be more convincing if examples or proofs of changes in perturbations are provided.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gTAtW2vzt1w", "reviewer": "Reviewer_i2cY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors in this paper present Multi-Integrated Domain Adaptive Supervision (MiDAS), a framework for fake news detection that ranks the relevance of existing models to new samples. MiDAS learns domain-invariant representations by integrating multiple pre-trained and fine-tuned models with their training data. To determine each model's relevance to a new sample, MiDAS applies local Lipschitz smoothness of the invariant representation in latent space. Empirical results show MIDAS improves significantly on generalization accuracy across 9 diverse fake news datasets.", "summary_of_the_review": "This paper presents a multi-domain adaptative approach for early fake news detection, which integrates pre-trained and fine-tuned models along with their training data to learn a domain-invariant representation. In this representation, the proposed framework adaptively selects the highest-ranked model to perform classification according to the relevancy between a new sample and model training datasets.\n\nIn general, the idea of using randomized Lipschitz smoothness measure to generate source model relevancy rankings is reasonable and effective for fake news detection. \n\nHowever, the contribution of the paper is incremental given the prior work local Lipschitz smoothness (Chen et al., 2022). \n\nThere are missing technical details:\n1. What are the t-test results for Table 2 and 3? \n2. What are the results in terms of other metrics such as F1 and coverage for Table 2 and 3? \n3. Complexity analysis of the proposed method? \n\nTo demonstrate the scalability and efficiency of the proposed method, evaluation on much larger datasets is recommended. ", "strength_and_weaknesses": "Strength:\n\n1. This paper focuses on an interesting problem, the detection of fake news, especially Covid-19 news. It is impactful and has a sociological significance. \n\n2. The idea of extending local Lipschitz smoothness in measuring the relevance of a new sample to existing models is somewhat novel. \n\n3. The paper is illustrative with an ablation study by varying the number of training datasets, types of loss functions, masked language model training, and loss weights. \n\nWeakness:\n\n1. The main idea of this paper is based on prior work: local Lipschitz smoothness from (Chen et al., 2022). However, comparison results with the method  LIGER from (Chen et al., 2022) are missing. \n\n2. The sizes of the datasets used in the evaluation are quite small and the numbers of samples are in the thousands. To demonstrate the scalability of the proposed method, much larger datasets are recommended. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PsBoL61RMFq", "reviewer": "Reviewer_5EyG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel approach to  the problem of out-of-domain  texts for fake news classification.  The method involves an adaptive model selector which employs Lipschitz smoothness concept to estimate model's relevancy.  \nThe paper shows superiority over a number of baselines and previously introduced methods.  \n\n", "summary_of_the_review": "The main observation about this paper is that it seem to introduce a general method applicable to text classification, yet only verifies it for quite a narrow, albeit important, area of covid-related fake news detection. \n", "strength_and_weaknesses": "strengths\n* the paper proposes a novel approach  that is superior to the presneted previosu approaches \n* the paper provides concise overview of the Lipshitz smoothness concept to explain its efficiency for the problem \n* the analysis is conducted on multiple test sets \n* ablation study and parameter analysis provides additional soundness to the paper  \n\n\nweaknesses: \n* more as a point of observation, the propsed method  does not use any features that makes it specifivcally focused on fake news detection or COVID dataset. It could have been applied and verified on groups of datasets for a variety of classificationn tasks. That could have made the work more impactful, and the empirical groundings more sound. This is my major concern towards this work: if the porposed method is correct, it should generalize across any text classification tasks (that satisfy Lipschitz smoothness requirements) and applying it towards a certain subrgroup of text classification (even as \"trendy\" as fake news and covid fake news in particular) diminishes the potential contribution, making the paper more apt for a focused workshop rather than the main conference venue.  \n\n* there's a few novel points introduced in the paper: a use of certain architecture, application of concept of Lipschitz smoothness, parameter analysis. However, there's some lack of clarity. The paper would benefit from more detailed explaining  why certain choices were made, what is the intended scope of the method, does it imply any retrsictions on the data or the models involved, etc?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "K7jTGF4RW0", "reviewer": "Reviewer_35Bs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces MiDAS, a novel method to address the problem of out-of-domain texts for fake news classification. MiDAS applies an adaptive model selector that utilizes the Lipschitz smoothness idea to estimate the model's relevancy. The author shows supremacy over several baselines and previously introduced methods.", "summary_of_the_review": "In order to learn a domain-invariant representations, the multi-domain adaptive method for premature false news detection presented in this study combines pre-trained and improved models with their training examples. According to the similarities between a new sample and model training datasets, the designed system adaptively selects the highest-ranked model to carry out classification in this representation. In general, it makes perfect sense and performs well for detecting fake news to use randomized Lipschitz smoothness metric to produce source model relevancy rankings.\n\nA few novel points are introduced in the paper: the use of particular architecture, the application of the concept of Lipschitz smoothness, and parameter analysis. However, there is some lack of clarity. The paper would benefit from a more detailed explanation of why confident choices were made, the intended scope of the method, does it imply any restrictions on the data or the models involved, etc.\n", "strength_and_weaknesses": "Strength:\n1. This paper focuses on an exciting problem, detecting fake news, especially COVID-19 news. It is impactful and has sociological value.\n2. Using local Lipschitz smoothness is a new way to figure out how well a new sample fits existing models, which is very exciting.\n3. An ablation study is used as an example in the study. The number of training datasets, types of loss functions, masked language model training, and loss weights are all changed.\n\nWeakness:\n1.\n2.\n3.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject