Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces MiDAS, a novel method to address the problem of out-of-domain texts for fake news classification. MiDAS applies an adaptive model selector that utilizes the Lipschitz smoothness idea to estimate the model's relevancy. The author shows supremacy over several baselines and previously introduced methods.
strength_and_weaknesses: Strength:
1. This paper focuses on an exciting problem, detecting fake news, especially COVID-19 news. It is impactful and has sociological value.
2. Using local Lipschitz smoothness is a new way to figure out how well a new sample fits existing models, which is very exciting.
3. An ablation study is used as an example in the study. The number of training datasets, types of loss functions, masked language model training, and loss weights are all changed.

Weakness:
1.
2.
3.
clarity,_quality,_novelty_and_reproducibility: Given the prior work on Lipschitz smoothness by Chen et al., (2022), the novelty of this paper is very small. It will be great if the author could compared their work with the work from the reference above.
The author should look through the problem setup and strategy (Section 3):  
"... where we have access to the training data Xi and weights wi." - Could you provide a more elaborated definition on "w_i"? Using weights as model parameters? "Each SM yields hidden embeddings through a feature extractor backbone or foundation model"—this implies certain assumptions about the source models. This should be stated clearly in the scope of this work.-- please check~
summary_of_the_review: In order to learn a domain-invariant representations, the multi-domain adaptive method for premature false news detection presented in this study combines pre-trained and improved models with their training examples. According to the similarities between a new sample and model training datasets, the designed system adaptively selects the highest-ranked model to carry out classification in this representation. In general, it makes perfect sense and performs well for detecting fake news to use randomized Lipschitz smoothness metric to produce source model relevancy rankings.

A few novel points are introduced in the paper: the use of particular architecture, the application of the concept of Lipschitz smoothness, and parameter analysis. However, there is some lack of clarity. The paper would benefit from a more detailed explanation of why confident choices were made, the intended scope of the method, does it imply any restrictions on the data or the models involved, etc.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a novel approach to  the problem of out-of-domain  texts for fake news classification.  The method involves an adaptive model selector which employs Lipschitz smoothness concept to estimate model's relevancy.  
The paper shows superiority over a number of baselines and previously introduced methods.  


strength_and_weaknesses: strengths
* the paper proposes a novel approach  that is superior to the presneted previosu approaches 
* the paper provides concise overview of the Lipshitz smoothness concept to explain its efficiency for the problem 
* the analysis is conducted on multiple test sets 
* ablation study and parameter analysis provides additional soundness to the paper  


weaknesses: 
* more as a point of observation, the propsed method  does not use any features that makes it specifivcally focused on fake news detection or COVID dataset. It could have been applied and verified on groups of datasets for a variety of classificationn tasks. That could have made the work more impactful, and the empirical groundings more sound. This is my major concern towards this work: if the porposed method is correct, it should generalize across any text classification tasks (that satisfy Lipschitz smoothness requirements) and applying it towards a certain subrgroup of text classification (even as "trendy" as fake news and covid fake news in particular) diminishes the potential contribution, making the paper more apt for a focused workshop rather than the main conference venue.  

* there's a few novel points introduced in the paper: a use of certain architecture, application of concept of Lipschitz smoothness, parameter analysis. However, there's some lack of clarity. The paper would benefit from more detailed explaining  why certain choices were made, what is the intended scope of the method, does it imply any retrsictions on the data or the models involved, etc?
clarity,_quality,_novelty_and_reproducibility: **Clarity**

Given that the paper introduces complex ideas, I'd suggest providing more details about the  approach to solution as well as the proposed method in a few sections. Some material may0Car be   

Introduction

"This degree of knowledge is defined by the overlap between an unlabeled sample and existing models’ training datasets"  - overlap in what sense?

3 PROBLEM SETUP AND STRATEGY

"... where we have access to the training data Xi and weights wi." - Could you provide a more elaborated definition on "w_i"?  weights as model parameters?

"Each SM yields hidden embeddings through a feature extractor backbone, or foundation model" - this implies certain assumpton sabout the Source Models. This should be stated clearly in the scope fo this work.


4.

"we use the masked language modeling loss from BERT and AlBERT pretraining."  - please, explain the choice of the LMs


4.2 RANDOMIZED LIPSCHITZ SMOOTHNESS - I'd suggest moving this section partially or entirely to apendix to provide more space for explaning some MiDAS specific details as well as other details o fthe current work to increase clarity.   

5.

"NVIDIA T100 GPUs."  - I was not able to verify Nvidia T100 GPUs (there are V100 and T1000 or T400). Either there's a typo, or  please provide a footnote with more details if it's a speacial case.


"an ‘oracle’ fine-tuned AlBERT model trained on the held-out dataset." - if AlBERT is used for the oracle, is it BERT's or AlBERT's LMs that are used fro MiDAS.


**Novelty**

From what I could judge by related work section and the refernces, the paper introduces a novel and original approach. 


summary_of_the_review: The main observation about this paper is that it seem to introduce a general method applicable to text classification, yet only verifies it for quite a narrow, albeit important, area of covid-related fake news detection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors in this paper present Multi-Integrated Domain Adaptive Supervision (MiDAS), a framework for fake news detection that ranks the relevance of existing models to new samples. MiDAS learns domain-invariant representations by integrating multiple pre-trained and fine-tuned models with their training data. To determine each model's relevance to a new sample, MiDAS applies local Lipschitz smoothness of the invariant representation in latent space. Empirical results show MIDAS improves significantly on generalization accuracy across 9 diverse fake news datasets.
strength_and_weaknesses: Strength:

1. This paper focuses on an interesting problem, the detection of fake news, especially Covid-19 news. It is impactful and has a sociological significance. 

2. The idea of extending local Lipschitz smoothness in measuring the relevance of a new sample to existing models is somewhat novel. 

3. The paper is illustrative with an ablation study by varying the number of training datasets, types of loss functions, masked language model training, and loss weights. 

Weakness:

1. The main idea of this paper is based on prior work: local Lipschitz smoothness from (Chen et al., 2022). However, comparison results with the method  LIGER from (Chen et al., 2022) are missing. 

2. The sizes of the datasets used in the evaluation are quite small and the numbers of samples are in the thousands. To demonstrate the scalability of the proposed method, much larger datasets are recommended. 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. 

Typos: 
section 4.1:  "TO train E" -> "To train E" 
caption of Figure 1:  should be  {M}_{(i = 1)}^k 

Given the prior work local Lipschitz smoothness (Chen et al., 2022), the novelty of this paper is thin. It'd be great if the proposed method could be compared with the method LIGER from (Chen et al., 2022). 

Any t-test results for the comparisons in Table 2 and 3? 
summary_of_the_review: This paper presents a multi-domain adaptative approach for early fake news detection, which integrates pre-trained and fine-tuned models along with their training data to learn a domain-invariant representation. In this representation, the proposed framework adaptively selects the highest-ranked model to perform classification according to the relevancy between a new sample and model training datasets.

In general, the idea of using randomized Lipschitz smoothness measure to generate source model relevancy rankings is reasonable and effective for fake news detection. 

However, the contribution of the paper is incremental given the prior work local Lipschitz smoothness (Chen et al., 2022). 

There are missing technical details:
1. What are the t-test results for Table 2 and 3? 
2. What are the results in terms of other metrics such as F1 and coverage for Table 2 and 3? 
3. Complexity analysis of the proposed method? 

To demonstrate the scalability and efficiency of the proposed method, evaluation on much larger datasets is recommended.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents an approach, MiDAS, to classify new samples in the situation where there are multiple domains and domain concepts drift over time. The proposed method has two components: a domain-invariant encoder, and an adapetive model selector. The encoder creates domain-invariant representations for data samples and the selector uses Lipschitz smoothness to estimate the best-fit model for a new sample. Experimental evaluation is performed on several fake-news datasets.

strength_and_weaknesses: Strengths:

+This paper introduces the problem of concept drift in misinformation clearly. The motivation of designing a domain-adaptive approach is clear.

+The proposed framework is well described with figures and details. The first part, the encoder, aims to learn domain-invariant representations for data samples. Based on the learned representations, this paper introduces the Lipschitz continuity and extends this to find the best-fit model from multiple source models. 

+Experiments are performed on multiple fake news datasets to test the performance of the proposed approach on classifying unseen/drifted distributions. Ablation studies are performed by changing the number of training datasets, types of loss functions, masked language model training, and loss weights.

Weaknesses:

First, this paper focuses on the application of fake news detection. However, the connection between the proposed model with fake news seems not strong. Very little analysis of fake news data distributions is provided. Datasets are not described in detail. The unique challenges in fake news detection are not introduced. If the proposed method is a general approach for any concept-drift problems, the experiments are only conducted on fake-news datasets.

Second, the motivation for using Lipschitz smoothness to estimate/find the best source model/function for an unlabeled and potentially drifting sample is not strong. The reasons for using this notion should be better elaborated.

Third, some notations can be better clarified. For instance, how is function theta specified is not clearly described in the methodology section. Equation 3 seems to use absolute difference, but the reason why this can be used for estimating the smoothness of source functions remains unclear. The motivation for using probabilistic Lipschitzness can be better explained. It would be more convincing if examples or proofs of changes in perturbations are provided.

clarity,_quality,_novelty_and_reproducibility: Clarity: The coherence of this paper can be improved. Especially, how the proposed work can solve unique challenges in fake news concept drift problems can be clarified. Notations should be described in detail.

Quality: The proposed method seems not well motivated. It is unclear how Lipschitzness smoothness is the best fit for fake news concept drift problems.

Novelty: the proposed idea is interesting. I think the intuition behind proposing this idea can be strengthened.

Reproducibility: code is provided in the supplemental material.

Minor comments: in Abstract, there is a typo: “a doman-invariant encoder”. Page 6: Ablation Study: “masked language model trainig”.

summary_of_the_review: This paper presents a solution for classifying samples when multiple source domains are present and concept drifts exist. The target application is fake news detection, however, the connection of the proposed framework with this specific application is not strong. The motivation for using Lipschitzness should be better described.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose MIDAS, a statistical machine-learning approach for fake news detection that ranks the relevancy of existing models to new samples. It has a domain-invariant encode and an adaptive module. The authors have empirically verified the model on 9 COVID fake news datasets. The main complaints from reviewers are: (1) The link between the particular method and the COVID fake news detection task is weak. It is unclear why the authors have chosen this particular method (local Lipschitz smoothness of the invariant embedding space) for this particular problem (fake news detection) at a particular time (COVID). (2) The motivation for using Lipschitz smoothness for domain generalization is not strong. There were some other minor concerns about clarity and other issues in the experiments. Overall, there is a clear consensus that this paper is not ready to be accepted by ICLR. 
justification_for_why_not_higher_score: The paper chose a specific technical approach to a specific task without good justifications. All reviewers agree that this paper is not ready to be published at ICLR. 
justification_for_why_not_lower_score: Reject is the lowest category.
Decision: Reject