Depth $F_1$: Improving Evaluation of Cross-Domain Text Classification by Measuring Semantic Generalizability
Authors: Parker Seegmiller, Joseph Gatto, Sarah Masud Preum


Reviewer paper summaries: This long paper proposes a new evaluation metric designed for cross-domain evaluation of text classification. It weights target-domain examples based on their dissimilarity to the source domain that the classifier has been trained on, with the most dissimilar examples receiving the highest weights. (Dis)similarity is computed using a statistical depth function, hence the name of the metric: Depth F_1 (or DF_1 for short). The motivation for this metric is that the standard evaluation procedure, where target-domain samples are unweighted, leads to a misleading assessment of the generalization capabilities of the models. Experiments are conducted on two benchmarks (in English only) for sentiment analysis and NLI. Experiments show how the impact of very dissimilar examples is diluted by the presence of similar examples, and how (source, target) domain pairings vary extensively in terms of similarity, and thus difficulty. Four classification models (which mainly rely on sentence-bert or LLaMa) are then benchmarked. The authors mention that this method could be adapted to other tasks.
Summary of strengths: - The motivation is well explained in the introduction, where they focus on safety-critical applications, where an accurate estimate of generalization is particularly important.
- The method to identify the most dissimilar samples could be useful in itself, for a more "in-depth" evaluation, as the authors say. But that part of the method is not novel (or not particularly novel), if I understood correctly. 
- Code will be made available, which will ease adoption and help reproducibility.
- The paper is clear and very well written.
Summary of weaknesses: - The novelty of the method is low, as the dissimilarity weights are computed using an existing method (or at least on a method heavily based on prior work), and the rest is just a weighted F-1 score.  
- The method assumes we can know exactly which examples were trained on, but this is increasingly not the case for proprietary models/APIs (which do not disclose their training data), so the applicability of the method may decrease as time goes on.
- The method does not exploit the labels of the target domain, only the text samples. This should at least be acknowledged. 
- The paper does not discuss the impact of the threshold lambda on the statistical reliability of the estimate of performance: increasing lambda reduces the size of the test set, which can make it less reliable, statistically speaking. 
- Experiments are only on English data, but this is pretty common. 
- The experiments do not show whether the metric might produce different *rankings* for a set of models, compared to vanilla F_1 evaluation. If not, this just provides a better point-wise estimate of generalization to samples that are very different from those seen in training, and might not be that helpful in terms of model *comparison*, though it might be helpful for assessing whether a particular model is deemed good enough for some safety-critical application.
Comments, suggestions and typos: - Footnote marker 1 should be placed after the full stop.
- Lines 145-146: "genres of English" is weird phrasing. In my mind, genres are not language-specific. Maybe replace with "genres in English"?
- Line 147: Use the "Author (Year)" format here. 
- Lines 164-168: Please specify whether any of these prior works were applied to the task of text classification. Otherwise, mention what other tasks they were applied to. You do mention later that Yan et al. (2019) evaluated text classification, but what about the others mentioned here?
- In the related work, you could mention other work that focuses evaluation on challenging subsets of the test data, even if they are not specifically targeted at cross-domain text classification. One example is "HardEval: Focusing on Challenging Tokens to Assess Robustness of NER" (https://aclanthology.org/2020.lrec-1.211/). 
- Footnote 2: use \url{} to make the link clickable.
- Lines 340-341: What follows "We consider the challenging multiclass classification setting, as ..." does not seem like an explanation of the difficulty of the task. Maybe rephrase to clear this up.
- Line 354: what is "slate"?
- Line 515: I don't find the term "semantic overfitting" appropriate or useful. Overfitting means learning features that don't even generalize in-domain. In general, I would avoid talking about overfitting in cross-domain settings, and just call this lack of generalizability.
Overall assessment scores: 3