Kati8/ICLR_2023_2025 huggingface dataset:
authors: Mohammed Adnan
authors: Yani Ioannou
authors: Kenyon Tsai
authors: Angus Galloway
authors: Hamid Tizhoosh
authors: Rahul G Krishnan
authors: Graham W. Taylor
reviews: 
Comments: This paper propose to detect shortcut by evaluating the mutual information between the latent variable Z of a network and the input X. The intuition is that the latent variable Z of a network learning shortcuts would have a lower mutual information with the input X. Based on the intuition, the paper propose to detect shortcut by calculating mutual information using neural tangent kernel (NTK).
Strengths: 1). Using mutual information to detect whether networks is an interesting topic.

2). This paper provide a detailed related work introduction.
Weaknesses: The proposed method seems infeasible.

1). In algorithm 1, a dataset is said to contain shortcut if $I(X_{test};Z)<I(X_{tr};Z)$. However, since $X_{test}$ have different distribution than $X_{tr}$, it seems natural to have $I(X_{test};Z)<I(X_{tr};Z)$. Therefore an important question need to be answered: are there any datasets satisfying $I(X_{test};Z)\geq I(X_{tr};Z)$?  There lacks empirical evidences in this paper to answer the question and I do not think the algorithm 1 could effectively detect shortcuts.

2). Unlike the proposed algorithm 1, the experiments in this paper, on the other hand, mainly compare networks trained on two datasets "with" and "without" shortcut. This approach also have problems since it requires comparing with a network trained on dataset that is "without" shortcut. By defining "without shortcut", it also involves domain knowledge and human expertise to detect shortcuts.

Based on the above two points, I think that the propose method have major flaws.

Minor issues:

1). the introduction of estimating mutual information using NTK is vague. For example, what is the definition of $\Theta(x,X)$? $\sigma$ is used in Eq.5 but it is written as $\Sigma$ in Eq.7.

2). Figure legends (e.g. Fig.4) could be more detailed.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The author claims that the failure of deep neural networks to generalize to out-of-distribution (OOD) data is often caused by shortcuts or spurious correlations in the training data, leading to a performance drop due to distribution shift. This paper aims to understand shortcut learning using information-theoretic tools and propose using mutual information (MI) between the learned representation and the input space as a domain-agnostic metric for detecting shortcuts in training datasets. To study the training dynamics of shortcut learning, a framework based on the Neural Tangent Kernel (NTK) in introduced, able to detect shortcuts and spurious correlations in training data without requiring class labels for the test data. Empirical experiments on multiple datasets, including MNIST, CelebA, NICO, Waterbirds, and BenchMD, demonstrate that MI can effectively detect shortcuts.
Strengths: 1. This paper can be viewed as a complementary method to existing out-of-distribution (OOD) detectors. It offers a domain-agnostic metric to detect shortcuts and provides a way to diversify the training data before deploying a model in safety-critical domains. By identifying shortcuts and spurious correlations in training datasets, it helps improve the robustness and reliability of models when faced with distribution shifts, making it a valuable tool for ensuring model performance in real-world applications.

2. The quality of the writing is commendable, and the experiments conducted to establish the effectiveness are adequate.
Weaknesses: 1. The core concept underpinning the method lacks novelty. In essence, during the training phase, the objective is to minimize $I(X; Z)$ on the training data $D_tr$. However, it becomes evident during the testing phase that $I(X_{te}; Z) < I(X_{tr}; Z)$, provided a domain gap exists between the training data $D_{tr}$ and the test data $D_{te}$. Subsequently, equations (4) - (10) are all employed to compute $I(X; Z)$ using well-established formulations from prior literature. Hence, the methods devised are not restricted to the detection of shortcuts and can be applied to data exhibiting substantial domain gaps. I recommend that the authors conduct additional experiments on out-of-distribution (OOD) detection.

2. Based on the experiments on partially correlated shortcuts as depicted in Figure 4, I have reservations about the method's limitations when it comes to detecting datasets with partially correlated shortcuts. This is because the hard line in Figure 4(a) appears to be very close to the dotted line in Figure 3(a). I recommend that the authors conduct additional experiments to further investigate the detection of datasets with partially correlated shortcuts.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: The authors utilize a neural tangent kernel framework to approximate the mutual information between input and learned representations. They hypothesize that mutual information should be smaller in the test set compared to the training set if there's a distribution shift present. They test their ideas on a number of datasets and compare their proposed methodology with out-of-distribution (OOD) detection algorithms.
Strengths: - The authors' topic of choice is timely, and is of both theoretical and practical significance.
- The authors involve a number of datasets in their experiments, and go beyond frequently used (semi)synthetic datasets in their investigation. Domain-specific empirical examination is especially likely to be informative in this topic.
Weaknesses: I believe that the issue the authors address requires conceptual clarity in terms of defining and diagnosing the problem, and a carefully constructed methodology and accompanying experiments to support the arguments made. I believe that the paper has important shortcomings in these respects. I summarize my main criticisms below, and provide detailed questions and observations regarding these in the next section.
- The paper presents no systematic justification of their proposed method, apart from some conjectural statements. 
- The paper is imprecise, unclear, and/or inconsistent with their conceptual structure or central notation.
- The experiments do not seem consistent with the onus of the paper's argumentation, present insufficient details on the specifics, and do not pursue some critical avenues to their fullest extent.
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: This paper proposed a framework for detecting spurious correlations or shortcuts implied in training datasets. The main hypothesis of this paper is that the information between input and embedding would be low provided that there are shortcuts in a dataset. The author leveraged a neural tangent kernel to estimate mutual information between input and embedding representation and empirically represented that their hypothesis is valid on the synthetic (MNIST with shortcuts), benchmarks (Waterbirds, CelebA, and NICO), and real-world medical datasets.
Strengths: - The empirical results support their hypothesis that the $I(X;Z) of a dataset with a shortcut is lower than that without a shortcut.
- The method is simple and easy to follow.
Weaknesses: - The proposed method is limited in the real-world application scenario. The proposed method requires a 'without shortcuts dataset' to detect whether there are shortcuts in the training dataset. I am not sure how many cases we can prepare the 'without shortcuts dataset' before we know whether the training dataset has a shortcut.
- (Kirichenko et al., 2022) represented that the model trained on Waterbirds using ERM has the ability to classify the foreground-only and background-only datasets. It conflicts with the main hypothesis that the model trained with the shortcut dataset will have a low $I(X;Z).
- The experiment graphs show that the mutual information is less discriminative than the losses, which diminishes the necessity of using mutual information to detect the existence of shortcuts in the training dataset.

Minor corrections
- Page 5 MNIST with synthetic shortcut) Figure 1 -> Figure 2
- Figure 2) Please denote the used Saliency map.
- Definition 2) Please state that the higher $\Gamma$, the better the generalization.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good