Title: What Variables Affect Out-of-Distribution Generalization in Pretrained Models?
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Embeddings

Out-of-Distribution Generalization

Tunnel Effect

Neural Collapse
Abstract: Embeddings produced by pre-trained deep neural networks (DNNs) are widely used; however, their efficacy for downstream tasks can vary widely. We study the factors influencing transferability and out-of-distribution (OOD) generalization of pre-trained DNN embeddings through the lens of the tunnel effect hypothesis, which is closely related to intermediate neural collapse. This hypothesis suggests that deeper DNN layers compress representations and hinder OOD generalization. Contrary to earlier work, our experiments show this is not a universal phenomenon. We comprehensively investigate the impact of DNN architecture, training data, image resolution, and augmentations on transferability. We identify that training with high-resolution datasets containing many classes greatly reduces representation compression and improves transferability. Our results emphasize the danger of generalizing findings from toy datasets to broader contexts.


Peer reviews:
comments: SUMMARY:
The paper sets out to evaluate empirically the generality (referred to as "universality") of the "tunnel hypothesis", the idea that compression at later layers may hinder OOD performance.

STRENGTHS:
Work on OOD learning, network probing and explainability is very relevant to the conference.

WEAKNESSES:
How empirical evaluations may provide any conclusive answer to the question of generality of a hypothesis is unclear to me. 

The use of linear probing cannot be taken as the yardstick for this study. The validity of any result is further compromised by the choice of SHAP in the place of a sound knowledge extraction method for the evaluation of the results. SHAP is known to produce unreliable results.

QUESTIONS:
What does it mean to "disentangle samples and classes"?

What is "diversity of inputs"?

LIMITATIONS:
Please define linear probe and probe accuracy and how it is used in the paper.  

It is very unclear how swapping tunnels might help you answer the question of mitigating forgetting.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the \"tunnel effect\" in NNs introduced in a NeurIPS 2023 paper. According to the “tunnel effect” the deeper layers compress representations, limiting OOD generalization. The authors challenge the prior assumption that the tunnel effect is universal. They imply that it's heavily influenced by training data diversity.\nThey analyze a large number of NNs on various datasets with varying resolutions and introduce three metrics to quantify the tunnel effect's strength. Using a SHAP-based approach, they disentangle the impact of augmentation, number of classes, sample size, resolution, and the architecture.\nTheir experiments suggest that increasing the diversity of the training data (more classes, augmentations, and higher image resolutions) mitigates the tunnel effect and improves OOD generalization. Depth and overparameterization have the opposite effect. \nBased on these findings, they propose a revised tunnel effect hypothesis where its strength is inversely proportional to training data diversity.\n\nSTRENGTHS:\nOriginality: This paper is original, in the sense that it challenges existing assumptions on  the universality of the tunnel effect hypothesis, which might have implications on understanding and improving NN’s OOD generalization ability. The authors propose a revised tunnel effect hypothesis which is a novel contribution.\n\nQuality: This paper is scientifically well-executed. The hypothesis is clear and the experiments are extensive (several NN architectures and dataset (configurations), which greatly helps to support their hypothesis. The SHAP-based analysis they performed to disentangle the relative importance of different variables is solid (through rigorous statistical analysis with Wilcoxon signed-rank tests and Cliff's Delta to measure effect sizes) and provides a quantitative measure of the effects.\n\nClarity: The paper is very well written and easy to follow. Figures and tables are very helpful to understand the key findings.\n\nSignificance: The paper’s findings may have significant implications for the “tunnel effect” hypothesis, though I am a bit skeptical of the new insights this brings us in terms of designing algorithms that have better OOD generalization properties.\n\nWEAKNESSES:\nAlthough the paper is a very solid work, the main worry that I have is about the confounding variables in the experiments and the revised hypothesis, which might limit the significance of the contribution:\n* For example, Figure 1 claims that the increased resolution mitigates the tunnel effect. But intuitively, if you increase the resolution of the image, you also increase the resolution of the activation maps in the bottleneck layers. Still this doesn’t tell us anything about the architectures per se. Yes, the tunnel effect can be “sidestepped” if you increase the resolution, but then you run into the same issues if you increase the depth.\n* OOD accuracy is measured on a number of datasets (NINCO [61], ImageNet-R [62], CIFAR-100 [60], CIFAR-10 [59], Oxford 102 Flowers [63], CUB- 173 200 [64], Aircrafts [65], Oxford-IIIT Pets [66], and STL-10 [67]). However, I wonder what such experiments tell us about the “kind” of OOD that we are testing for. For example, if one tests for compositional or systematic OOD generalization, it becomes very clear what the learnings are, and which particular flaws the models have. I feel that on these benchmarks it is hard to make the similar conclusions.\n* Certain findings I find limited in the sense that they are already known, e.g. “Our metrics reveal that increasing the ID class count, higher resolutions, and using augmentations improve OOD generalization”.\n* In these figures/results, typically normalized accuracy or performance retained are discussed. However, in ML it is also important to understand the actual accuracy, and these are not easy to find in the paper. Ideally for each figure/results discussion, the reader would be pointed to a table (in the appendix) that contains absolute numbers on the datasets. For example, it is claimed that \"overparametrization negatively impacts OOD generalization .. since increasing overparametrization decreases OOD performance **retained**\".\nHowever I feel that without the absolute numbers this claim cannot be made. E.g. if both the IID and OOD performance improved by increasing overparametrization, but the IID performance improved more than OOD performance, one cannot claim that OOD performance decreased, quite contrary.\n* L281: “Depth. Our SHAP analysis revealed that increasing depth impairs OOD generalization.” - Depending on how you increase depth (e.g. if you have pooling layers), to me this is an obvious consequence of additional compression that you may have in that case.\n* The paper claims that their continual learning results contradict [9] and suggest the “tunnel” plays an essential role in mitigating catastrophic forgetting. However, they only replicate the experiment with ResNet-18 on ImageNet-100. To strengthen this claim, the authors could expand the continual learning analysis to include different architectures and datasets.\n\nQUESTIONS:\n1. This citation has the wrong year and NeurIPS number: “Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miło ́s, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36, 2024.”.\nPlease cross-check other references too.\n2. SHAP papers should be cited the first time it is mentioned in the text.\n3. Could you clarify a bit further the reasoning / logic behind the “Metric 3: ID/OOD Alignment”? I find it a bit hard to wrap my head around it in its current form. Specifically why you choose to have a product of the two values and the respective “baseline” values that are subtracted.\n4. What do you mean by “more hierarchical features” in L136? Isn’t the tunnel effect reduced because the size of the activation maps is increased, so the bottleneck is decreased?\n5. L275: Perhaps a more appropriate wording here would be \"7x7 kernel size\" instead of \"7x7 stem\"?\n\nLIMITATIONS:\nThe limitations and societal impacts have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper studies how well self-supervised models transfer after self-supervised pretraining.  Using linear probe experiments on top of frozen encoders, and by varying the depth in the model being fed to the linear probe, they study the effect of network depth on ID performance and OOD (transfer learning) performance.  They find that the previously studied "tunnel effect hypothesis" can be diminished by increasing the diversity of pretraining data (which they demonstrate using data augmentation).

STRENGTHS:
The paper builds on previous works to clarify their results and provide important additional context.

WEAKNESSES:
Scope:  The results of the paper seem fairly narrowly scoped.  I am not sure how these findings can really inform pretraining in ways that aren't already well understood (e.g. "use more diverse data" is already a fairly well established paradigm).  Thus it seems like the impact of this paper is fairly limited.

Experimental evaluation seems somewhat limited:  While the authors do run a large number of individual training runs, it is not clear to me that they performed a particularly wide experimental study.

QUESTIONS:
How can we interpret these results in the context of other methods like SSL, where the learning goals seem to be more directly targeting removal of any tunnel effects?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

comments: SUMMARY:
This paper studies the factors influencing out-of-distribution (OOD) generalization of pre-trained DNN embeddings through the lens of the tunnel effect hypothesis, which suggests deeper DNN layers compress representations and hinder OOD performance.

STRENGTHS:
- The paper includes a sufficient amount of experiments, offering detailed and substantial content.

- The main purpose of the paper is clear and easy to understand.

WEAKNESSES:
- The technical contribution is limited. The experiments on OOD generalization discussed in the paper are rather trivial and do not yield sufficiently insightful conclusions.

- The paper is poorly written. As an evaluation-focused study, it fails to effectively organize the relationship between the arguments and experiments, making it rather disorganized.

- The paper does not provide significant theoretical contributions, focusing primarily on empirical results without delving into the underlying theoretical frameworks or offering new theoretical insights.

QUESTIONS:
please refer to the weakness.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-15