Scaling Channel-Invariant Self-Supervised Learning

Authors: Alice V. De Lorenci, Seung Eun Yi, Théo Moutakanni, Piotr Bojanowski, camille couprie, Juan C Caicedo, Wolfgang Maximilian Anton Pernice
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self Supervised Learning, Vision Transformer, microscopy, channel adaptive

Abstract:
Recent advances in self-supervised pre-training of foundation models for natural images have made them a popular choice for various visual systems and applications. Self-supervised strategies have also shown promise in non-RGB scientific imaging domains such as in biology, medical and satellite imagery, but their broader application is hampered by heterogeneity in channel composition and semantics between relevant datasets: two datasets may contain different numbers of channels, and these may reveal distinct aspects of an object or scene. Recent works on channel-invariant strategies report substantial advantages for those that account for variable channel compositions without sacrificing the ability to jointly encode channels; yet, how these strategies behave at scale remains unclear. We here show that, surprisingly, trained across large-scale microscopy datasets, independent-encoding of channels consistently outperforms joint-encoding methods by a substantial margin. We explore this result along an extensive set of experiments and open-source a new general purpose feature extractor for fluorescent microscopy images, DINO BoC, that sets a new state-of-the-art across challenging benchmarks, including generalization to out-of-distribution tasks and unseen channel combinations at test time.

Decision: reject
Reviewer scores: 3, 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1729110274599

thread_type:
official_review

<thread_object>
creator: reviewer_nzma

<content>
summary:
Some applications of ML (such as biology, medicine, remote sensing) feature different channels across datasets. This channel robustness challenge is under-explored in ML since natural imagery applications do not often face this challenge — they consistently use 3 channels (RGB). This paper focuses on fluorescent microscopy images to investigate channel robustness. (I'm using channel robustness to mean generalizing to missing channels and new channels.) A prior work, Channel-ViT tokenizes channels separately and can be robust to missing channels (similar to remote sensing models like SatMAE and Presto). This paper introduces a ViT architecture called DINO-HA. This paper thoroughly benchmarks Channel-ViT, DINO-HA, and DINO-BoC (Bag of channels) using large backbones (ViT-L) and large pretraining datasets. This paper finds that DINO-BoC often outperforms Channel-ViT, evaluated as a frozen feature extractor.

strengths:
- This paper is well written
- Extensive experiments are performed. Specifically, using ViT-Large models, two different SSL algorithms, different pretraining datasets, and several in-distribution and out-of-distribution test sets.
- From my reading, the core finding is that a "bag of channels" approach can often outperform models that allow inter-channel reasoning. This is interesting and surprising.

weaknesses:
- Channel-ViT (the baseline) beats DINO-HA on 17 of 26 test sets on page 8. And DINO-BoC beats Channel-ViT on 16 of 26 test sets on page 8. I'm not convinced that DINO-HA is valuable to the community beyond these fairly negative results. 
- I do not consider DINO-BoC to be a novel method. It process channels separately using a ViT, then concatenates embeddings.

I recognize this is an applied paper and the bar for ML novelty is lower, however I question the value to the broader ICLR community. That being said, the experiments are extensive and the findings are interesting.

questions:
As far as I can tell, DINO-BoC concatenates channel embeddings to form the full-sample embedding using the CLS tokens. This embedding is a D * N-dimensional vector (where D is the ViT width and N is the number of channels). And the full-sample embedding of Channel-ViT is a D-dimensional vector, taken from the CLS token. Although in the appendix it says the CLS token is concatenated across the last 4 layers for linear probing.

Can you please clarify how you build full-sample embeddings for each of the three methods? If Channel-ViT's embeddings are indeed D-dimensional and DINO-BoC are D * N-dimensional, I'd like to see results when they have the same sized embeddings. This can be done by taking the mean between channels (for DINO-BoC) to achieve D-dimensional embeddings. This can be done by concatenating the mean-pooled patch representations for Channel-ViT to achieve N * D-dimensional embeddings (specifically, you'd pool the patches for each channel, then concatenate these pooled channel-wise embeddings).
</content>
</thread_object>

</thread>
<thread>
creation date:
1729882692089

thread_type:
official_review

<thread_object>
creator: reviewer_ty5v

<content>
summary:
The paper presents a large set of SSL experiments in the microscopy imaging domains and scales existing methods with the existing DINOv2 approach for channel-variant images.

strengths:
The paper addresses an important problem related to channel-varying images in microscopy imaging and conducts an extensive set of experiments, including OOD generalization, within this domain. The trained models generally outperform the existing literature.

weaknesses:
1) The paper's title, "Scaling Channel-Invariant Self-Supervised Learning," suggests broad applicability across various imaging domains. However, the experiments are limited to fluorescent microscopy images despite the relevance of channel-invariant features in other fields, such as medical and satellite imagery, as also acknowledged by the authors in the abstract. Prior work on channel-invariant learning has demonstrated applications across diverse image domains [1], including satellite imaging. A broader experimental scope of imaging domains (e.g., see [1]), rather than larger datasets on a single one, would align the paper’s claims with the actual generality and applicability of channel-invariant self-supervised learning across relevant fields. 

2) Contribution (i), which involves selecting DINOv2 and integrating it with the Bag of Channels approach (an existing SSL algorithm combined with a known channel-invariant module), is challenging to consider a meaningful contribution on its own since the combination of these methods lacks substantial technical novelty, as it doesn’t introduce new mechanisms or address specific challenges unique to this approach. 

3) The technical novelty (contribution (ii)) is limited, as the proposed Hierarchical Attention module appears to be a combination of existing methods (Channel-ViT and Bag of Channels) with the addition of extra "channel" tokens and corresponding attention masking. Furthermore, the clarity of presentation of the technical contribution needs improvement, as I explained in the next point. It is also unclear to me the motivation behind introducing DINO HA since, in Tables 1, 2, and 3, the best method is usually DINO Bag of Channels and not DINO Hierarchical Attention. What is the purpose of proposing two methods, with one clearly better than the other? 

4) The technical presentation of contributions (i) and (ii) is somewhat unclear and misleading. More precisely, the main technical section 3, which is named "Problem Formulation" (renaming needed), should instead focus on the description of the introduced modules and clearly discern the contributions of the paper (sections 3.3 (?) and 3.4) against the already existing strategies (i.e., sections 3.1, 3.2, 3.5). To clarify the contributions, it would help to split Section 3 into two sub-sections (Preliminaries vs Methodology) to help the reader better grasp the actual contributions. I would also suggest a better design of Figure 2, e.g., in a single panel, highlighting the differences between previous and proposed work. 

5) The related work section concerning SSL with microscopy images and section 4.1 are comprehensive but verbose and would need some summarization and re-writing.

[1] Bao et al., Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words, ICLR 2024

questions:
Unfortunately, while the paper tackles a relevant problem and demonstrates better performance through a large set of experiments, the contribution is limited by a lack of generalizability across (untested) imaging domains and, most importantly, by insufficient technical novelty.

See the weaknesses section to improve the technical presentation of the contributions. I am open to hearing the feedback from the authors on the rebuttal period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648117672

thread_type:
official_review

<thread_object>
creator: reviewer_6fwq

<content>
summary:
The paper proposes a self-supervised learning method for heterogeneous multi-channel images. The proposed method uses DINO v2 as a backbone and integrates bag of channels and hierarchical attention into Channel ViT. The work is evaluated on microscopy images extensively.

strengths:
1. The experiments on microscopy datasets are extensive and the reported numbers provide empirical understandings.

weaknesses:
Major Concerns:
1. The title describes a generic method for multi-channel images, but the paper focuses only on microscopy images. Considering that different multi-channel images may exhibit different behaviors, the title seems to be a bit big. The authors shall either broaden their experiments to include other types of multi-channel images, or narrow the title and framing to focus specifically on microscopy images.
2. The title states this work focuses on self-supervised learning. However, after reading the whole paper, the method is not generic. On the contrary, it seems to be designed based on DINO v2. No discussion on the choice of DINO v2 at all.  The authors shall provide a clear justification for choosing DINO v2 as their base method. Additionally, please discuss how your approach might generalize to other self-supervised learning frameworks, or explain why your method is specific to DINO v2.
3. The contributions on self-supervised learning is not clear to me. In my opinion, the paper seems to be a multi-channel variant of DINO v2, which is an incremental work. But, the paper tries to tell a different story. If this work is about self-supervised learning while leveraging multi-channel images, please specify why this work differs from studies of self-supervised learning on RGB images. If this work focuses on adapting existing self-supervised learning works on multi-channel images, please clarify it from the title to introduction. 
4. The technical writing is a bit hard to follow. The presentation of the method proposed is unclear. The paper spends quite some space the background while the methodological details are provided in the appendix. It is recommended to reorganize the related works and method sections to show key methodological results.

questions:
1. How does this method work on multi-channel images such as remote sensing or multi-spectral images?
2. Does this method work on other self-supervised learning framework?
3. Follow the same logic, will better self-supervised learning backbone show better results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699195696

thread_type:
official_review

<thread_object>
creator: reviewer_pzt7

<content>
summary:
This work study different channel invariant strategies to train vision transformers for self supervised learning. These approaches are study in the context of fluorescent microscopy images where different datasets carry different combination of channels. Channel invariance allows for self-supervised pretraining using a combination of all existing datasets which translates to better pretrained models. Authors also proposed hierarchical attention as a novel way to train channel invariant models.

strengths:
* Paper is well written and technically sound
* The work address a significant problem for applying machine learning to microscopy images at scale
* Proper evaluation of the different approaches is coducted

weaknesses:
* It is not clear how the bag of channels approach claimed as yours is different to Microsnoop form Xun et al.
* The proposed hierarchical attention approach rarely beat the bag of channel approach 
* Work only tested on a single application domain.

questions:
* Do you expect this approach to work for other application domains with image modalities of different number of channels with limited channels matching across datasets like it is often the case in remote sensing?
* Figure 2 legend can be improved by adding more details to it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732702328

thread_type:
official_review

<thread_object>
creator: reviewer_azkt

<content>
summary:
This paper proposes solutions for the challenge of handling varying number of channels, in particular for microscopy images where the number of channels can be different in each dataset. The authors use three main self-supervised approaches on DINOv2, with Bag of Channel approach performing the best across the benchamrks used.

strengths:
- The problem statement and related work are well-written, and the problem is relevant
- The experimental section is reach. The authors provide results and comparisons on several datasets and benchmarks
- The code is going to be publicly released for future uses.

weaknesses:
The main weakness of the paper is the limited contributions. The best performing method, which the authors call Bag of Channels, was proposed by (Xun et al, 2024). This paper only applies this method on DINOv2. Although the experiments and comparisons, alongside introduction of the Hierarchical Attention Model is valuable, these contributions could be a better fit for other conferences/journals rather than ICLR.

questions:
The experiments are performed on datasets with 2-5 channels,
- Is there any correlation between the number of channels and the method performances?
- Why didn't you use datasets with more channels (as stated in L38, up to 8 channels)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732465640881

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all five reviewers for their constructive feedback. In response, we provide new experimental results as well as broad revisions that, we think, substantially strengthen the paper. We here address the shared major concerns regarding the generality and novelty of our method. We also address more specific critiques for each reviewer individually. 

**Domain generality**

A shared critique concerned the generality of our approach, which our title seemed to allude to, but which indeed was not experimentally documented in our results. To address this, we provide a new set of experiments on a new domain, specifically aerial images of a methane source prediction multi-sensor dataset, Meter-ML, demonstrating that DINO-BoC here too achieves performance on par with the SOTA while using far less pre-training data and frozen features (Appendix J). We also narrowed our title to “Scaling Channel-Invariant Self-Supervised Learning for Microscopy Images”.

| Approach                                    | Test dataset | Pre-training dataset            | mAP   |
|---------------------------------------------|--------------|---------------------------------|-------|
| Meter-ML (Zhu et al 2022)               | NAIP, S2, S1 | NAIP, S2, S1 (85K)              | 51.7  |
| LHRS-bot (Muhtar et al 2024)              | NAIP, S2, S1 | LHRS-Align-Recap (1.1M images) | 71.8 |
| VHM	(Pang et al 2024) | NAIP, S2, S1 | VersaD (14M images) | 72.7 |
| DINO BoC (ours)                                | NAIP, S2     | Meter-ML NAIP, S2 train (85K)   | 70.9  |

| Approach                                    | Test dataset | Pre-training dataset            | mAP   |
|---------------------------------------------|--------------|---------------------------------|-------|
| Meter-ML (Zhu et al 2022)               | NAIP         | NAIP (85K)                      | 54.8  |
| SatMAE (Cong et al 2022)                | NAIP         | fMoW Sentinel                   | 76.9  |
| ScaleMAE (Reed et al 2023)               | NAIP         | fMoW RGB (363K)                 | 78.4  |
| USatMAE (Irvin et al 2023)                | NAIP         | USAtlas NAIP (3.6M)             | 83.7  |
| DINO BoC (ours)                                | NAIP         | NAIP  (85K)                     | 82.2  |
| DINO BoC (ours)                                | NAIP         | NAIP, S2 (85K)                  | 81.9  |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733306248447

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for taking the time to consider our answers and for their feedback. We would like to emphasize that the significance of our work lies in the finding that a technically simple strategy, that of encoding each channel independently, when combined with ViTs and the DINOv2 SSL method (DINO BoC model), outperforms all previously proposed channel-invariant models for microscopy. This includes those that incorporate more complex joint-channel-encoding strategies. As a result, we not only set a new state-of-the-art for microscopy images but, more importantly, put forth a new direction for the field and provide critical guidance for practitioners.

Recent papers on channel-invariant methods (Kraus et al 2024, Bourriez et al 2024, Bao et al 2024, Pham & Plummer 2024) have built a consensus suggesting that joint-channel-encoding is indispensable in the channel-heterogeneous settings. Our experimental results, however,  demonstrate for the first time that joint-channel-encoding can, in fact, be detrimental.

We support this claim with a substantial set of experimental results:
- (Section 4.3) DINO BoC outperforms all other channel-invariant models when trained on a mix of microscopy datasets and evaluated both on known and novel data sources.
- (Section 4.4) DINO BoC consistently surpasses joint-channel-encoding methods on cross-dataset generalization tasks.
- (Section 4.5) DINO BoC establishes a new state-of-the-art on the CHAMMI benchmark (Chen et al 2024), a challenging benchmark for OOD generalization tasks. To the best of our knowledge, CHAMMI is the only standardized microscopy benchmark for channel-invariant models.

Our claim is further substantiated by experiments on DINO HA -- a technically innovative method -- that despite outperforming all previous approaches, still falls behind DINO BoC. The experiments with DINO HA are significant because this approach balances the strategies of jointly and independently encoding the channels, therefore it solidifies our argument and understanding of why certain approaches succeed or fail.

**Paper structure**

In response to the reviewers’ constructive suggestions, we have significantly reorganised our paper to better present our results. 

Methodological details are now more clearly outlined in Section 2 (Method). Section 4 (Experiments) has been restructured to better highlight experimental results that were previously not sufficiently emphasized and additional experiments were incorporated:
- The choice of the SSL method DINOv2 is explained in Section 4.3, where we show that DINOv2 yields significant improvements over MAE, as demonstrated in Table 1.
- We present novel experimental results directly comparing DINO BoC to Microsnoop. Notably, DINO BoC outperforms Microsnooop on the Cyclops dataset, where Microsnoop had reported its largest gains, despite being highly out-of-domain for DINO BoC.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates self-supervised learning regimes for heterogeneous multi-channel images. The submission uses DINOv2 as a backbone and integrates bag of channels and hierarchical attention for channel-variant images. Initial experimental work concerned microscopy images (latterly aerial images). Investigation of challenges, specific to channel-varying images, can be considered an interesting pursuit and work in this area is welcome.

Post-rebuttal, a clear majority of reviewers remain unconvinced regarding the technical innovation and opt to retain their overall negative view on the paper. AC does not find a clear motivation to overrule in this case and therefore recommends rejection. 

Authors received extensive feedback relating to the technical aspects of the work and are encouraged to take this onboard towards successful resubmission at an alternative venue.