Scaling Channel-Adaptive Self-Supervised Learning
Authors: Alice V. De Lorenci
Seung Eun Yi
Théo Moutakanni
Piotr Bojanowski
Camille Couprie
Juan C. Caicedo
Wolfgang Maximilian Anton Pernice
Venue: Accepted by TMLR ()

Abstract: Recent advances in self-supervised pre-training of foundation models for natural images have made them a popular choice for various visual systems and applications. Self-supervised strategies are also promising in non-RGB scientific imaging domains such as in biology, medical and satellite imagery, but their broader application is hampered by heterogeneity in channel composition and semantics between relevant datasets: two datasets may contain different numbers of channels, and these may reveal distinct aspects of an object or scene. Recent works on channel adaptive strategies report substantial advantages for those that account for variable channel compositions without sacrificing the ability to jointly encode channels; yet, how these strategies behave at scale remains unclear. We here show that, surprisingly, trained across large-scale datasets, independent-encoding of channels outperforms joint-encoding methods by a substantial margin. We validate this result along an extensive set of experiments on various datasets from cell microscopy to geospatial imagery. Our DINO BoC approach sets a new state-of-the-art across challenging benchmarks, including generalization to out-of-distribution tasks and unseen channel combinations at test time. We will open source the code, along with model weights that constitute a new general purpose feature extractor for fluorescent microscopy.

Published in Transactions on Machine Learning Research (06/2025)
Scaling Channel-Adaptive Self-Supervised Learning
Alice V. De Lorenci', Seungeun Yi², Theo Moutakanni", Piotr BojanowskiP, Camille Couprie
Juan C. Caicedo", and Wolfgang M. Pernice**
University of São Paulo, Brasil (Work done during on internship at FAIR, Meta.).
Meta FAIR Paris, France, University of Viaconsin-Madiaon, USA, 4 Columbia University, New York USA
Corresponding Authors
Reviewed on OpenReview: https://ppaaeerrrrr.e//////e/tep
Abstract
Recent advances in self-supervised pre-training of foundation models for natural images have made them popular choice for various visual systems and applications. Self-supervised strategies are also promising in non-RGB scientific imaging domains such as in biology, medical and satellite imagery, but their broader application is hampered by hetcrogencity in channel composition and semantics between relevant datasels: two datasets may contain different numbers of channels, and these may reveal distinet aspects of an object or scene. Recent works on channel adaptive strategies report substantial advantages for those that account for variable channel compositions without sacrificing the ability to jointly encode channels; yet, how these strategies behave at scale remains unclear. We here show that, surprisingly, trained across large-scale datasets, independenn-encoodinn of channels outperforms joint- encoding methods by substantial margin. We validate this result along extensive set of experiments on various datasets from cell microscopy to geospatial imagery. Our DINO BoC approach sets a new state-of-the-art across challenging benchmarks, including generalization to out-of-distribution tasks and unseen channel combinations. We open-source code and model weights for a new general-purpose feature extractor for fluorescent microscopy.
1 Introduction
Advances in self-supervised learning (SSL) recently enabled pretraining of powerful general feature ex- tractors across large collections uulabeled natural imaging data (Oquab al., 2023] He et al.) 2022 Chen al. 2020). This approach also holds major promise for scientific imaging domains. Combined with ever more powerful methods to automate image- acquisition across thousands of single cells and con- ditions, foundation models for fluorecent mieroscopy stand to substantially empower biologists to query mechanisms of disease, to accelerate drug discovery, and interrogate cellular biology at unprecedented scale and salient detail (Chandrasekaran et al. 2021). Earth observation foundation models could help ad- dress important socictal challenges, such as by cs- timating aboveground biomass (Muszynski et al.) 2024), tree canopy heights [Tolan et al. 2024), classi- fying methane sources (Zhu al. 2022), and beyond (accoste et al. 2023). Yet, the application of SSL methods in these and related domains face key
Coll typo
IPA-TTV
HRA
CHAMM
Coll ooli st.
DIND BoC (ours)
DINO HA (ours)
Channel-VIT
Figure 1: DINO-BoC is a channel-agnostic approach for SSL in chamel-heeterggenons data domains that outperforms channel-adaptive methods and advances the SOTA across a variety of benchmarks in cellular microscopy (results relative specialized baselines).

---

Published in Transactions on Machine Learning Research (06/2025)
technical challenge: scientific images often feature diverse sets of channels, with variable semantics, tailored to address specific questions. For example, to observe given biological phenomenon of interest, biologists routinely design bespoke imaging protocols to reveal specific sets of cellular structures to the exclusion of others. The data-landscape fluorescent microscopy thus consists of a vast number of small-to-moderatte scale datasets with helerogeneous channel combinations (see Fig. A contrast. to urau (RGB) images, most models trained on one fluorescent microscopy dataset can thus neither be re-used in other studies, nor draw on other datasets, yielding representations of limited expressivity and that generalize poorly (Chen) et al. 2024 Similar issues arise in geospatial data analysis (Fig. 23). Learning powerful unified feature extraction models (i.e. foundation models) for scientific imaging domains therefore requires methods that accommodate heterogeneous channel combinations which we describe broadly as ttannel-adaptive models.
A
Channel content/ semantics of microscopy datasets
B
Aerospatial channels (MeterML dataset)
Protein
Nucleus
DNAIRNA
Tubulin
PM-
ER
Mttcchodtte
NIR
Red
Green
Blue
Conntal
Water
Source datasets
Open Cel
Red
Red Red Red
Short Wive Intrared
Sentrwl
2-20m
Call Pakiing
Sendnal 1-1mmm
Santinal
Any
Punclate i Giotuler
Filmertoos
Ratioulste
Morphological Archetype
Figure 2: Training across datasets with various numbers of channels. (A) We consolidate images from
(HPA-FOV) Le et al. (2022 (HPA-SC), Cho et al. ) (OpenCell), Viana et al. (2023) (PP3))(TTG) (WTC) and Doron et al. (2023 (Cell Painting), into a dataset (extendedCHAMMD that reflects the diversity in channel number, order, and semantics, that characterizes fluorescent microscopy studies. Despite some conventions (e.g. the nuclous usually imaged using blue fluorescing stains), there is no necessary correspondence between specific channels and/or wavelengths, and their biological semantics. Images are pseudo-colored according to emission wavelength. CellPainting visualizes Plasma Membrane (PM) and Actin in channel. (B) In aero-spatial imagery, images acquired at different resolution and using different wavelength acquisition protocols are often available, but it remains a challenge to fully exploit all image modalities. We benchmark models on the MeterML dataset. (Zhu et al. 2022
Human Protcin Atlas (
one sample shown here.
Fluorescent microscopy has proven particularly fruitful test-bed for the development of such methods. As proposed by Xun et al. 2024)) technically simple solution to the challenge of variable channel inputs is to model channels independently by passing them through the model one-by-one, and by reassembling full-image representations by concatenating channel-specific output embeddings post-hoc. This has the theoretical advantage of yielding fully channel- channel-agnostcc agnostic models that are inherently compatible with arbitrary channel sets, but sacrifices the ability to (explicitly) learn chammel-ineeractions. This strategy noticeably breaks with established best practices for RGB images, and one may expect channel-agnostic models to broadly fail in key analytical use cases that require cross- coss-channel reasoning, such the analysis of reference channels that provide ground-truie truth organelle position (Human Protein Atlas 2019 Lacoste et al, 2024b. Consistently, recent works report significant advantages for more complex approaches that reconcile conventional joint-chanmel-encodiing with variable number of input channels (up to some maximum) through customizations of vision transformers (ViTs) (Bao et al. 2024 Bourriez et al. 2024 Pham & Plummer 2024]. Yet, previous studics employed inconsistent sets comparatively small pre-training datasots, often with limited channel- diverrstyy making their results difficult compare. They further relied either only on supervised learning objectives (Bao et al. 2024), and/or did not evaluate generalization to out-of-distributiio
localization relative to
2

---

Published in Transactions on Machine Learning Research (06/2025)
(OOD) tasks and datasets with unseen channel combinations key metrics of success that will determine the utility of general purpose channel-adaptive feature extractors in practice.
We here conduct a first, large-scale comparative study into the scaling properties of channel-adaptive methods, and compare their performance across uniform model architectures, learning objectives, and rigorous benchmarks. We note that inconsistent and often missing labels across microscopy (and other scientific) datasets render supervised methods ill-suited to the task of learning foundation models. We hence focus our work on state-of-the-art (SOTA) SSL strategies which have been recently shown to yield rich representations of cellular morphology channel- homooeeoos microscopy datasets (Doron et al. 2023 Kraus et al. 2024
Our study finds that, at scale, self-supervised ViT models (Oquab et al. 2023 trained on individual channels via a Bag of Channels (BoC) approach siguificantly and outperform models that jointly encode channels across an extensive set of testing regimes in mieroscopy as well as geospatial data. Our results contradict previous reports and challenge the assumption that jointcchamel-oncodinng is beneficial in non-RGB domains. Our main contributions are as follows:
We conduct a systematic comparative study of channcl-adaptive SSL methods across large and diverse datasets.
We report that, surprisingly, independentt-encodingg of channels outperforms joint-chanmel-encodinng strate- gies across an extensive set of experiments including in-domain, cross-dataset, and OOD generalization setups, challenging key assumptions on the optimality of jointchhamel-oncoding in non- Ron domains.
We substantiate our results through a set of control experiments to directly analyze the impact of joint by virtue of a novel channel-adaattiv Hierarchical Attention scheme, as well as an ablation of SSL objectives.
versus independeu-cchanneeeeood
We open-source a now general-purper feature extractor, DINO-BoC, that sets a new SOTA for cross- dalaset learning in fluorescent microscopy and is compatible with arbitrary channel combinations. Our approach also achieves SOTA results on the geospatial Meter-ML dataset when using all channels.
2 Related Work
Self-supervised learning. SSL describes family of methods that leverage general pretext tasks to learn rich and ever more complcte representations of the relationships inherent to a data domain directly from its samples, without the need for supervisory labels. The success of SSL both in Natural Language
Processing (Devlin et al. 2019 Radford et al. 2019 Touvron et al. 2023) and, more recently, in Computer Vision (Chen et al. 2020 Caron et al.) 2021 He et al.) 2022 Assran et al. 2023) can in large part be attributed to well vell-documented - if still only incompletely understood - scaling propertics that in particular transformer-based architectures trained with SSL methods exhibit with growing dataset and model sizes. SSL methods in vision can be broadly divided into generative strategies, such as Masked Autoencoders (MAE) (He et al. 2022), which are trained by reconstructing masked or corrupted regions of the input, and contrastive methods, including DINOv2 (Oquab et al. 2023). the current state-of- state-of-the-art (SOTA) method for large-scale SSL for natural images. We investigate the extent to which the scaling properties of SSL methods generalize to scientific imaging domains with heterogeneous channel composition.
Fixed Channel SSL in fluorescent microscopy. Both contrastive and generative SSL approaches have been recently shown hold promise for the derivation of powerful, unbiased, general-purposs feature extractors for scientific imaging domains, including cellular microscopy (Doron et al. 2023 Kobayashi et al, 2022). However, unable to accommodate heterogeneous channel combinations, these studies remained confined to datasets-specific models with limited scale. Our study pioneers large-scale cross-dataset SSL for domains with variable channel inputs and delivers a new SOTA gencral-purpose feature extractor for this domain.
SSL for geospatial imaging. Similar to fluorescent microscopy, the majority of existing 'foundation models" for geospatial imaging data (Szwareman et al. 2024 Reed et al.) 2023] Cong et al., 2022) are trained fixed sets of channels (bands). This limits their broader application across channel--eteergeeeeuu
3

---

Published in Transactions on Machine Learning Research (06/2025)
Chamellheterogeneous datasets
A
1
(3)
(2)
(3)
1.2
1
B
C
Channel-Adaptive
Channel-Agnostic
(Channel-ViT, ChadAVit, DINO-HA)
(DINO-BoC)
Training Inference
Training (BoC Sampling)
Inference
X1
PV
Mask
Incompatible
(Too many channels)
CLD
0L
Channel-ViT (Kmox
VIT
VIT
Early fusion via coittchannel--ncooding
DINOv2
DINOv2
Default Cross-
Hierarchical Attention (HA)
DINOv2
DINOv2 Y1
711
Late fusion via concatenation and downstream processing
DINOV2
Mask
Figure 3: Strategies for datasets with (A) variable-channel compositions (solid, dashed, dotted borders). (B) Channel-Adaplic methods, once trained, can accommodate variable channel nunbers up maximum (here m4, with full cross-channel computation (akin to early fusion). Inset: Our Hierarchical Attention approach limits cross-channel computation for channcl-adaptive methods to the class token. (C) DINO-BoC is similar to late-fusion multi-modal strategies and trains on individual channels, randomly drawn (thick borders A) from the datasets (no cross-channel computation). On inference, the encoder can be used embed images with arbitrary channel combinations by concatenating channel-wise outputs.
datascts. Moreover, most of these carth obscrvation foundation models use MAE as pre-training strategy. Yet, comparisons on satellite benchmarks (Wang et al. 2022) exhibit superior performance of DINO over MAE. Anysat (Astruc et al. 2024) is the first approach to allow for pretraining on datasets with an heterogeneous number of channels in geospatial imaging. However, this approach relies on domain-specific metadata for positional embedding (based on ground sample distance) that are unavailable other scientific domains. contrast, approach is generic and compatible with any imaging domain.
Channel-Adaptive SSL. To account for variable channel inputs while allowing for cross-channel computa- tion, Channel-ViT (Bao et al. 2024). ChAda- VIT (Bourricz et al. 2024), ViT (Pham & Plummer 2024), and Kraus el al. (2024) convert the variable-channels problem into one of variable sequence length, which transformers natively support. Kraus et al. (2024) uses MAE with channel-specific decoders during training, while Channel- ChAda- ViT. and DiChaViT introduce learnable channolembeedding channel-specific information. (Bao al. (024)) uses supervised learning objectives, and additionally proposes hierarchical channel sampling (HCS) approach during training. (Bourriez et al. 2024) trains with DINO
preserve
4

---

Published in Transactions on Machine Learning Research (06/2025)
Caron al. 2021 with ViT-S on relatively small datasets. We uniformly use ViT-L architectures, and sys- tematically compare the scaling properties of channel-adaptive jointchhamel-enoddin to channelinndeeenden encoding across large and diverse pre-taining datasets and extensive benchmarks.
In particular, an alternative approach for cross-dataset learning in variable-channel domains is to ignore the association between channels altogether, by training on single-channel (i.e. gray-scale) inputs, yielding Channel-Agnostee models. CytolmageNet (Hua et al., 2021) proposed collapse channels into one by averaging across them. This approach irrevocably discards the semantic information carried by distinct channels. Closest to DINO-BoC, Microsnoop (Xun et al., 2024) instead trains individual channels sampled from small custom dataset (about 10k images) that includes fluorescent microscopy images, using a convolutional U-Net [Ronneberger et al. 2015) with a masked SSL strategy. Whole-image representations are reassembled post-hoc by concatenating output embeddings for each channel. However, pre-training data and training code are not available. We here study BoC sampling for large-scale channel-heterogeneon cross-dataset training, using ViTs and SOTA SSL objectives, in systematic comparison to contemporary channel-adaplive methods, and two distinct scientific imaging domains. DINO-BoC outperforms Mierosnoop by large margin and is fully open-source. While different in approach, we here conceptually treat BoC approaches. We further explore the continuum between models
models as a special case of with full vs. no cross-channel computation by virtue of a new Channel- -ndative ViT architecture with a Hierarchical Attention (HA) mechanism (see Fig. 3)
Multi-modal learning. Channel-adaplive learning shares technical similarities with multi-modal learning if one were to interpret channels as different data modalities, For example, MultiMAE (Bachmann et al., 2022) extends the MAE pre-training objective to handle a variable number of modalities by randomly sampling patches from distinct modalities that are passed through a common transformer encoder. The training objective is to reconstruct the masked-out patches using task specific decoders. By building the input sequence with patches from different modalities, MultiMAE resembles -nnne- strategies. Contrastive Icarning has also been explored in the context of multi-modality (Liang et al. 2023). Here, however, the different views are built from different modalities, whereas in case of channel-adaptive models, views are constructed through image augmentations (e.g. random cropping), such that each view contains the same channels. (Madaan et al., 2024) consider the dichotomy and possible tradcoff between capturing inter- vs. modalit dependencies, albeit in a supervised learning scenario. In contrast to channel-adaptive strategies that encode channels jointly, BoC models cannot explicitly learn inter-channel relationships and hence allocate all capacity intra-channel dependencies. Finally, the multi-modal literature distinguishes approaches performing early fusion of features at the first layers of models late fusion, i.c. feature combination at the last layers. Analogously, joint-chamel-encoding strategies perform early fusion, whereas the BoC approach resembles late fusion methods (Fig. 3
3 Method
Given channel-het dataset X, where sample z(f) has Kj channels, let K Kmax denote the maximum number of channels across all images. The goal of channel-adaptive models is accommodate images with a heterogencous channel combinations.
3.1 Channel-Adaptive SSL baseline: Channel ViT
We implement joint-chammo-encoddin strategies following Channel-ViT (Bao et al. 2024 ChAda-ViT
[Bourriex et al.] 2024). and DiCha VIT (Pham & Plummer 2024). that adapt the patchfication process of standard ViTs and add channel embeddings (sce Fig. 3B, Fig. 3A). Specifically, given an input image with K
channels x E IR xxW where (H, W) is the ion of the original image, and a patch size S, we reshape
an image into a sequence flattened single-channel patches of length N KHW/S?:
TK,N] Fkis T
x F1,1
TI,N
FK,1
5

---

Published in Transactions on Machine Learning Research (06/2025)
where, in 2k,i, k indicates which channel the patch belongs to and its raster position. A linear embedding layer is applied to each patch Ti,i resulting in a sequence of patch embeddings hę of dimension D, to which a learnable class token TCLS is prepended. To retain channel information, learnable channel embeddings are added along with the usual positional embodding to the patch emboddings hg.5. Let ps (i (i= ,.....,. denote the positional embeddings and 6 (k = .....: K) the channel embeddings. The resulting sequence of patch embeddings for a sample x is:
[ECLS 11, + P1+ + hg.N + PN + ck] E RDX(LNNK)
result, the variable number of channels problem turns into one of variable sequence length, benefiting from the transformer's inherent capability handling arbitrary sequence longths. Note that if the maximum number of channels per image the pre-training data Kmax; this method canmot be used on images that. have more channels than Kmax: as there will be no trained channel embeddings for the extra channels. We scale this approach using ViT-L architecture and update the pre-training recipe to the improved DINOv2. For terminological convenience we refer to these model configurations as Channel- ViTs (Fig, 33).
3.2 Channel-Agnostic SSL: DINO-BoC.
A straightforward strategy to deal with channel number variability to separately encode each channel, using common backbone (Figure 3C); we will refer this strategy as the Bag of Channels approach. During training, each imput to the model consists of a random single channel extracted from random multi-channel image, reshaped into a flattened of sequence of length N = HW/52:
I2 SR
11
EN
After projecting the patches with linear embedding layer, add positional embeddings and prepend the class tokens to obtain for sample x:
ECLS h1 P1
hN + +NN] E Dx(LNN)
At inference time, features independently extracted for each channel of the image using the shared backbone. These channel-wise features are then aggregated e.g., via concatenation to obtain image- level representation.
3.3 Channel Adaptive Hierarchical Attention model
This work introduces a novel baseline that balances joint. and aadepenneen-chamn--enoood strategies. This method can be seen as exploring the trade-off between Channel-Vizs and DINO BoC, by limiting inter-channel interactions.
In this approach, the image is reshaped into sequence of single-channel patches x E S*NK After embedding the patch tokens with lincar layer, both global class token #ClS and channel class tokens ICH are inserted into the sequence, only position embeddings are used:
JECLS &CH, 11.1 +
J1,N + PN
&CHg bK,1 + +PI
hKNN + +
A specialized attention mask is employed in the Multi-Head Self-Attention blocks. Tokens within a single channel (channel class token and corresponding patch tokens) can only allend to other tokens within that same channel, therefore, at this level, the channels are processed independently. The global class token, however, attends all channel class tokens, enabling inter-channel reasoning a higher semantic level. This approach, termed DINO Hierorchical Attention (DINO HA) model, and the attention mask are illustrated in Figure 4 B,D (see Appendix D for more details).

---

Published in Transactions on Machine Learning Research (06/2025)
4 Experiments
We test the merits of DINO BoC on diverse benchmarks, and compare it to existing channel-adaptive
strategies. After introducing datasets and implementation details in Section 4.1 Section
4.2
demonstrates
the impact of choosing the SSL. DINOv2 method (Oquab et al. 2023) instead of MAF He et al. 2022
and compares DINO BoC to Channel-ViT and Microsnoop {Xun ct al. 2024). In section 4.3 we further
investigate the advantages of DINO BoC on cross-datasel generalization tasks. Then, in section 4.4 we
evaluate self-supervised DINO BoC, Channel-Vill ViT, and DINO HA models in the CHAMMI benchmark
designed to assess performance in in-distribution and OOD tasks compared to statc-of-the-art supervised channel-adaplive approaches. Section 1.5 compares results achieved on an aero-spatial dataset. Finally, Section 4.6 assesses the impact of the BoC strategy with respect to usual training on RGB images.
4.1
Datasets, benchmarks and implementation details
4.1.1
Datasets
We leverage multiple microscopy datasets with varying numbers of channels. In particular, we use the Human Protein Atlas, WTC-11, JUMP-CP and Cyclops datasets for evaluation tasks. Additionally, we employ the CHAMMI benchmark, a standardized evaluation framework for channel-adaptive models.
Human Protein Atlas dataset. The subset of the Human Protein Atlas (HPA) data that is considered is the one of the Kaggle competition Human Protein Atlas (2019), concerned with the subcellular distribution of the proteins encoded by different genes. It covers 35 cell lines and 28 subcellular structures of protein localization. There are 113, 545 images total, with four channels. There is also single cell version of the same dataset (Le et al. 2022) obtained through segmentation of the field-of-view (FOV) images. The HPA Single Cell dataset contains 839, 612 images.
WTC-11 dataset. This dataset is a version of the WTC-11 hiPSC Single Image Dataset v1 (Viana ctt al. 2023) of the Allen Institute curated for the CytoData Symposium 2022 hackathon (Allen Instituto 2022). The dataset contains 214, 037 3D images of cells, we used the maximum z-projection of the original images. The dataset provides cell-cycle stage annotations, with six stages. The images have one bright-field (BF) channel and three fluorescence channels; BF was excluded.
Cell Painting dataset. We utilize the Cell Painting (CP) dataset curated by Moshkov et al. (0224). totaling 8, 423, 455 images with five channels. The dataset has the objective of allowing the study of the response of cells to different compound treatments and gene over-expression experiments.
JUMP-CP dataset. This datasel, used by [ao et al. (2024), is processed version of the data made available by the JUMP-Cell Painting Consortium (Broad Institute 2021 It contains 229, 228 single cell images. We used only the five fluorescence channels in our work.
Cyclops dataset. We used the same dataset as described by Xun et al. 2024 consisting of 28,166 2-channel yeast, cell images from the Cyclops database (Luu et al. 2018).
OpenCell dataset. The OpenCell dataset was introduced by Kobayashi et al. (2022) and encompasses 3,111 different tagged proteins. In total, 1, 134, 592 images were made available, with two fluorescence channels. More details on the five datasets mentioned above are given in Appendix B
CHAMMI benchmark. The CHAMMI benchmark (Chon et al. 2024) includes dataset curated from the WTC-11, HPA Single Cell and Cell Painting datasets. In total there are 220, 284 images, of which 100, 145 are used for training. The benchmark is standardized evaluation framework for channel-adaptive models. It presents a comprehensive set of nine tasks for channel-adaptive models of varying complexity, that evaluate the ability of the models to generalize to new biologteally-rolevant experimental regimes. As such it positions itself as an indispensable benchmark to evaluate those models. The images from each data source present in the CHAMMI dataset are split into one training set and several test sets, designed for specific tasks, Tasks with suffix 1 are IID classification problems, where the test and train data follow the same distribution. Tasks with suffix greater than 1 evaluate the OOD generalization capabilities of the model, and simulate biologically-reeevan application scenarios (see Appendix C.1 for a detailed description of each task)
7

---

Published in Transactions on Machine Learning Research (06/2025)
CHAMMI dataset. We extend the CHAMMI train set to total of 7, 748, 662 images, incor- porating additional data from both the source datasets and new data sources. The extended training dataset preserves the OOD characteristies of the CHAMMI tasks (see Appendix C.2p.
Meter-ML dataset. The Meter-ML dataset contains images acquired by multiple sensors: four channels for NAIP (National Agriculture Imagery Program in the USA) images at resolution 720 x 720, four channels from Sentinel-2 at resolution 72 72, and lower resolution images from Sentinel-2 (S2) and Sentinel-1 (S1). The task consists in classifying sources of methane emissions in six categories (CAFOs, Landfills, Mines, ProcessingPlantss RefineriesAAndTerminnd WWTreatment).
4.1.2 Implementation details
When pre-training the models, care was taken to ensure that the models processed the same amount of data. For the DINO BoC model, sample consists of single channel, whereas for the other models, a sample is an image with all of its channels. Therefore the former must be trained for more iterations to achieve fair comparison. For each pre-training dataset, the Channel-Vii and DINO HA models were pre-trained for 45, 000 iterations. Taking into account the average number of channels in the pre-training dataset, the Bag of Channels model was trained for a proportionally larger number of epochs. The batch size used was of 1024 for all models, and the batch size per GPU was set to 8, except for DINO BoC model, for which 32 fits in memory. On our largest dataset, we trained the models for about 2 days using 16 nodes, or 4 nodes for the DINO BoC model. Unless specified otherwise, we trained ViT large models. More details are provided in Appendix M
On the ExtendedCHAMMM dataset, the channel-adaptive models were pre-trained with balanced sampling across data sources.
4.2 Scaling channel agnostic feature representations with DINOv2
To assess the merits of channcl-adappive SSL strategies at scale, we assembled CxtMn as a large and diverse pre-training dataset, and compared the performance of channe-adapptie models on HPA- FOV, WTC, and JUMP-CP test-sets against fixed-channel baselines (Doron et al. 2023) pre-trained on ExtendedCHA.MII subsets with channels matching the respective target test sets (Table 1b. The JUMP-CP dataset is not included in Exten)edCHAMMII therefore it evaluates the generalization capability of channel- adaptive models novel data sources. Results for ablation removing datasets from the CHAMn datasct are presented in Appendix I and results all cight JUMP-CP channels are listed in Appendix J
Surprisingly, we observed that models trained to independently encode channels (BoC), especially when trained with DINOv2, outperform joint-channel encoding models (Chamnel-Viil across all tasks, and often with a substantial margin. DINO BoC also performs better than fixod-channel models on three out of four lasks, including the novel JUMP-CP dataset. Notwithstanding intuitions about the value of learning channel interactions in sele-supervised pretraining, these results indicate that, at larger scale, BoC models are more effective in leveraging diverse microscopy data, and learn improved encoders compared to SOTA channel-adaptive methods that encode channels jointly. Although DINOv2 substantially outperforms MAE, our results suggest that BoC's advantages hold for her SSL objectives as well (Table [.. DINO HA performs on par with DINO BoC on this IID benchmarking setup. Next, we evaluated the impact of the network size on DINO BoC. Remarkably, while ViT-L expectedly leads to improved performance, even with a ViT-S, DINO BoC outperforms both ViT-I, MAF BoC and Channel-ViT models.
Importantly, in several domains, MAE has been shown to yield features that are particularly well-suited to fine-tuning (He et al. (2022); Girdhar et al. (2023): Huang et al. (2022). ). We hence also compare DINO BoC with MAE pretraining in a finetuned setting, which boosts the performance of all models all datasets, excepted for Cell Types classification. Of particular relevance to practitioners, our results indicate that DINO BoCperformance advantage over MAE persists in fine-tuning regimes, with consistently better results. On the two most challenging tasks, namely protein localization, and JUMP-CP perturbation classification, the performance boost brought by DINOv2 is particularly strong (MAF-Bor A.EBB.C: +2.5, DINO-BoC +4.7). Absent the ability of directly pre-training Microsnoop on ExtendedCHAMMI (training code is not available), the performance of MAE BoC serves as proxy for Microsnoop performance, as it uses the same SSL method
8

---

Published in Transactions on Machine Learning Research (06/2025)
employed in Table In addition, we demonstrate in Table 2 that DINO BoC outperforms Microsnoop the challenging Cyclops dataset (on which Microsnoop reported the largest gains) by substantial margin, while especially fortifying performance on rare classes (up to 20 points). Unexpectedly, the BoC approach, and DINO BoC in particular, thus not only matches, but outperforms jointcchamel-encodiin inchuding on protein-localizttion prediction, setting new state-of-the-arl.
Table 1: Comparison of channel-adaptive models trained on the ExtendedCHAMMI dataset and fixed-channel models. The first three rows (CelIDINO) are fixed-channel baseline models, separately pre-trained either on the HPA-FOV, JUMP-CP or WTC datasets. Best channel-adaptive results in bold; best results overall with frozen backbones underlined. Note that the fixed-channel models can only be evaluated on data with matching number of channels.
Model
SSL
Network size
Chaunel adaptive
Training set
HPA-POV F1 Protein loc.
HPA-FOV FI JUMP-CP
WTC F1
method
Cell type
Accuracy
Cell cycle
CHUIDNNO2
DINO
VIT-L
HPA-POV JUMP-CP
65.0
80.3
CellDINOv2
DINOv2
VIT-L
44.3
CelIDINOv1
DINOv1
VIT-L
X
WTC
82.3
Channel-Vai VIT
DINOv2
VIT-L
V
ExtendedCHAMMI
57.4 -14
90.4 -a.a
39.4 -co
87.2 eva
DINO
DINOv2
VIT-L
ExtendedCHAMMM
61.4 aa
457
MAE
VIT-L
54.0
90.8 41.5
39.3 -s.o
89.4
BoC
DINOv2
VII-S
/
ExtendedCHAMMMI
55.6 -44
90.7 -3.4
14.5 402
91.0 +6.7
DINOv2
VIT-L
61.7 .a.a
91.1 +1.
45.2 +4.0
90.5 +x.3
BoC fine-tuned
MAE
VIT-L
/
ExtendedCHAMMI
64.3
90.1
57.5
93.1
DINOv2
VIT-L
66.8
90.4
62.2
93.6
Table 2: Comparison to Microsnoop on the Cyclops dataset. DINO BoC dramatically outperforms Microsnoop, especially on the least frequent classes (out of 16).
Class
Budtip
Cell periphery
Budneck 2.4%
Actin 3.8%
All
frequeney
1.5 %
1.9%
Mierosnoop [unn et al.
2024)
32.1
96.4
43.4
48.0
75.9
DINO BoC
62.1
97.5
72.6
63.7
83.1
4.3
Cross- dataset generalization
Despite exhibiting superior scaling properties (at least for fluorescent microscopy data), it is not clear whether independent-channel encoding also provides benefits when confronted with unseen datasets and channel combinations key application for foundation models in this field. We hence compared cross-dataset generalization capabilities of DINO Channel-VITY VIT, DINO HA and DINO BoC models. To this end, we trained models exclusively either HPA-FOV or JUMP-CL -PP, and evaluate them HPA-FOV, JUMP-CP and WTC test sets. Once morc, we find that DINO BoC outperforms DINO Channel- ViT on all cross-datasct tasks (Table 3 Noticeably, these results suggest that cross-channel reasoning is detrimental for the emergence of general, rather than specialized, dataset-specific features via SSL pretraining. DINO HA yields consistently better performances than Channel- Nevertheless, DINO HA still falls behind DINO BoC(Table 3), corroborating the conclusion that ceappendent-chaan--neeeeee is the winning strategy for channel-adaptive models.

---

Published in Transactions on Machine Learning Research (06/2025)
Table 3: Cross-dataset generalization of channel-adaptive models. DINO BoC shows superior performance on unseen channel combinations. In-dataset results are shown in gray for reference.
Model
Channel adaptive
Training set
HPA-FOV F1
НPA-FOV F1
JUMP-CP Accuracy
WTC F1 Cell cycle st.
Protein loc.
Cell type
DINO Channel-ViT
HPA-FOV
65.5
90.9
35.3
80.0
DINO HA (Ours)
HРA-FOV
66.7
91.3
37.3
88.9
DINO BoC (Ours)
/
HPA-FOV
65.2
91.5
40.2
89.8
DINO Channel-ViT
/
JUMP-CP
29.5
82.0
53.4
81.8
DINO HA (Ours)
/
JUMP-CP
30.3
85.2
52.0
84.2
DINO BoC (Ours)
JUMP-CP
31.6
85.0
41.3
90.5
4.4
Out-of-distribution generalization CHAMMI
We next assessed out-of-distribution generalization of channel-adaptive models leveraging the challenging CHAMMI benchmark set. Table 4 reports results for the 1-NN classifier protocol as defined by Chen et al. (2024). Given the modest size of CHAMMT's training set (ca. 100k images), the CHAMMT benchmark thus far is dominated by supervised channel-adaptive models, trained from scratch, that leverage labels for training. The top-performing model, HyperNet, inspired by Hypernetworks (Ha et al. 2017), uses a CNN backbone and an MLP that generates kernel weights for the initial convolutional layer of each input channel. Consistent with our previous results, we find that DINO BoC outperforms DINO Channel-ViT on OOD tasks for two out of three datasets. Remarkably, DINO BoC further exceeds the best supervised baseline on average OOD scores for WTC (5.3%) and CP (8). For completeness, we also report DINO HAresults, which underperform.
SSL methods tend to benefit from pre-training on a larger corpus of data. Consistently, we find that pretraining on our ExtendedCHAMMM dataset further improves the performance of channel-adaplive SSL models (Table 5) compared to the default CHAMMI training set (Table + even though part of the additional data comes from datascts unrelated those on which the models are evaluated on.
Table 4: F1 scores for 1-NN search the CHAMMI test set. The models were pre-trained on the CHAMMI train split. Lines 1-6 report the results of Chen et al (2024) for CNN-based models trained from scratch in supervised fashion. Line reports the performance of FixedChannels, that consists a scparate model trained for cach fixed channel combination. Lincs 2-6 are channel-adaptiee models. Lincs 7-8 are self-superviee isedd channel-adaptive ViTs. Best results between channel-adaplive self-supervised approaches are bold.
Average
OOD
WTC
HPA Task2
CP
Model
Mean
WTC
HPA
CP
Tuskl
Task2
Task1
Tusk3
Task1
Task2
Task3
Task4
PixodChsuucls
50.0
64.8
59.2
25.9
64.9
64.8
80.7
76.3
42.1
66.0
48.1 1
23.0
6.6
supervisod
Depthwise
51.7
65.2
61.4
25.6
65.9
65.2
84.9
81.3
47.5
67.3
47.8
22.6
6.5
TargetParam
49.6
59.0
62.3
27.3
69.5
59.0
83.7
79.4
45.2
71.7
50.8
23.4
7.7
SlieeParam
45.7
56.8
54.6
25.6
61.6
56.8
77.0
69.0
40.3
64.6
47.5
22.2
7.1
HyperNet
53.7
66.1
67.1
27.8
72.6
66.1
88.7
85.8
48.3
72.0
51.7
24.7
6.9
Template mixing
46.6
56.5
57.7
25.7
63.1
56.5
80.8
74.1
41.3
67.1
46.8
22.7
7.5
DINO Channel-Vii VII
42.6
45.3
53.6
29.0
68.7
45.3
92.2
65.2
42.0
95.2
51.5
25.2
10.3
DINO НA (Ous)
40.4
46.3
47.4
27.5
62.4
46.3
88.7
59.9
34.8
94.3
59.0
16.6
6.8
DINO BoC (Ours)
48.8
71.4
43.3
31.6
79.4
71.4
87.0
56.4
30.2
93.5
20.1
4.4.1 Linear probe for CHAMMI
We note that supervised methods are prone encoding spurious correlations inherent to training sets, leading
to poor performance OOD tasks (Pernice al. 2023
Mao et. al. 2021
Barbu et al. 2019) By contrast,
10

---

Published in Transactions on Machine Learning Research (06/2025)
Table 5: F1 scores for 1-NN search on the CHAMMI test set. The self-supervised models were pre-trained on the ExtendedCHAMMI dataset. Line 1 presents the best performing supervised baseline (HyperNet), which can only be trained on the annotated subset of CHAMMI.
Average OOD
WTC
HPA Task 2
CP
Model
Mean
WTC
HPA
CP
Task 1
Task
Task 1
Task 3
Task 1
Task 2
Task 3
Task 4
HyperNet
67.1
72.6
66.1
88.7
48.3
51.7
6.9
DINO Channel- ViT
46.2
55.6
28.9
64.5
92.1
45.9
53.5
11.3
DINO HA (Ours)
43.2
54.3
46.9
28.4
66.4
54.3
90.1
60.4
33.5
93.6
57.1
15.8
12.1
DINO BoC (Ours)
51.6
79.0
43.0
32.7
79.4
79.0
86.6
59.3
29.6
92.6
57.6
22.1
18.5
our results in CHAMMI's 1-NN OOD regime indicate that self-supervision, and channcl-adaptive SSL with BoC in particular, yields less biased representations that are substantially more robust to domain shifts. Still, while supervised methods that naturally promote clustering of the data in the embedding space according to their labels, SSL methods have been shown to yield nested embedding spaces (Doron et al. 2023). For example, for the HPA task, DINO BoC features first cluster by cell type, while the protein localization are retained as nested factor of variation (see Appendix G). This organization is likely suboptimal for ncarest-neighbor classification of c.g. novel cell type (HPA Task 2), or for a known cell type but novel protein localization calegories (HPA Task 3). We hence further evaluated model performance using linear probe.
Table 6: F1 scores for a linear probe on CHAMMI test set. Self-supervised models were pre-trained on the ExtendedCHAMMI dataset, while the supervised HyperNet was pre-trained on CHAMMI.
Average OOD
WTC
HPA
CP
Model
Mean
WTC
IIA
CP
Task1
Task2
Task1
Task2
Task 3
Task1
Task2
Task3
Task4
NeterNet
85.3
82.9
27.9
87.9
85.3
94.4
92.5
73.2
93.5
51.5
17.3
15.0
DINO Channel VrT
59.8
66.9
76.7
35.9
83.1
66.9
88.2
84.9
68.4
80.5
54.5
23.3
30.0
DINO HA (Ours)
62.7
76.2
72.4
39.5
68.0
76.2
88.5
82.4
62.4
91.7
61.6
27.5
29.3
DINO BoC (Ours)
89.2
74.9
39.7
90.5
88.3
84.7
65.0
90.5
60.5
25.8
32.7
Linear probing is also of particular interest for DINO BoC: unable to learn channel interactions explicitly, we note that DINO BoC, on 1-NN evaluation, underperforms both Channel-ViT and supervised baselines on the HPA protein localization tasks, which require spatial reasoning across channels. Even the minimal opportunity provided by linear probing disentangle relevant from irrelevant information, and for DINO BoC in particular, to relate information across channels, may thus further improve the performance of SSL models.
Consistently, comparison of Tables and shows that the use of a linear probe substantially improves the scores of SSL channel-adaptive models overall, and of DINO BoC on HPA Task 2 and 3 in particular. In result, DINO BoC performs comparably to Channel-ViT, as well as DINO HA, on all protein localization tasks. In result, DINO BoC surpasses other models, including the HyperNet supervised baseline, on the mean OOD score, suggesting that ample information suitable to cross- crasnchannee integration preserved in its channel-specific embeddings to be readily utilized post-hoc. This highlights the potential of this simple strategy to yield powerful biological feature extractors for wide-ranging applications.
4.5 Experiments on geospatial imagery
We next asked whether the merits of DINO BoC are limited to fluorescent microscopy applications, or whether the approach could be useful in other domains. We hence evaluated DINO BoC on the geospatial dataset Meter-MI, introduced by Zhu et al. 2022
(Table 7
Zhu et al. (2022) showed that the highest resolution NAIP images lead to better performance, and that using all bands could improve the result of one class accuracy. We thus trained one DINO BoC model on NAIP,
and model on all all the S1, S2 and NAIP channels.
11

---

Published in Transactions on Machine Learning Research (06/2025)
Table 7: DINO BoC outperforms the state-of-the-art models when using all channels on the Meter-ML dataset. Top: AUROC of models trained on NAIP and Sentinel data. Bottom: AUROC of models using only NAIP channels at inference.
Approach
Architecture Channel adaptive
Test dataset
Pre-training dataset
Meter-J ML Zlu al.
2222
1euseNet-121
NAIP, S1
ImageNNel
51.7
LIIRS-bot
Muhttr
2021
VLM
NAIP, S2, S1
LIRS-AAlig-Reess (1.1M images & text)
71.8
VHM [ Pang al
2024
VLM
X
NAIP, S2, S1
VersaD (1/M images & text)
72.7
DINO Channel VIT
VIT-L
NAIP, S1, $2
NAIP, S1, S2 train (85K)
64.4
DINO BoC (ours)
VIT-L
NAIP, S1, S2
NAIP, S1, S2 train (85K)
78.1
Approach
Architecture
Channel adaptive
Test dataset
Pre-training dataset
mAP
Meter-ML Zhu et al.
NenseNet-121
ImageNet
Supervised Cong
2022
VrT-L
NAIP
MoW Sentinal (770K)
MatMAE
Cong,
422
VIT-L
NAIP
FMo Sentinel (770K)
76.9
ScaleMAE Reed
al.
2123
VIT-
X
NAIP
[MoW RGB (363K)
78.4
USaLMAE Irvin dl al.
2023
VIT-L
X
NAIP
USAdla NAIP (3.6M)
83.7
DINO Channel
VIT-L
NAIP
NAIP, S1, S2 (S5K)
63.8
DINO Channel VIT
WI-L
NAIP
NAIP (85K)
69.3
DINO IA
VIT-
NAIP, (S5K)
77.3
DINO IA
VIT-L
NAIP
NAIP (85K)
77.6
DINO BoC [ours]
VTE.
NAIP
NAIP, S1, S2 (85K)
81.3
DINO BoC (ou7s)
VIT-L
7
NAIP
NAIP (85K)
81.9
Using the exact same normalization and evaluation protocol as for the microscopy benchmarks, DINO BoC outperforms the state-of-the-art by a large margin when using all channels. Using only NAIP imagery, it is close to state-of-the-art approaches, even though it is pre-trained with much smaller datasets (two orders of magnitude less than the approach Irvin et al. (2023)), and does not use remote sensing specific architectures. This strongly supports the generality of our approach wider range of imaging domains.
4.6 DINO BoC on ImageNet
Given major performance gains we observed for DINO BoC, we wondered whether the merits of independent processip are limited technical imaging domains in which (in contrast RGB images) channels
typically capture highly uncorrelated, distinct structures (Figure 2 (Bao et al. 2024). Therefore, we assess
the impact of the DINO BoC default DINOv2 (Oquab et al. 2023 on the classical ImageNet dataset.
Deng el al. 2009h. Top-1 classification accuracies are provided in Table
Table 8: Mean accuracy on the ImageNet 1k val set. VYT-L models, linear evaluation.
Model
Channel adaptive
Batch size Channels
Top-1 Accuracy
fixed channels fixed channels
1024
3072
ImageNet1K
81.9
3072
9207
ImageNet1K
83.1
DINO HA DINO BoC(Ours)
1024 3072
ImageNetIK
78.6
3072 x 3072
ImageNet1K
82.8
Surprisingly, removing joint-channe embeddings prior seems to be beneficial even for extraordinarily well-explored domains like RGB images. With a fixed number of data seen during training (lines one and three, 81.9 vs 82.8), the comparison is favorable to DINO BoC. If we compare results given a constant amount of gradient steps, the accuracy of the fixed channels approach is only slightly higher than DINO BoC (0.3 points). Given the vast flexibility of the BoC approach in terms of applications, this small difference only underlines the potential of DINO BoC. For completeness, we also included comparison to DINO HA with lower batch size to make it fit into memory.
12

---

Published in Transactions on Machine Learning Research (06/2025)
5 Conclusions
We report results on a large-scale study into self-supervised channcl-adaptive training strategics, as step towards general-purpose feature extractors for imaging domains with heterogeneous channel combinations, and in particular for fluorescent microscopy. Scaling the BoC approach using Vision Transformers and the state-of-the- self-supervised DINOv2 method, DINO BoC outperforms models that rely on inter- channel reasoning, and positions as the leading channcl-adaptive approach. In addition to its strong performances, DINO BoC is notable for its simplicity, lacking any priors; whereas joint-encoding methods adapt to variable channel numbers up to some maximum (see Appendix J], DINO BoC is channel- agnostic, rendering it suitable to arbitrary channel combinations. We also note that even the thcoretical advantage of a uniform embedding space produced by joint-encoding methods (Bourriez et al. 2024) compared to BoC in practice remains unclear (see Appendix H). Instead, we show that DINO BoC substantially outperforms Channel- ViT in generalization to unseen channel combinations, and OOD tasks at test time. More broadly, our results call the utility of joint-channel-encee question. Indeed, find that the DINO BoC approach achieves performance par with state-of-the-art results for geospatial images, well as RGB images, out-of-tho-box. The code is available at https ADAPPIEE DINTIVEEDDIOOO.. md
as a prior for SSL pretraining into
References
Allen Institute. Cylodata symposium 2022. http:///aeeutttuoooo//////////ud
2022.
Mahnoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogration, 2023.
Guillaume Astruc, Nicolas Gonthier, Clement Mallet, and Loic Landricu. Anysat: An carth observation model for any resolutions, scales, and modalities. arXiv preprint arXiv:2412.4412, 2024.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimac: Multi-modal multi-tan task masked autoencoders. In European Conference on Computer Vision, pp. 348-367. Springer, 2022.
Yujia Bao, Srinivasan Sivanandan, and Theofanis Karaletsos. Channel vision transformers: An image is worth c x 16 x 16 words. Tn International Conference on Learning Representations, 2024.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Nicolas Bourriez, Thab Bendidi, Ethan Cohen, Gabriel Watkinson, Maxime Sanchez, Guillaume Bollot, and Auguste Genovesio. Chada-vit: Channel adaptive attention for joint representation learning of heterogeneous microscopy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
M.A. Bray, S. Singh, H. Han, et al. Cell painting, a high-content image- image-based assay for morphological profiling using multiplexed fluorescent dyes. Nature Protocols, 11:1757-1774, 2016.
Mark-Anthony Bray, Sigrun M. Gustafsdottir, Mohammad H. Rohban, Shantanu Singh, Vebjorn Ljosa, Katherine L. Sokolnicki, Joshua A. Bittker, et al. A datasct of images and morphological profiles of 30 000 small-molecule treatments using the cell painting assay. GigaScience, 6(12), 2017.
Broad Institute. Jump-cell painting consortium.
2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
13

---

Published in Transactions on Machine Learning Research (06/2025)
Srinivas Niranj Chandrasckaran, Hugo Ceulemans, Justin D. Boyd, and Anne E. Carpenter. Image-based profiling for drug discovery: due for machine-learning upgrade? Nature Reviews Drug Discovery, 20: 145 149, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning, 2020.
Zitong Sam Chen, Chau Pham, Siqi Wang, Michael Doron, Nikita Moshkov, Bryan Plummer, and Juan C Caicedo. CHAMMT: A benchmark for channel-adaptive models microscopy imaging. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2024.
Nathan H Cho, Keith C Cheveralls, Andreas-David Brunner, Kibeom Kim, André C Michaelis, Preethi Raghavan, Hirofumi Kobayashi, Laura Savy, Jason Y Li, Hera Canaj, et al. Opencell: Endogenous tagging for the cartography of human cellular organization. Science, 375(6585), 2022.
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Newral Information Processing Systems, 35:197 211, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pollern recognilion, pp. 248 255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. Tn Proceedings of the 2019 Conference of the North Americon Chapter of the Association for Compulational Lènguistics, 2019.
Michael Doron, Théo Moutakanni, Zitong S Chen, Nikita Moshkov, Mathilde Caron, Hugo Touvron, Piotr Bojanowski, Wolfgang M Pernice, and Juan C Caicedo. Unbiased single-cell morphology with self-supervised vision transformers. bioRziv, 2023.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEFE/CC /CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2023.
David Ha, Andrew Dai, and Quoe Lc. Hypernetworks. In International Conference on Learning Represen- lations, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Stanley Bryan Z Hua, Alex X Lu, and Alan M Moses, Cytoimagenet: A large-scale pretraining dataset for bioimage transfer learning, 2021.
Po-Yao Huang, Hu Xu, Juncheng Li, Alexci Bacvski, Michael Auli, Wojeicch Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35:28708 -87200, 2022.
Human
Human protein atlas image classification. httts://wwwwkaagle..om /wtpp//tiiisos//nmaaaaa wwwpuauunnnnnannnnaeeeaaeaeeeeee ppoteinnattss-ina---lsiiiffieee cmass-classification coneiiiication/vervvieee /overview, 2019.
Protein
Atlas.
Jeremy Irvin, Lucas Tao, Joanne Zhou, Yuntao Ma, Langston Nashold, Benjamin Liu, and Andrew Y Ng. Usat: A unified self-supervised encoder for multi-sonsor satellite imagery. arXiv preprint Xrxi:22312.00 021999, 2023.
Hirofumi Kobayashi, Keith C Cheveralls, Manuel Leonetti, and Loic A Royer Self-supervised deep learning encodes high-resolution features of protein subcellular localization. Nature Methods, 19:995-100 1003, 2022.
14

---

Published in Transactions on Machine Learning Research (06/2025)
Oren Kraus, Kian Kenyon-Dean, Saber Saberian, Maryam Fallah, Peter McLean, Jess Leung, Vasudev Sharma, Ayla Khan, Jia Balakrishnan, Safiye Celik, et al. Masked autoencoders for microscopy are scalable learners of cellular biology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan Sherwin, Hannah Kerner, Björn Lütjens, Jeremy Irvin, David Dao, Hamed Alemohammad, Alexandre Drouin, et al. Geo-bench: Toward foundation models for earth monitoring, Advances in Newral Information Processing Systems, 11551800-10990 2023.
Jessica Lacoste, Marzieh Haghighi, Shahan Haider, Chloe Reno, Zhen-Yuan Lin, Dmitri Segal, Wesley Wei Qian, Xueting Xiong, Tanisha Teclucksingh, Estchan Miglictta, et al. Pervasive mislocalization of pathogenic coding variants underlying human disorders. Cell, 2024.
Trang Le, Casper F Winsnes, Ulrika Axclsson, Hao Xu, Jayasankar Mohanakrishnan Kaimal, Diana Mahdessian, Shubin Dai, Ily Makarov, Vladislav Ostankovich, Yang Xu, et al. Analysis of the human protein atlas weakly supervised single-cell classification competition. Nature Methods, 19:1221-1229, 2022.
Paul Pu Liang, Zihao Deng, Martin Q Ma, James Y Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36:32971 -17998,,98,.. 2023.
Alex Lu, Yolanda T Chong, Ian Shen Hsu, Bob Strome, Louis-Francois Handfield, Oren Kraus, Brenda J Andrews, and Alan M Moses. Integrating images from multiple microscopy screens reveals diverse patterns of change in the subcellular localization of proteins. eLife, 7:c31872, 2018. ISSN 2050-084X.
Divyam Madaan, Taro Makino, Sumit Chopra, and Kyunghyun Cho. Jointly modeling inter-& intra- modality dependencics for multi-modal learning. Advances in Neural Information Processing Systems, 37: 116084-116105, 2024.
Chengzhi Mao, Augustine Cha, Amogh Gupta, Hao Wang, Junfeng Yang, and Carl Vondrick. Generative interventions for causal learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 3947-3956, 2021.
Nikita Moshkov, Michael Bornholdt, Santiago Benoit, Matthew Smith, Claire McQuin, Allen Goodman, Rebecca A Senft, Yu Han, Mehrtash Babadi, Peter Horvath, et al. Learning representations for image-based profiling of perturbations. Nature Communications, 15:1594, 2024.
Dilxat Muhtar, Zhenshi Li, Feng Gu, Xucliang Zhang, and Pengfeng Xiao. Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. ar preprint arXiv:240..00 02544, 2024.
Michal Muszynski, Levente Klein, Ademir Ferreira da Silva, Anjani Prasad Atluri, Carlos Gomes, Daniela Szwarcman, Gurkanwar Singh, Kewen Gu, Maciel Zortea, Naomi Simumba, Paolo Fraccaro, Shraddha Singh, Steve Mcliksetian, Campbell Watson, Daiki Kimura, and Harini Srinivasan. Finc-tuning of goospatial foundation models for aboveground biomass estimation, 2024. URL httpp::/arxxi....... org/ /20/ 19888
Maxime Oquab, Timothóe Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Picrre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2023.
Chao Pang, Xingxing Weng, Jiang Wu, Jiayu Li, Yi Liu, Jiaxing Sun, Weijia Li, Shuai Wang, Litong Feng, Gui-Song Xia, and Conghui He. Vhm: Versatile and honest vision language model for remote sensing image analysis, 2024.
W. M. Pernice et al. Out of distribution generalization via interventional style transfer in single-cell microscopy. 2023.
Chan Pham and Bryan A Plummer. Enhancing feature diversity boosts channel-adaptie vision transformers. Tn Proceedings of the 37th International Conference on Neural Information Processing Systems, 2024.
15

---

Published in Transactions on Machine Learning Research (06/2025)
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAT I blog, 188):9, 2019.
Colorado Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendacle, and Trevor Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 4088 4099, 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Compuling and Compuler-Assisled Intervention MICCAT 2015, pp- 234-241, 2015.
Daniela Szwarcman, Sujit Roy, Paolo Fraccaro, Porsteinn Elí Gislason, Benedikt Blumenstiel, Rinki Ghosal, Pedro Henrique de Olivcira, Joao Lucas de Sousa Almcida, Rocco Sodona, Yanghui Kang, et al. Prithvi- e0-2.0: A versatile multi-temporal foundation model for earth observation applications. arXiv preprinl 2.X.:2412.022922 2024.
Jamie Tolan, Hung-I Yang, Benjamin Nosarzewski, Guillaume Couairon, Huy V Vo, John Brandt, Justine Spore, Sayantan Majumdar, Daniel Haziza, Janaki Vamaraju, et al. Very high resolution canopy height maps from rgb imagery using self-supervised vision transformer and convolutional decoder trained on aerial lidar. Remote Sensing of Environment, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Pierre-Emmant Emmanuel Albert, Amjad Almahairi, Yassine Babaci, Denis Bashlykov, Siddharth Batra, Anurag Bhargava, Shruti Bhosale, et al. Llama: Open and efficient. foundation language models. arXiv preprint (XXi:2302.13971, 2023.
Matheus P Viana, Jianxu Chen, Theo A Knijuenburg, Ritvik Vasan, Calysta Yan, Joy E Arakaki, Matte Bailcy, Ben Berry, Antoine Borensztejn, Eva M Brown, et al. Integrated intracellular organization and its variations in human iPS cells. Nature, 613:345 354, 2023.
Yi Wang, Nassim Ait Ali Braham, Zhitong Xiong, Chenying Liu, Conrad M Albrecht, and Xiao Xiang Zhu. Ssl4eo-s12: A large-scale multi-modal multi-temporal dataset for self-supervised learning in earth observation. arXio preprint Xiw:2211.0704. 2022.
G.P. Way, T. Natoli, A. Adeboye, L. Litichevskiy, A. Yang, X. Lu, J.C. Caicedo, B.A. Cimini, K. Karhohs, D.J. Logan, M.H. Rohban, M. Kost-Alimova, K. Hartland, M. Bornholdt, S.N. Chandrasckaran, M. Haghighi, E. Weisbart, S. Singh, A. Subramanian, and A.E. Carpenter. Morphology and gene expression profiling provide complementary information for mapping cell state. Cell Systems, 13:911-925 2022.
Dejin Xun, Rui Wang, Xingeai Zhang, and Yi Wang. Microsnoop: A generalist tool for microscopy image representation. The Innovation, 5, 2024.
Bryan Zhu, Nicholas Lui, Jeremy Irvin, Jimmy Le, Sahil Tadwalkar, Chenghao Wang, Zutao Ouyang, Frankie Liu, Andrew Y Ng, and Robert B Jackson. Meter-ml: A multi-sensor earth observation benchmark for automated methane source mapping. arXiv preprint 2r1::2207.11166, 2022.
16

---

Published in Transactions on Machine Learning Research (06/2025)
A
Detailed overview
A
B
VIT
Ru
Aza
Nuz
LINEAR INTEDONNG LAYUR
C
KEYS
CLS
CH
J/L
QUERIES
CH
3/2
he.
No.
CH
BACKBONE
Figure 4: Overview of different channel-adaptive strategies. (A) Joint-channelencodding strategy of Channel- ViT (Bao et al. 2024 Bourriez et al.] 2024]: the image is reshaped into a sequence of single-channel patches and channel embeddings are used to retain channel information. (B) Hierarchical Attention model: a specialized attention mask is used to enforce independent- -ndepeneen--haameeeenddiii via channel class tokens 2CH, while the global class token FCls supports inter-channel reasoning. (C) Independent--hhmme--nnood note that common backbone is used. (D) Attention mask of the Hierarchical Attention model.
strategy,
B Datasets
This section provides detailed description of the datasets, and of the channels they encompass.
Human Protein Atlas datasct. The Human Protein Atlas (HPA) is an initiative that aims to map all human proteins across cells, tissues and organs. The subset of the data that is considered is the one of the Kaggle competition Human Protein Atlas (0119), concerned with the subcellular distribution of the proteins encoded by different genes. covers cell lines and 28 subcellular structures of protein localization. It covers thirty five cell lines and more than 13, 000 proteins. The subcellular localization of each protein was classified into more of 28 subcellular structures.
17

---

Published in Transactions on Machine Learning Research (06/2025)
The images were acquired using immunffhorescenece and confocal microscopy. Four fluorescence dyes binding to (0) microtobules, (1) encoded protein, (3) nucleus and (4) endoplasmic reticulum are imaged in different. channels. There are 113, 545 images in total.
There is also a single cell version of the same dataset Le et al. 2022 obtained through segmentation of the FOV images. The HPA Single Cell dataset contains 839, 612 images.
WTC-11 dataset. The dataset contains 214, 037 3D images of cells, spanning cellular structures. Other than tagging the structure of interest with a fluorescent protein (FP), fluorescent DNA and cell-membrane dyes were employed. The images have four-channels: bright-field; mucleus; cell membranc; structure of interest. Given that the focus of this work is to develop foundation channel- model for fluorescent microscopy, the bright-field channel was discarded.
The dataset provides cell-cycle stage annotations. The six possible labels are MO, M1M2, M3, M4M5, M6M7 single, M6M7 _cmpl__o where MO through M7 denote cell cycle stages.
Cell Painting dataset. The Cell Painting Dataset Doron et al. (0223) used is the combination of the LINCS (Way et al. 2022), BBBC036 (Bray et al. 2017) and third curated dataset. (Moshkov et al. 202-]) which includes BBBCO22. All of those datasets were obtained following the Cell Painting protocol Bray et al. (2016), standardized morphological profiling assay that images six fluorescent dyes in five channels, revealing eight cellular components. The components visualized in each channel are (0) nucleus; (1) endoplasmic reticulum; (2) nuclcoli, cytoplasmic reticulum; (3) actin, golgi, plasma membrane; and (4) mitochondria. The goal of the studies included in the Cell Painting Dataset was to quantify the response of the cells to different perturbations: either compound treatments or gene over-expression experiments. Overall, the dataset includes 400 compounds and 80 gene over-expression experiments, totaling 8, 423, 455 images.
CHAMMI dataset. The CHAMMI dataset was curated from the WTC-11, HPA Single Cell and Cell Painting datasets. It includes 65, 103 images from the WTC-11 dataset covering six tagged structures; 66, 936 images from the HPA Single-Cell dataset covering 18 cell lines and 8 protein localization classes, only images with single protein localization annotation were selected; and 88, 245 images from the Cell Painting dataset covering seven compound experiments, including the negative control. In total there are 220, 284 images, of which 100, 145 are used for training.
JUMP-CP datasct. We considered the version of the JUMP-CP dataset used by Bao et al. (2024] it is processed version of the data made available by the JUMP-Cell Painting Consortium Broad Institute/ (2021). h. Each image includes the five Cell Painting channels and three brightfield channels (HighZBF, LowZBF and brightfield).
The datasets generated by the JUMP-Cell Painting Consortium have the goal of enabling image-based drug mechanisms of action determination. As such, it encompasses multiple chemical and genetic perturbations, This particular version of the JUMP-CP dataset contains 229, 228 single cell images.
OpenCell dataset. The OpenCell dataset was introduced by Kobayashi et al. (2022) and consists of confocal images encompassing 1,111 different tagged protcins. On average, each protein was imaged in 18.59 field of view images. Crops containing from to 3 complete cells were extracted from each image, resulting in approximately 800 cropped images per protein. In total, 1, 134, 592 images were made available. In addition the tagged protein, a nuclear marker was used visualize the nucleus. From the nuclcar channel, they constructed a distance map and segmentation masks. However those two additional channels were not used for the purposes of this work.
18

---

Published in Transactions on Machine Learning Research (06/2025)
C Details on the CHAMMI benchmark and dataset extension
C.1 CHAMMI benchmark
The CHAMMT benchmark Chen et al. (2024] is a standardized evaluation framework for channel-invariant models. It presents comprehensive set of nine tasks for channcl-invariant models of varying complexity, that evaluates the ability of the models to generalize to new biologically-releant experimental regimes. The images from each data source present in the CHAMMI dataset are split into one training set and several test sets, designed for specific tasks. Tasks with suffix 1 are IID classification problems, where the test and train data follow the same distribution. Originally, the CHAMMI benchmark considers a Nearest Neighbor (NN) evaluation.
The WTC-11 data used for cell-cycle stage classification. The train set contains images with one of four cellular structures taggod: muclear speckles, mitochondria, microtubules, or Golgi apparatus. Images of W Task2 are tagged with three novel cellular structures, the task evaluates whether the model is able to classify cell-cycle stages when an unseen cellular structure is tagged.
The HPA data supports protein localization classification. The train split covers 17 cell lines and four protein localizations: muclear speckles, mitochondria, microtubules, or Golgi apparatus. The H Task2 images come from a novel cell line but covering the same protein localizations as the train split. The H Task3 images come from the same cell lincs as the train split, but labeled with one of three novel protein localizations.
Lastly, the Cell Painting data is used for perturbation classification. The train set includes images of cells coming from 9 plates and perturbed with one of three treatments, as well as negative controls; The C' Task2 images are perturbed with the same treatments the train split and coming from the same data sources, however they belong to a set of 3 novel plates. The Tosk3 includes the same treatments as the train split, but coming from the BBBC022 dataset and covering 4 novel plates. Finally, the C_ Task4 images are from the same set of plates and data sources as the train split, but the cells are perturbed with novel treatments.
For tasks that introduce new labels that are not present in the train set (HPA Task and CP Task 4), a leave-one-out evaluation strategy is employed. Taking the example of HPA Task 3, the test data is split into sub-groups according to the cell line. Then, for each nub-group, the NN search is computed on both the training data and the remaining sub-groups. For CP Task 4, the data is split by plate ID.
C.2 ExtendedCHAMMI dataset
The ExtendedCHAMMI dataset extends the CHAMMI train split to a total of 7, 748, 662 images, using additional data from both the source datasets and new data sources, while preserving the OOD characteristics of the CHAMMI tasks.
In order to build the EstendedCHAMMI dataset, the HPA FOV, HPA Single Cell, WTC-11, Cell Painting and OpenCell datasets were used. The samples belonging to the IID tasks W Task1, H Task1 and C. C_Task1 were removed from the WTC-11, HPA Single Cell and Cell Painting datasets, respectively. Furthermore, the images of the HPA FOV dataset containing cells present on H Task1 were removed as well. With respect to the OOD tasks, the unseen tagged cellular structures for WTC-11; cell lines and protein localizations for HPA FOV and HPA Single Cell; and plates, data sources and treatments for Cell Painting were removed. The resulting number of images per dataset is summarized in Table g
Hierarchical attention model training objective
The hicrarchical attention model is based on singlo-channel patch approach and, in addition to a global CLS token, it also inserts into the sequence channel CLS tokens. Moreover, it leverages a hierarchical allention mask (Figure } that constrains a channel's patch and CLS tokens to only interact with one another, while the global CLS token attends to the channel CLS tokens. Consequently, the model produces both a global image representation, and channel-level representations. In view of this, additional loss terms were included in the DINOv2 training objective, to account for the channel-level representations.
19

---

Published in Transactions on Machine Learning Research (06/2025)
Table Data included on the ExtendedCHAAMI dataset. The third column lists the amount of data that was included on ExtendedCHAAMM over the total size of the data source. For each data source, the number of channels and image type is specified: field-of-view (FOV) images, cropped FOV images containing smaller number of cells, images of single cell. The last column lists which biological or experimental factors were discarded from the original datasets to preserve the OOD characteristic of the CHAMMI tasks.
Dataset
Image type
Size
Channels
Discarded factors
IPA Single Cell
One cell
296670/839612
Cell line (IEK 293): protein localization (cytosol, endoplasmic reticulum, nuclooplasm)
WTC
One coll
179994/214037
Taggod collular components (centrioles, tight june- tions, actin bundles)
OpenCell
Cropped FOV
113/592/1134552
None
Cell Painting
Cropped FOV
6103565 F103565/8423455
5 Plates (SQ000151255, SQ00015168, SQQ0015221); data source (BBBCU22); treatments (BRD- K11129031, BRDD-62310379, BRD-K77947974)
HPA FOV
FOV
33841/102190
4 Cell line (HEK 293); protein localization (cytosol, endoplasmic reticulum, nucleoplasm)
Consider an image z, and let ((x) denote the set global crops æ, and C(x) the set of all crops of æ, note that (xx) C C(x). Furthermore, let (CLSI (u) and pt |CLS| (u) be the CLS tokens, transformed into probability p
vectors, output by the student and teacher networks for crop u. Then, the DINO loss for a sample is:
Σ 11 mEG(w) vEC(x) esfuu
CCLS]
(w), (v))
Pt
where H-)) denotes the cross-entropy. With the additional channel CLS tokens, loss is extended to:
|CHc], the DINO
Σ Σ
(CLS))
(CLS) H (p! (CTS)(u). (u). (OLS) (v))
[CH] Aino
Σ
H
Pr [CH)
(u), (u)) (004)())))
uES(x) vEC(1)
Another component of the DINOv2 loss is the KoLco regularizer, that encourages the uniform span of features within batch. Other than applying the KoLeo loss to the set of global CLS tokens for batch, the loss is also scparately applicd to the set of all channel CLS tokens in the batch, with weights (CLS) and [CH The remaining component the DINOv2 loss, the iBOT masked-inage- mnteeeiiinn-mmdllinggg loss, is left unchanged. We gave an equal weight the losses on the global and channel CLS tokens. For the pre-training on the small scale CHAMMT dataset only, the DINO and KoLeo losses channel class tokens were discarded, due to instabilities during training.
E
Training and Evaluation details
E.1 Pre-training details
Unless specified otherwise, we trained ViT-Large models with default patch size 16, and the default parameters of DINOv2 except a drop path rate of 0.1, a teacher momentum of 0,996, a learning rate of 5.0 10 and 20 warm-up epochs. Transforms used include random contrast and brightness augmentations, flips and random resize crops of size 224.
E.2 Evaluation details
The feature vector used in the evaluations with the exception of the evaluations on the CHAMMI benchmark obtained by concatenating the CL.S tokens across the last L. layers (T. 4), as well the channel-wise
20

---

Published in Transactions on Machine Learning Research (06/2025)
average pooled patch tokens. If D is the dimension of the tokens (for VIT-L D 1024), and K is the number of channels, then the feature size for Channel-ViT is LD + KD, for DINO BoC it is L.D + KD and for DINO HA it is L(1 + K)D + KD.
For the evaluations on the CHAMMI benchmark (Tables and 6), only the CLS tokens are used and I. 1, therefore the feature size for Channel-ViT is D, for DINO BoC it KD and for DINO HA is
(1 11+K3 D. We ablate the impact of feature dimension in Appendix K
We used the AdamW optimizer and one cycle Cosine scheduler. We used the same train/val/teen splits Bao et al. (2024) for the JUMP-CP dataset. For HPA, we used the same train/val val splits as Doron et al. (2023) For WTC, we created 80% - 10% 10% uniformly distributed train/val //a//vll//teee /tettt splits. For every evaluation, we trained 14 classifiers varying the learning rate bet 10-4 and 1, and selected the best classifier on the val set. We trained all classifiers for 4350 iterations on GPUs with a batch size per gpu of 32 for HPA-FOV and WTC, and 128 for JUMP-CP. To train the lincar classifiers on HPA-FOV, the following transforms are used random crop of size 384, flips and self normalization. For evaluation, center crop of size 384 x 384 is laken, followed by self normalization. For JUMP- -PP, used the same normalization as in Bao et al. (2024) instead of self normalization, and crops of size 224. For WTC, used self normalization and crops of size 224.
F Influence of Hierarchical Channel Sampling
Bao et al. 2024) introduced a channel sampling technique for the training of ChannelViT, denoted hierarchical channel sampling (HCS). For an image z with K channels, HCS consists in sampling number m E (1,..., , K) uniformly at random and then randomly selecting m channels from x without replacement.
Results summarized in Table 10 suggest that Hierarchical Channel Sampling (Bao et al., 2024 hinders the performance of single-channel patch models. The channel sampling technique was found by Bao et al. (2024) to boost performance when pre-training and evaluating on the same dataset, mainly in the supervised scenario of missing channels at evaluation time. However it does not translate into improvements on the channel heterogeneous setting explored in this work. We postulate that plays the role of a regularizer in the supervisod context, but that this strategy is not well adapted SSL.
Table 10: Influence of HSC on DINO Channel- VIT models. The models were pre-trained on the ExtendedCHAMMM dataset and evaluated on CHAMMI.
Average OOD
HPA Task2
CP
Model
HCS
Mean
WTC
HPA
CP
Task1
Task2
Task1
Task3
Task1
Task2
Task3
Task1
DINO Channel- VIT
x
43.6
46.2
55.6
28.9
64.5
46.2
92.1
65.3
45.9
89.0
53.5
21.8
11.3
DINO Channel-Ve VFT
39.9
39.5
51.9
28.4
66.4
39.5
88.5
62.3
41.5
90.2
56.0
22.3
6.9
G Hierarchy of factors of variation in the feature space
SSL methods learn image representations using the samples themselves as the supervisory signal, therefore no label information controls the organization of the features in the embedding space.
In the channel-invariam models pre-trained on HPA FOV, we observe an emerging clustering of the features according to hicrarchy of semantic concepts, illustrated in Figure 5 The features first cluster by cell type, while the protein localization is retained nested factor of variation.
Channels as confounders in unified feature space
To assess the utility of a unified feature space produced by channel- invariinn models such as the one proposed by Bourriez et al. (2024). we explore the effect of ablating channels in the HPA FOV datasct. Specifically, we remove nucleus and ER channels from random half of the dataset and compare the resulting features against those of images with all channels within jointly computed UMAP space (Figure 6
21

---

Published in Transactions on Machine Learning Research (06/2025)
13
13
12
-20
(a)
(b)
Figure 5: UMAP of the HPA FOV dataset highlighting different factors of variation. UMAP space of the HPA FOV features obtained from the DINO BoC model pre-trained on the same dataset, colored according to (a) cell lype and (b) protein localization. In (b) only samples with a single protein localization are displayed. The features are obtained separately for each channel, and then concatenated.
We observe that the data is clustered into distinct groups depending on the channels, and samples with different channels can hardly be compared to one another, even if their features have the same dimension. Therefore common embedding space does not constitute advantage of Channel-Vis over DINO BoC.
Impact of the removal of different pre-training datasets
Table 11: Ablation removing one dataset from the ExtendedCHAMMI dataset. We report the linear evaluation results for DINO BoC.
Training set
HPA-FOV F1
HPA-FOV F1
Accuracy on JUMP-CP
WTC F1 Cell cycle st.
Protein loc.
Cell type
ExtendedCHAMMM
61.7
91.1
45.2
90.5
minus WTC
60.4 1.3
91.1
43.7
90.9 0.4
minus Cell painting
60.2
91.1
44.1
89.8
minus HPA (FOV, single cell)
41.7 .20.0
89.9
46.9 +17
92.3
minus CelnCell
60.9
91.0
44.2
90.0
To study the influence of specific pre-training datasets on the performance on others, we remove some pre-training datasets in Table [I and evaluate the performance on HPA-FOV, JUMP-CP and WTC. As expected, when removing both HPA datascts, the protein localization performance is severely altered, but the cell type classification, a much casier task remains accurate. Not much difference observed on the HPA tasks when removing the other datasets. In general, removing any dataset hurts the overall performance.
22

---

Published in Transactions on Machine Learning Research (06/2025)
Al channels
(0.1)
13
13
-20
(a)
(b)
Figure 6: UMAP of the HPA FOV dataset to assess utility of unified feature space. a) UMAP space of the HPA FOV features obtained from the Channel-ViT model pre-trained on the same dataset, comparing leatures computed when the model sees all four channels of the dataset, blue, and features computed with only channels (microtobules and protein), in orange. b) Same UMAP in (a) but filtered for images with only one protein localization and colored according to them.
J Results on the eight channels of JUMP-CP
As shown in Table 12 both DINO HA and DINO BoC are flexible, resulting in improved performance when the downstream tasks involves a larger number of channels than at pre-training time. Here, the pre-training dataset contains up to 5 channels, while the models are evaluated with up to
Note that the Channel ViT approach cannot be evaluated on images that contain more channels than the maximun number of channels per image of the pre-training data, since there are no trained channel embeddings for the extra channels.
Table 12: Mean accuracy on the JUMP-CP dataset with models pre-trained on Extended- CHAMMI with a maximum of five channels.
JUMP-CP 5 channels
JUMP-CP 8 channels
Channel ViT
39.5
45.2 DINO BoC
DINO HA
51.4
45.2
51.6
K Ablation on feature dimension
In Appendix E.2 we describe how the features are obtained for each of the channel-invarianl methods. In particular, joint and independent channel encoding strategies naturally lead different sized embeddings, since the former results in image level representation, while the later results in channel level representations.
23

---

Published in Transactions on Machine Learning Research (06/2025)
Let D denote the backbone's token dimension, K the number of channels, and L the number of last layers the CLS tokens are taken from. Then, when using both CLS and channel-wise average pooled patch tokens, the feature size for Bhaanee-ViiT is LD + DD, for DINO BoC it is LKD + KD and for DINO HA it is L(1 KYD + K.D When only CLS tokens are used, the feature size for Channel-VV ViT is LD, for DINO BoC it is LKD and for DINO HA it is L(1 + K.
In order demonstrate that DINO BoC outperforms DINO Channel-Vin ViT due to the quality of the features and not due to its dimension, consider two setups where both models have the same feature size. Those setups are:
For Channel-ViT we use only the CLS token from the last layer; while for DINO BoC we average pool the CLS tokens for each channel. Thus for both models the features are D-dimensional.
2. For Channel-ViT we concatenate the CLS token from the last layer to the channel-wise average pooled patch tokens, thus the feature is D + K D-dimensionnal. On the other hand, for DINO BoC we concatenate only the CLS tokens for each channel, resulting in K D-dimensional feature.
The results obtained for those setups on the CHAMMI benchmark are listed in Table 13 In both cases DINO BoC outperforms DINO Channel-ViT.
Table 13: F1 scores for a lincar probe on CHAMMI test sct: Ablation with similar embedding size.
Feature
Average OOD
WTC
HPA Tusk2
CP
Model dimensson
Mean
WTC
HPA
CP
Task1
Tusk2
Tusk1
Task
Tuske
Task2
Task3
Task4
DINO Chameel-VTT
D
59.8
66.9
76.7
35.9
83.1
66.9
88.2
84.9
68.4
80.5
54.5
23.3
30.0
DINO BoC
D
65.4
86.7
67.9
41.5
89.4
86.7
82.9
79.1
36.7
83.8
61.2
26.6
36.8
DINO Channel-VPs
KD
37.4
86.4
DINO BoC
KD
67.9
89.2
74.9
39.7
90.5
89.2
83.3
84.7
65.0
90.5
60.5
25.8
32.7
L Ablation on the training number of iterations
To compensate for lower number of images seen during training, trained BoC modcls for K times more iterations Ihan other models. We report in Table 14 the performance of the models when trained using the same number of iterations.
Table 14: AUROC on the Meter-ML dataset in function of number of pretraining iterations.
Using NAIP images only.
Number of pretraining iterations
AUROC
Channel ViT
34K
69.3
DINO HA
34K
77.3
DINO BoC
34K
78.7
DINO BoC
4 x 34K
81.9
24


Reviews:
[{"review_id": "bvZvd73LBo", "reviewer": "Reviewer_i3CN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed Bag-of-Channels (BoC) approach is conceptually simple and performant.\n- Extensive experiments are conducted on a wide variety of downstream tasks and two relevant domains (microscopy and geospatial imaging).\n- Having the hierarchical attention model as I kind of “bridge” between the two extreme approaches is a nice touch. \n\nWeaknesses:\n- Lack of fine-tuning results: in many practical situations practitioners have a reasonable amount of data to conduct fine tuning on a foundation model for a specific task. Thus it would be interesting to determine whether there are advantages or disadvantages to the BoC approach to pretraining when fine tuning. Furthermore, it is well known that MAE style pretraining requires fine tuning to be competitive with contrastive pre training (see He et al. 2023 Masked Autoencoders are Scalable Vision Learners), so it would be more fair to compare DINO BoC to MAEs in this setting.\n- I think there is an opportunity to connect to the broader literature on multi-modal learning that has been missed. For example Liang et al. 2024 “Factorized Contrastive Learning: Going Beyond Multi-view Redundancy” explores the importance of considering information that is unique to each modality (channel in this setting) along side the shared information. Could this framework be used to understand the success of BoC (i.e. channel mixing approaches squeeze out task relevant information that is not shared across channels/modalities)?\n- The description of the BoC approach is somewhat difficult to understand given the presentation. My understanding is that it involves training one backbone on only single channel inputs (from each modality), and then at test time ensembling the outputs for each channel. I think stating this plainly in its own section (i.e. between 3.2 and 3.2.1) could improve the readability of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the importance of a particular design choice (how channels are ‘fused’) when training foundation models to be applied in regimes where observations are image-like but may have diverse or missing channels across downstream datasets. The author’s find that a simple baseline: ensembling/concatenating the outputs of a model trained independently on individual channels/measurement types to be surprisingly effective, outperforming various other strategies for handling variable channel input on a variety of evaluations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- If at all possible, including some fine tuning experiments would improve the work in my eyes. If not possible I still think including some discussion of how results may change in this setting is warranted.\n- I think some discussion of the relationship between the methods compared here and the broader multimodal learning setting al la Liang et al. 2024 would be good to have.\n- Please include a more explicit description of the DINO BoC method (potentially as its own section).\nDINO HA is missing from some (most?) results tables, I think including it for all evaluations would make the messaging even clearer.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tqI8Ux3toc", "reviewer": "Reviewer_jgAJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Provides experimental results on a wide range of datasets in multiple domains.\n- Detailed ablation studies and supplementary experiments.\n- Source code will be open sourced.\n- Includes brief details about unsuccessful routes of investigation.\n\n### Weaknesses\n- The methodology is unclear and it is difficult to differentiate between existing methods and the proposed methods. It appears the explanation of the claimed SOTA methodology, DINO BoC, is missing or is not clearly identified.\n- Many results for DINO HA are close to and occasionally superior to DINO BoC. This weakens the evidence for the claim that the independent encoding of channels outperforms joint encoding.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- A large scale comparative study of channel-invariant self-supervised learning methods.\n- Shows that independent-encoding outperforms joint-encoding on in-domain, cross-dataset, and out-of-distribution experiments.\n- Introduces a novel channel-invariant self-supervised learning method.\n- Open sources said new SOTA general purpose feature extractor.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Related Work (Would strengthen)\n- The paper mentions MAE, but there have since been derivatives better suited to training encoders on multiple channels/modalities. Addressing one such as MultiMAE https://arxiv.org/pdf/2204.01678 could strengthen the claim.\n\n### Methodology (Critical)\n- A textual explanation of DINO BoC needs to be included or be better signposted. \n- The delimitation between the contributions of the paper and existing methods needs to be clearer.\n- Describe only methods in Section 3 that are used for experimentation as it is unclear if which and if all methods are used. A brief overview of existing methods may be better suited in the related work section.\n\n### Results (Would strengthen)\n- The results for DINO HA are not far behind and sometimes superior to DINO BoC which weakens the claim that independent channels encoding is superior. Including results for DINO HA where possible in Tables 2, 4, 5, 7, and 8 would strengthen this claim.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dBwm3zr1yq", "reviewer": "Reviewer_bUfp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- 7 datasets with fluorescent microscopy imaging.\n\nWeaknesses: \n- DINO BoC largely resembles a late fusion approach commonly used in multimodal learning, where each channel or modality is independently encoded before aggregating representations. From a methodological standpoint, the idea appears incremental. \n- The overall performance boost appears to be inconsistent. It works better in some cases, while in others, it works worse. \n- Only one dataset for geospatial imaging. It would be interesting to see whether claims shown with fluorescent microscopy imaging will hold.\n- Tables for multiple experiments do not report results as mean$pm$std over multiple runs. Hence, the variability of the model's performance is not evident.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a channel-invariant self-supervised learning method, DINO BoC, for non-RGB scientific imaging. DINO BoC is a Bag of Channel (BoC) variant of DINOv2. The manuscript demonstrates that independently encoding channels consistently surpass joint-channel encoding strategies in extensive benchmarking experiments, including out-of-distribution and unseen channel combinations.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Misuse of invariance terminology. Strictly speaking, invariance in representation learning implies that a model's representation remains unchanged under certain transformations or permutations. However, the authors use the term \"channel-invariant\" to describe models that can handle different numbers and types of input channels across datasets without retraining. Additionally, channels are not interchangeable, given that each imaging channel typically captures unique and complementary information, such as distinct cellular structures in microscopy or specific spectral bands in geospatial imagery.\n- In multimodal learning, a common conceptual framework is to distinguish between inter-modality dependencies (relationships across different modalities) and intra-modality dependencies (relationships within a single modality). Recent frameworks, such as Inter- & Intra-Modality Modeling (Madaan et al., NeurIPS 2024), explicitly address these dependencies. The current submission explores analogous concepts but within the scope of multi-channel single-modality imaging data. Specifically, the DINO BoC method emphasizes intra-channel dependencies by independently encoding each channel without modeling explicit inter-channel interactions. Given this conceptual overlap, such multimodal frameworks could provide additional theoretical context and improve performance (as I noted in weaknesses about inconsistent performance on some tasks).\n\nMadaan, Divyam, et al. \"Jointly Modeling Inter-& Intra-Modality Dependencies for Multi-modal Learning.\" Advances in Neural Information Processing Systems 37 (2024): 116084-116105.", "broader_impact_concerns": "None noted.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ne14ij5J0a", "reviewer": "Reviewer_R5tD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: same as the contributions I mentioned above.\n\nWeaknesses:\n1. I have a question regarding the main claim in the paper: \"our results call the utility of joint-channel-encoding as a prior for SSL pretraining into question.\" :\nDifferent channels contain some information (tagged with the protein/stain etc.). My question is how the author ensures that the information within these channels is mutually exclusive (or fully independent) or not? If the channels carry information that does not have any added value for another channel, it is likely that joint-channel-encoding does not have any advantage over channel-invariant methods. So, I was pretty much looking forward to seeing an experiment that tries to answer this, but I haven't found anything. \nTherefore, I suggest the authors design a controlled experiment where they pick specific channels (those with specific proteins tagged or sth similar) that are known to interact with each other and investigate their hypothesis again. It's likely that joint-channel-encoding methods perform well for those proteins, and underperform other methods in proteins with low interactions.\n\n2. From the manuscript: \"HA for testing the hypothesis that limiting inter-channel interactions enhances the performance of channel-invariant models\". The best way to test this is using statistical hypothesis testing. Looking at both Tables 3 and 6, the results are mixed (compared to BoC); it seems inconclusive with regard to this hypothesis. One way to do it is to create different variations of the model (30 random seeds) and test on the datasets, and see if the authors can reliably test this with H-0 and H-1. Otherwise, it is better to adjust the claims in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents an extensive empirical study on self-supervised learning strategies for imaging domains with heterogeneous channel compositions. It introduces the DINO Bag of Channels (DINO BoC) approach, which independently encodes each channel using a shared Vision Transformer backbone and then aggregates the resulting embeddings, challenging the conventional idea that joint-channel encoding is always superior. \n\nAdditionally, the authors propose a novel Hierarchical Attention mechanism that limits inter-channel interactions to a high-level global token, exploring the balance between independent and joint encoding strategies. When scaled with the state-of-the-art DINOv2 framework, the DINO BoC method not only outperforms joint-channel encoding approaches across diverse benchmarks but also shows robust generalization to traditional RGB datasets like ImageNet 1k. Overall, it is a well-written manuscript, and I found their idea quite interesting with some concerns regarding their methodology.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. A separate figure for BOC and HA, and emphasize that what is the role of each component in the study. The current figure is somewhat hard to understand and differentiate between the work proposed and the role of each component.\n\n2. Ablation study on the iterations of training since it is different from baseline. It can be one with the same iterations as the baseline and one with the current iterations to just show how the training time is impacting the performance of BoC. \n\n3. authors' justification or empirical evidence with regard to w1 and w2 mentioned above.", "broader_impact_concerns": "A limitation statement about the aggregation method used (only hierarchical attention, while there are many,) which can impact the result.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a straightforward but empirically effective approach, called DINO BoC, for self-supervised learning in the context of scientific imaging with heterogeneous channels. It provides an extensive empirical evaluation, open-source resources, and suggests a shift in how practitioners might approach channel variation during pre-training.

However, the reviewers consistently pointed out the some weaknesses, which have claimed to be solved by the authors and were accepted by the reviewers.  These weaknesses are:

- Insufficient methodological clarity: The description of the DINO BoC architecture and its operational details should be isolated in a dedicated section for better comprehension. Claimed to be addressed.
- Incomplete evaluation: Fine-tuning results are absent, and the comparison with more recent multimodal or masked modeling methods (e.g., MultiMAE, Inter-/Intra-modality frameworks) is underdeveloped. Claimed to be addressed.
- Overly strong claims: Statements such as "independent encoding is better at scale" could be tempered given that DINO HA shows comparable performance in many settings and that variance across datasets is not quantified.
- Misuse of terminology: Terms like "invariant" may mislead, since the method handles channel heterogeneity, not invariance in a strict sense. Claimed to be addressed.
Decision: Accept (poster)