ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning
Authors: Chau Pham
Juan C. Caicedo
Bryan A. Plummer
Venue: NeurIPS 2025 poster ()
Keywords: masked autoencoder
vision transformer
multi-channel imaging
representation learning
Primary area: deep_learning
TL;DR: We introduce ChA-MAEViT - a robust, high-performing model that encourages cross-channel interactions for multi-channel imaging.

Abstract: Prior work using Masked Autoencoders (MAEs) typically relies on random patch masking based on the assumption that images have significant redundancies across different channels, allowing for the reconstruction of masked content using cross-channel correlations. However, this assumption does not hold in Multi-Channel Imaging (MCI), where channels may provide complementary information with minimal feature overlap. Thus, these MAEs primarily learn local structures within individual channels from patch reconstruction, failing to fully leverage cross-channel interactions and limiting their MCI effectiveness. In this paper, we present ChA-MAEViT, an MAE-based method that enhances feature learning across MCI channels via four key strategies: (1) dynamic channel-patch masking, which compels the model to reconstruct missing channels in addition to masked patches, thereby enhancing cross-channel dependencies and improving robustness to varying channel configurations; (2) memory tokens, which serve as long-term memory aids to promote information sharing across channels, addressing the challenges of reconstructing structurally diverse channels; (3) hybrid token fusion module, which merges fine-grained patch tokens with a global class token to capture richer representations; and (4) Channel-Aware Decoder, a lightweight decoder utilizes channel tokens to effectively reconstruct image patches. Experiments on satellite and microscopy datasets, CHAMMI, JUMP-CP, and So2Sat, show that ChA-MAEViT significantly outperforms state-of-the-art MCI-ViTs by 3.0-21.5%, highlighting the importance of cross-channel interactions in MCI. Our code is publicly available at https://github.com/chaudatascience/cha_mae_vit.

Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning
ChA-MAEViT:
Chau Pham Boston University chauphan@bu.ed edu
Juan C. Caicedo Morgridge Institute, UW-Madison juan. caiceddwissc.. edu
Bryan A. Plummer Boston University bplum@bu. edu
Abstract
Prior work using Masked Autoencoders (MAEs) typically relies random patch masking based on the assumption that images have significant redundancies across different channels, allowing for the reconstruction of masked content using cross- channel correlations. However, this assumption does not hold in Multi-Channel Imaging (MCI), where channels may provide complementary information with minimal feature overlap. Thus, these MAEs primarily learn local structures within individual channels from patch reconstruction, failing to fully leverage cross- channel interactions and limiting their MCI effectiveness. In this paper, we present ChA-MAEViTI an MAE-basee method that enhances feature learning across MCI channels via four key strategies: (1) dynamic channel-pate masking, which com- pels the model to reconstruct missing channels in addition to masked patches, thereby enhancing cross cross-channel dependencies and improving robustness to vary- ing channel configurations: (2) memory tokens, which serve as long-term memory aids to promote information sharing across channels, addressing the challenges of reconstructing structurally diverse channels; (3) hybrid token fusion module, which merges fine-grained patch tokens with global class token to capture richer repre- sentations; and (4) Channel Aware Decoder, lightweight decoder utilizes channel tokens to effectively reconstruct image patches. Experiments satellite and mi- croscopy datasets, CHAMMI, JUMP-CP, and So2Sat, show that ChA-MAEViT significantly outperforms kate-of-the-ant MCI-ViTs by 3.0-21.5%, highlighting the importance cross-channel interactions in MCI. Our code is publicly available at https://githin com/ http//ataaaiiiinnneaaaaa cha_mae._... _aa_mae_vii vit
Introduction
Visual encoders generally process fixed-channel inputs, such as RGB, during both training and testing phases (2/9. However, in satellite imaging [10), robotic sensing [1I,, cell microscopy [12 13] and medical imaging [14, the input data can vary both the number and type of channels. These varying channel configurations arise from differences in sensor modalities, acquisition settings, or experimental conditions. Multi-Channel Imaging (MCI) models are designed learn feature representations from heterogeneous channels whose type and number vary during both training This adaptability allows single model to effectively support diverse channel configurations, thereby reducing computational costs and minimizing the risk of overfitting [12].
and testing 2 13]
Prior MCI-Masked Autoencoder (MAE) models (e.g.. [5] demonstrated promise in capturing local spatial structures by learning to reconstruct randomly masked patches in multi-channel images. These approaches assume that images exhibit significant redundancy across channels, enabling the reconstruction of masked patches using unmasked patches from similar channels. While this
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(A)
Patch Masking (Prior work)
(B) Dynamic Channel-Patch Masking (ours)
Masked
Image
Unmasked Patches
Patch festures
Chunnel Attention
Masked
Imsgee
Unmaaked Parches
Patch features
Patch to Channel Attention
la
10
Charnel
Channel
Encoder
Encoder
Cross
Within
channel Interaction
channel
14
inferaction
Channel & [CLS]
5 [CLS]
Maskeul
Channe
Menmory
Tolien
Memory
Token
Channel & [CLS]
5 [CLS]
Figure I:
Image patch interactions in MCL. (A) Prior Work on MCI-MAEs (e.g., CA-MAE (T]
employ random patch masking to train the model to reconstruct the masked patches. In the attention map, where each row represents the average attention score of all patches in a channel towards other channels and the [CLS] token (last column), we show each patch primarily attends to its own channel (the diagonal) and the [CLS] token. This suggests that patch masking may not effectively promote cross- cooss-channel interactions in MCI. (B) In contrast, Dynamic Channel-Pate Patch Masking (ours) encourages more interactions between patches across different channels by using both channel and patch masking. Also, memory tokens serve as long-term memory to help information aggregation across channels. Its attention pattern demonstrates a more uniform distribution across channels, indicating that each image patch can learn more meaningful interactions.
assumption works well for natural images, where color channels typically show strong correlations, presents challenges in MCI scenarios. In these situations, complementary sensors (e.g., multispectral and LiDAR) may capture distinct physical properties with minimal feature overlap. As shown in Fig. [a), when using patch masking, the patch-to-channel attention concentrates on its own channel (the diagonal) and the [CLS] token. This indicates that prior MAE methods mainly focus on visible patches within the same channel, neglecting cross-channel feature interactions, restricting the model's ability to learn useful features that require information from multiple channels.
To address this challenge, we introduce Channel-Aware MAE (ChA-MAEViTY which enhances cross-channel learning in MCI through four key improvements summarized in Fig. 2 First, we propose Dynamic Channel- (DCP) Masking, which adap- tively masks both patches and channels during training, compelling the model to reconstruct these missing patches and channels using the remaining patches (Fig. 2 left). DCP Masking adjusts the channel masking ratio to enhance feature learning and robustness to missing channels during inference. Fig. (Tbb) shows our approach evenly redistributes patch attention scores across channels, highlighting improved cross-channel interactions. However, in MCI, reconstructing masked channels challenging since each channel may encode unique features that are not easily inferred from others.
multichannel Vision Transformer
To enable the model to retain global information across all channels during reconstruction, we introduce the use of memory tokens in MCI (Fig. 2 middle). Inspired by register tokens [16), which are extra tokens added in the input to reduce artifacts in the feature maps of ViTs, these learnable embeddings serve long-term memory to retain key cross-channel information in both the encoder and decoder. In addition, we use a hybrid token fusion module in the encoder to combine fine-grained patch tokens with the global class token for a richer representation (Fig. middle).
Finally, we employ Channel-Aware Decoder that simultaneously processes patch tokens from all channels. Existing methods often rely separate decoders for each channel [I] 18), which do not scale well with many input channels. Our shared lightweight decoder utilizes channel tokens to provide channel-specific behavior and memory tokens improve cross-channel feature reconstruction, enhancing performance while reducing computational costs.
Most of our experiments use both self-supervised (SSL.) and supervised learning to boost, performance, showing a similar complementary benefit from combining them (e.g.. 221). However, Appendix also evaluates SSL by itself, where we show approach still outperforms prior work.
The most relevant work to ours focuses on enhancing cross-channel interactions 23 26]. However, these approaches mainly use single-modality images, such as RGB, where channel similarities facilitate strong correlations. As noted earlier, this often does not generalize to the more complex MCI domain where channels convey varying information, e.g., a robot sensor can contain LiDAR. RGB, and thermal camera. Thus, MCI requires balance between preserving unique information in each channel and effectively modeling the complex relationships between channels. MCI models must also be robust to missing channels to support varying channel configurations [12 [3]. This means that cross- chasncl interaction should not only improve the overall representation but also

---

Multi-choneel
Dynamic Channel- Patch Masking (Sec. 3.1)
Combine with chousel tolaus
Add sunok tokens
image
(Sec. 3.3)
Sharod Patch Projection
Channel
Enowder
-Aware Decoder
(Sec, 3.4)
Hybrid
Token Fusion
Memory
(Sec. 3.2)
Task
Reconstraction
[Memorry
Unmaskadf Masked Iranee Patch
CLS Chansell tokenss Class
Memoor
Uiomaskad patchi Mask token
Memoor
Mirmary tnkens
Memory
Figure 2: Our ChA-MAEVIT approach enhances cross-channel learning via four key components: 1) Dynamic Shannel-Patche Masking, which compels the model to reconstruct varying proportions of missing channels and patches, thus improving interactions across channels and robustness to the absence of some channels (Section 3.I. 2) Memory Tokens, which act as long-term memory to facilitate information sharing between channels (Section 3.2). 3) To reconstruct the masked patches and channels, we use a Channel-Aware Decoder that leverages channel tokens for image reconstruction, enhancing performance while minimizing computational costs (Section 3.3 4) A
Hybrid Token Fusion module, which combines fine-grained patch tokens with global (CLS) token
to improve feature representation (Section 3.4
involve learning useful redundancy across channels. ChA-MAEViT addresses these challenges by introducing DCP Masking to enhance cross-channel interaction while allowing the model to learn important channel-specifie features, resulting in more robust and tigher-performing model.
Our key contributions are follows:
We explore ChA-MAEViT, which integrates MAEs into MCI-ViTs for improved robustmess that reports a gain of 3.0 21.5% across three diverse MCI datasets, CHAMMI [12, JUMP-CP (27].
So2Sat [10], and consistent gains Cloud-38 [28].
We propose Dynamic Channel-Patch Masking, which adaptively masks both channels and
patches during training, requiring the model to reconstruct them to boost cross-channel interaction.
We introduce a Channel- Decoder that uses channel tokens to reconstruct image patches
more efficiently, thereby enhancing performance for downstream tasks.
We use memory tokens - learnable vectors to retain cross-channel information and aggregate channel-specific features, and hybrid token fusion module merge the [CLS] token with fine- grained patch tokens for enhanced feature representation.
2 Related Work
Many MCI methods do not use self-supervised objectives [3 29 30), but rather focus on aspects like diversity and robustness to missing channels. However, as noted in the introduction, self-supervised objectives like Masked Autoencoders (MAEs) can provide complementary information to further boost performance. MAEs themselves are often built for RGB images (e.g., [19 31 32
or those [8], 1 based assumptions of strong correlations
that are also strongly aligned like thermal images
between channels does not extend many MCI images. Many methods other domains followed
suit, performing channel-wise masking on protein markers [3 or masked both of spatial and spectral channels [34] 35). However, these methods rely on a fixed ratio of masked channels, which can be inadequate for the varying channel configurations found in MCL. In addition, prior work uses dual-branch architectures that treats spatial and channel representations independently. limiting meaningful interactions between channels, especially when spatial alignment is not guaranteed. In contrast, we dynamically mask a variable number channels each time. By combining both channel and patch-level masking within single encoder, we achieve better feature interaction.
The work most similar to ours CA-MAE [ which randomly masks a fixed proportion of patches from all channels and employs distinct decoders for reconstruction. In contrast, ChA-MAEVIT employs both channel and level masking with a shared decoder, and memory tokens serve as long-term memory to facilitate feature interactions.

---

There are other types of self-supervised learning methods beyond MAEs that have been explored in other settings (e.g., [3
36 451). However, similar to MAEs, they often make assumptions that do not generalize to MCI images. For example, self-distillation methods (e.g., SimSiam [40]. BYOL 41. DINO 42) process two different views through two encoders, and then map one view to the other using a predictor network. These methods often rely on complex augmentations designed for natural images to generate dual views, such as color jittering and Gaussian blur. These are less effective for MCI, where heterogeneous channels (e.g., LiDAR, thermal, RGB) carry distinct physical properties. Such augmentations can distort critical features like depth or intensity, hindering the model's 's ability to capture cross-channel dependencies essential for MCI [46, 47]. This makes masked modeling techniques attractive (e.g.. MAE [31]. SimMim [32 ), as they do not rely on augmentations that may be challenging to apply with more complex and varied images.
32
3 Channel-Aware MAE for Improved Cross-Channel Learning
Given a multi-channel image (MCI) denoted as X, which consists of various channels c, E C. our objective is to train a model M that takes X as input to generate representations and/or predictions. Following 12 K3 29 3D], we focus on the MCT setting where the model M is trained using all available channels C but tested with subset of those channels Ctest S C. We describe the details of the four main components of our proposed framework, which is illustrated in Fig. 2
3.1 Dynamic Channel-Patel Masking
As shown in Fig. aa), image patches in prior work seem to attend mostly to their own channels while neglecting the others, potentially missing rich interactions among channels. To mitigate this issue, we propose Dynamic Channel-Patel (DCP) Masking, which integrates both patch and channel-level masking strategies. We start creating image patches (Appendix B which results in n tokens per channel, each with d-dimensional embeddings, for e channels.
Our masking strategy consists of two components: random patch masking and dynamic channel masking. First, random patch masking, aims to generate a mask p_mask E {0, 1]mxe that applies a fixed masking ratio rp (e.g., 75%) to mask patches independently across each channel. Specifically, for each channel j. we uniformly sample a set of [n rpJ positions from the available patch positions, denoted S₁ C f1.2...., n with [Sy| n Tpl. We then mask these patches as
1 if E S3 where p_ mask,,, indicates the patch at position i of channel P_ p_mask,,, - { otherwise'
masked, and p. mask,,, 0 indicates unmasked. All channels have the same number of masked patches, but locations vary per channel because they are independently sampled, as opposed to prior work that first generates the mask for one channel and then replicates it to all other channels [35 48
The second component, dynamic channel masking, aims to generate a mask c_ c_mask E {0, 1] nxc that adaptively mask out some channels. Here, the term "dynamic" refers to the varying number of channels masked during training, rather than adaptation based on the specific input data. We start by uniformly sampling the number of channels to be masked, denoted as k ~ (0, 1, .,..., 1J. Then, we uniformly sample a set of channels, denoted as C' C (l.....c) c. with |C"| = k, and mask these
1
j E
channels out as C_ mask,
otherwise This masking approach is inspired by Hicrarchical
0t
Channel Sampling (HCS) 30 However, unlike HCS, which serves as a channel dropout technique that completely removes selected channels, our approach employs masked channels as supervisory signals, designating them as labels for the reconstruction process. This enables the model to directly learn inter-channel relationships, thus enhancing greater cross-channel feature interaction.
Finally, we integrate these two components for training images in our DCP Masking strategy by introducing hyperparameters Ppatch. Pehannel E (0,1). These values divide the unit interval into three sections to randomly determine how to use one or the other type of mask, as follows:
P_ mask
if 8 < Ppatch
mask
c mask
if Ppatch < Ppatch + Pehannel otherwise
(1)
p mask V c e_mask
4

---

where s is a selection value chosen uniformly at random. In practice, we found that optimizing the model with these two masking strategies simultaneously can be difficult, as it may lead to excessive information loss, making image reconstruction challenging. To address this, we adjust the hyperparameters to alternate between the two masking strategies separately. Specifically, setting Ppatch Pehanmel 0 merges both patch and channel masks into unified mask, and setting Ppatch = Pehannel = 0.5 allows the model to switch between patch and channel masks. We adopt these two straightforward configurations for all our experiments. Refer to Appendix [D] for the procedure DCP Masking, and Appendix for more analysis on the hyper-parameters
3.2 Memory Tokens
Reconstructing masked channels is challenging due to their inherent differences, since each channel encodes unique features not easily inferred from others, e.g., reconstructing LIDAR from RGB. Inspired by the concept of register tokens [6),, which are used to reduce artifacts in the feature maps of ViTs, we introduce memory tokens for MCI (Fig. 2 middle). These memory tokens are learnable embeddings that serve as long-term memory, allowing for the storage of global information across all channels. During training. these tokens gather channel-specific features using self-attention mechanisms, helping the model retain and propagate information across layers that might otherwise be lost due to masking. Additionally, these tokens assist in decoding image patches to facilitate the reconstruction process. During inference, memory tokens enable the model to retrieve stored context, effectively addressing the issue of missing channels. Similar to the [CT.S] token, memory tokens are incorporated into the input during both training and inference. However, while the [CLS] token is utilized as the final representation, the memory tokens are excluded.
Formally, we prepend memory tokens {Mi into the input sequence, resulting in [tcrs;; MI;...; My; f1,1; 12,15... in.cc where tcss and t.,j are the class token and patch token at i-th location of j-th channel, respectively. After masking some patches (Section (3.1, the remaining patches are fed into transformer encoder. Following [29 30), spatial information is incorporated through learnable positional embeddings, while channel-specific properties are captured by special channel tokens, each represented as a learnable embedding. These channel tokens are concatenated with the patch tokens and jointly processed by the transformer encoder and decoder.
3.3
Channel-Aware Decoder
After processing the unmasked tokens with the encoder, we reconstruct the masked patches using the unmasked ones with Channel- Aware Decoder. Unlike prior MCI-MAE methods that separate decoders for each channel or modality [8], we employ a single decoder to process tokens from all channels simultaneously (Fig. 2 right). This scales better the number of input channels (up to 18 our experiments), while also boosting performance.
Let the output sequence from the encoder be [īo,s; Mi; M1 E1; :.. tn], where v denotes the number of visible (i.e., unmasked) patches fed into the encoder. This sequence is shorter than the original image patch length due to the missing masked patches. To reconstruct these masked patches, we utilize u (n v) mask tokens m,, where each mask token is shared, learned vector representing the missing patch.
Incorporating Channel-Specific Information. To reconstruct channel-specific information, we combine patch tokens, whether visible (f,) or masked (m.), with their corresponding channel tokens, which are also optimized by the encoder. We define f()) function that returns the corresponding channel token for given patch token. The input to our Channel-Aware Decoder is thus:
T = [er.s Mi;...; M, + f(,) f + /(2); : + (f..); m; + [(m:), m, + (mm))
Incorporating channel tokens provides a more channel-specific context, which enhances the recon- struction process. Additionally, we incorporate learnable positional embeddings, which are shared with the encoder, to provide information about the position of each patch in the image.
Patch Reconstruction. We pass the token sequence Î into several Transformer Blocks, followed by Decoder Head to reconstruct the pixels of each image patch. Let T E [R(x=))dd represent the output of the patch tokens from the final Transformer Block. The Decoder Head is a linear layer W Weecoder E dxpp where p is the image patch size. The pixel reconstruction of the i-th patch of the
5

---

j-th channel, is computed as R,j (T W Weccddr)) E R We found that a lightweight decoder with just 1 Transformer Blocks is sufficient, aligning with findings from prior work [9 31
Reconstruction Loss. We use standard L2 loss on the image pixel, and the Fourier space L1 loss introduced in D. both computed only on the masked patches. Let P > Zj=1 maski,j
be the number of masked patches of the input image. Given &., R the original pixels of patch i in channel j. and its reconstruction, respectively, the reconstruction loss is as follows: Lpiiel -m= ش= maskij [1((x1,,, &tz); Liourier = Σn Eو= mask,,) [=((F(x,,)) (F(R,))
Lrccon = (1 - (1) Spixel + Ay- Ltourier:
where F is the amplitudes of Fast Fourier Transform. Following [I]. we set a fixed Ay to 0.01.
3.4 Hybrid Token Fusion
Prior work either utilizes the [CLS] token [16 49 or average pooling of patch tokens [ 19 for downstream tasks. We found that incorporating both our framework yields better per- use learnable query vector Apuich to interact with the patch tokens. The query attends to the patch tokens through cross-attention, and then combines with the [CLS] token to generate fused represen- T2)), where T, : represents
formance (Fig. 2 middle). Given the output sequence from the encoder,
tation frusion = fcL. @ the elementwise product, and a the sigmoid function. The fused representation, Stusion integrates global context from the [CLS] token while also being refined by detailed spatial information from the patch tokens. We then use a multi-layer perceptron (MIP) with GELU activation (50) to enhance the fusion representation and tailor it for downstream tasks: finnl- Linear (GEL (LieerLiierr(faaas
Training Objective. The output of Section 3.4 is then used to compute the task loss, e.g., cross- entropy for classification. Our final loss consists of three components: the primary loss for the specific task Ltask. regularization term La and reconstruction loss Lrecon (Eq. (2) as follows:
Lanal (1 Arecon) (Lrack + >a La) + Arecon -ereoon
where Aa is balance the task loss and the regularization term, and Arecon is to balance the re- construction loss with other losses. For regularization La, we use the losses introduced in [29]. in which Channel Diversification Loss applies to the channel tokens, and Token Diversification Loss applies to the patch tokens to encourage diversity learning. We set small Xa (e.g.. 0.001) as suggested by [29), and a fixed Areron = 0.99 for all experiments.
Experiments
Datasets. We evaluate on three cell microscopy and satellite datasets. (i) CHAMMI 12 channel- adaptive benchmark consists of varying-channel images sourced from WTC-11, HPA and CP, with 3, and 5 channels respectively. Together, these three datasets contain 220K mieroscopy images, in which, 100K images are for training, while the rest for testing across domain generalization tasks. (ii) JUMP-CP [27 a cellular imaging dataset, where each image has channels, consisting of fluorescence and 3 brightfield channels. We use compound perturbation plate BR001 16991. which contains 127K training, 45K validation, and 45K test images, across 161 classes. (iii So2Sat [TO] contains synthetic aperture radar and multispectral optical image patches from remote sensing satellites, with 18 channels (8 from Sentinel-1, 10 from Sentinel-2) and 17 climate zone classes. We use the city-split version, consisting of 352K training, 24K validation, and 24K test images.
Baseline methods. We compare to the following methods: DepthwiseViT [12) processes each input channel through a depthwise convolution layer, averages the filtered features, and feeds them into ViT backbone.
TemplateMixingVin (52 54 learns channel weights via shared parameter templates, forming a
patch projection layer for ViT backbone.
Hyper ViT [51] uses a neural network to generate channel-specific weights, concatenating
them into a patch projection layer for ViT backbone.
CA-MAE D extends MAE for multi-channel imaging. We add task loss to adapt this baseline.
ChAda- VIT 13 employs a shared projection for channel-wise feature extraction, combining the tokens with positional and channel embeddings for ViT processing.

---

Table 1: Comparison of channel adaptive models. We report the accuracy with standard deviation on the test set of three runs. "Full" refers to inference on all channels, while "Partial" means inference on subset of channels. ChA-MAEViT outperforms other baselines consistently across three datasets. Note: all models contain the same number of parameters, except CA-MAE due to its separate decoders (e.g., 4X parameters on So2Sat).
CHAMMI 12
JUMP CP 27
So2Sat 10
Model
Avg score
Full
Partial
Full
Partial
HyperNetVill ViT [51]
56.0811041
47.07
42.43 60.73-0.24
41.88.10.85
DepthwiseVil 12
61.8010.43
49.86=0.45
44.9810.71
60.41=0.22
43.41=11.00
TemplateMixing ViT (4
58.16:10.42
52.48=0.27
43.85:1.73
55.86=0.10
37.28:1034
CA-MAE [ + Sup. loss
59.15+
69.54
20.93++0.5
64.21
15.75+0.83
ChAda- ViT
13
63.9311.42
65.03
42.15422.3
56.98-1:46
12.38:12.33
Channel 30
64.97:10.58
67.51
56.49:10.53
61.03
46.16.10.10
DiCha ViT
29
69.7710.41
69.1910447
57.9810.411
63.3610.11
47.76.10.23
ChA-MAEVIT (ours)
74.63
90.7310.14
68.0510.21 67.44 52.1110.49
ChannelViT 30 Similar to ChAda- VIT, while incorporating Hierarchical Channel Sampling. DiChaViT 29 improves ChanneIViT with regularization terms and diverse channel sampling.
Additionally, we investigate the effect of combining the best supervised baseline, ViT, with the following SSL to evaluate the importance of incorporating self-supervision: MAE [ 31
SimCLR 37 38 SimSiam (4)), iBOT [43, and DINOv2 (44 More details in Appendix C
Implementation details. All baselines employ ViT-S (21M) as the backbone. our method, we remove one transformer block from the encoder to accommodate the decoder, ensuring that approach maintains the same number of parameters as the baselines. Refer to Appendix for details.
Metrics. We reported top-1 classification accuracy for the classification tasks on So2Sat [10 and
JUMP- CP 27.. For the representation learning tasks on CHAMMI 12] following the average F1 scores across WTC-11 and HPA.
29 we report
4.1 Results
Table I compares ChA-MAEVIT with different MCI-ViTs. ChA-MAEVIT significantly outper- forms the baseline models by an average of 10.0% across three datasets. Specifically, ChA-MAEViT exceeds the state-of-the-a model DiCha ViT by 5.0% on CHAMMI, channel-adaptiiee benchmark. For JUMP-CP and So2Sat, conducted evaluations in both Full (using all channels) and Partial (using subsets of channels) settings. In the full setting, our model surpasses the next best models by 21.5% and 3.0%, respectively. Additionally, ChA-MAEViT demonstrates its robustness in the partial settings, where we test JUMP-CP using five fluorescence channels and So2Sat using eight Sentinel-1 channels, showing improvement of 10.0% and 4.5% points respectively over prior work.
Table 2 presents the impact of integrating various SSL methods with the tep-pffformig supervised approach, DiCha VIT (29]. In the last row, we train DiCha VIT to reconstruct the masked patches using only our DCP Masking, without incorporating memory lokens, Hybrid Token Fusion, or Channel-Aware Decoder. While the combination with MAE gives the highest performance, DCP still significantly outperforms the best SSL-enhanced variant by 0.6 - 5.6% across three datasets. The performance gap highlights the effectiveness of DCP, which substantially exceeds the improvements gained from solcly combining with SSL objectives. Additionally, by only processing the unmasked patches, our method achieves significantly faster runtime compared to other SSL methods, e.g., 6X faster than DINOv2, making it both performant and computationally efficient for MCI tasks. Refer to Appendix C and Appendix Fig. for more discussion runtime and FLOPs comparisons.
To further demonstrate our approach's ability to generalize, Table reports segmentation performance on 38-Cloud (28), complementing our results on the classification (So2Sat [10], JUMP-CP 27 and representation learning tasks (CHAMMI [22 reported earlier. ChA-MAEVIT outperforms all baselines across evaluation metrics, demonstrating its ability to boost multi-channel learning.
7

---

Table 2: SSL methods when combined with the best supervised baseline DiChaViT [29] The last row shows the combination of DiCha VIT with Dynamic Channel-Pated Masking, Incorporating SSL. in DiCha ViT results in improvements. Best result with our masking strategy. Boldfaces and underlines indicates best and second best numbers, respectively.
JUMP-CP
So2Sat
CHAMMI
DiChaViT 29
Full
Partial
Full
Partial
SimCLR [37 38
70.7211.40
67.12110.99
56,9610068
64.4411.58
49.42.10.3
SimSiam
40
70.44
68.6410.56
57.72:10.99
64.0710.31
48.52:10.71
iBOT
43
70.71+0.44
68.87+0051
57.81+0.47
63.11+0.32
47.84
DINOv2 44
70.03:+0.28
66.9140.52
56.18:+0.66
63.42.+0.27
49 49.20-+0.72
MAE
1 31
70.27 78.62
64.21440.3
62.88:00.99
47.764
DCP Masking
71.47
84.02.10.49
65.72.10.43
66.02.10.22
50.52..0.45
Table 3: Comparing segmentation Performance on 38-Cloud [28]. We report the average and
standard deviation from three runs. ChA-MAEViT outperforms all baselines in all metrics.
Model
Accuracy
IoU
Precision
Recall
F1
ChannelViT 30
0.945:10.003
0.843+00012
0.919=0.015
0.911
0,915
DiChaViT 29
0.857=00011 0:0004
0.924=0.007
0.922++0006
0.023
CA-MAE 1I
0.946:10.002
0.845+0.003
0.917=0.005
0.916+10003
0.916400002
DiC 229 + CA-MAE []
0.959:10001
0.886#0.004
0.939
0.943+00004
0.941 #00002
ChA-MAEVIT (ours)
0.964
0.894
0.943
0.945
1.944
Table 4: ChA-MAEViT ablation study. Replacing any component from ChA-MAEVIT, such as substituting our DCP Masking with the random patch masking from CA-MAE (I ("w/o DCP Masking"). " results in a performance drop. The decline is most significant when DCP Masking is excluded. Incorporating all components consistently enhances performance across all three datasets.
CHAMMI 12)
JUMP-CP (27
So2Sat 10
Model
Avg score
Full
Partial
Full
Partial
ChA-MAEVIT
74.63
90.73
68.05
67.44
52.11
w/o DCP Masking
70.51
88.01
52.33
64.50
28.70
w/o Hybrid Token Fusion
73.84
88.25
66.23
65.48
51.40
w/o Memory Tokens
73.62
87.81
67.21
65.18
50.46
w/o Channel-Aware Decoder
72.95
87.52
67.05
65.78
49.88
Ablation study of ChA-MAEViT. Table presents the model's performance when component is replaced. Specifically, "w/o DCP Masking" indicates that we replace our DCP Masking with random patch masking in CA-MAE M. "wfo Hybrid Token Fusion" uses [CLS] token instead of Hybrid Token Fusion module, "w/o Memory Tokens" means no memory token is being used, and "w/o Channel-Aware Decoder" indicates replacing our Channel-Aware Decoder with CA-MAE's Separate Decoders. The results highlight the critical role of each of the components, especially DCP Masking, its removal has the most detrimental effect on performance.
Inference under varying channel configurations. Table 5 evaluates ChA-MAEViT and the best baseline DiCha VIT 29 when tested on varying numbers of channels of JUMP-CP. We train the model using all eight channels and assess performance after removing channels, e.g., testing with seven channels, as shown in column "7." we averaged all C% 8 possible combinations (refer to Appendix Table for for detailed results). ChA-MAEVIT demonstrates enhanced robustness, particularly when some channels are missing during inference.
Comparing masking strategies. Table 6 compares various masking strategies applied to ChA- MAEVIT. For fixed ratio approaches, we test several ratios (e.g., 50%. 75%) and report the best results. DCP Masking consistently outperforms others, achieving highest scores both Full and Partial settings. Notably, while random patch masking is effective in Full, it experiences a significant performance drop Partial. In contrast, dynamic channel masking methods adapted from (29 30 provide better adaptability for partial settings, and our DCP Masking approach improves even more.
8

---

Table 5: Performance on varying channel configurations during inference. Columns report the mean(sd)) across all channel combinations on JUMP-CP [27] e.g., "7" indicates testing on 7 out of channels (C3 combinations). ChA-MAEViT consistently shows improved robustness with missing channels at test time. Note that the reported std reflects variation across channel combinations, not model training variance. Refer to Table the Appendix for model variance report.
Number of channels at inference
Method
8
6
5
4
3
2
DiT 29
69.19 (1.91 54.49u2.4) 46.35(1.4) 38.0011.4) (0.09 23.97(4.9) 20.90(1.6)
ChA-MAEVIT 90.73 83.36(8.3) 74.55(1.77 63.46(138) (0.85 (1.2) 27.62(6.4) 21.59(2.1)
Table 6: Mask strategies in MCI when using with ChA-MAEVIT. While Random Patch and Channel Sampling perform similarly on Full channels, Random Patch experiences a significant drop in Partial channel settings. Dynamic Channel-Patch (DCP) Masking demonstrates its effectiveness in both Full and Partial channel settings, outperforming all strategies across three datasets.
CHAMMI
JUMP-CP
So2Sat
Mask Strategy
Avg score
Full
Partial
Full
Partial
Random Patch (fixed ratio) (e.g., 19
55 56]
70.51
88.01
52.33
64.50
28.70
Random Patch (dynamic)
68.23
85.54
55.86
64.84
31.92
Channel (fixed ratio) [8
47.97
73.00
65.25
65.22
38.59
Hierarchical Channel Sampling (dynamic, adapted (30)
69.86
83.71
67.75
65.20
48.77
Diverse Channel Sampling (dynamic, adapted 29
71.57
84.69
67.86
65.49
51.05
Channel + Patch (fixed ratio) [3| 48
48.46
69.41
62.19
62.20
32.18
DCP Combination (Pchannel Pptcch 0) (ours)
73.75
85.95
68.36
67.44
52.11
DCP Alternate (Pehannel = Ppatch = 0.5) (ours)
74.63
90.73
68.05
66.47
50.69
In addition, we analyze two variants of DCP that we use in all our experiments: Combination and Alternate. The DCP Combination unifies both patch and channel masks, while the DCP Alternate switches between the two for each training iteration. Both variants outperform traditional patch- and based masking, highlighting the effectiveness of joint channel- -pacc masking. Refer to Appendix F.5 for more analysis and hyperparameter settings for the DCP Masking.
4.2 Model ablations and sensitivity analysis
Impact of Memory tokens. Fig. aa & (b) show the accuracy achieved by numbers of memory tokens. Using memory tokens enhances performance, but beyond a certain point yields diminishing returns. This suggests that while memory tokens can improve performance, excessive reliance on them may negatively impact the interaction of patch features. We found that default of 4 memory tokens works well across the three datasets.
Attention patterns of Memory tokens. Fig. 41 shows the attention patterns between image patches and memory tokens of the encoder. Each group of channels displays distinct preferences for specific memory tokens. Fig. a shows that the VH channels primarily focus memory token 8, while the Lee-filterd channels have stronger attention toward memory token 1. Similarly, for JUMP-CP in Fig. b)) the Brightfield channels allocate more attention memory token 3, whereas Fluorescence channels show a slight preference for memory token 1. This suggests that each type of channel utilizes different memory tokens store the global information necessary for feature extraction.
Token pooling strategies. Table compares the performance of various token pooling strategies on CHAMMI and So2Sat In general, combining the [CLS] token with the average of the patch tokens gets better performance than either alone. By utilizing both the global [CLS] token and the fine-graine lne-grained patch tokens, Hybrid Token Fusion consistently outperforms others across all settings.
Reconstruction loss lambda. Fig. 3(e & (d) analyze the effect of the reconstruction weight Arecon (Eq. 3 on model performance. Consistent with prior work [19 22 large value of Lrovon gives the best results. However, performance drops significantly using just the MAE loss (i.e., Areeon = 1), noting the contributions of both losses. In all our experiments, we set a fixed Crecon to 0.99.

---

Accumaiy (Pull)
Areunayy (Partal))
75
Accuraty
Score
60
66
74
10
Full
Partial
8 16
73
16
0.01 0.1
0.5 0.9
0.99
0.01 0.1 0.5 0.9 0.99
Number memory tokens
Number of menoory tokens
Arecnn
(a) So2Sat
(b) CHAMMI
(c) So2Sat
(d) CHAMMI
Figure 3: Impact of the number memory tokens and reconstruction lambda Arecon (Eq. (3). (a)
& (b) Using 8 tokens improves performance, however, using more memory tokens (e.g. 24)
may reduce the effectiveness. (c) & (d) Arecon = means without the reconstruction loss, while
Arecon 1 indicates only using the reconstruction loss. For Areco 1 on So2Sat, we run linear probing. Areon 0.99 works best for ChA-MAEViT on both datasets.
Memory
06
Memory
08
token
04
token
07
03
Lee-filtered
Bands
Pluorsteence
Brightfield
channels channels
B2-B12
chaunels
channels
Channel
Channel
(a) So2Sat
(b) JUMP-CP
Figure 4: Attention between image patches and memory tokens of the encoder. Each channel group focuses on different memory tokens. (a) So2Sat: VII channels utilize memory token 8, whereas Lee-filtered channels attend more to memory token 1. (b) JUMP-CP: Brightfield channels focus on memory token 3, while Fluorescence channels favor memory token 1.
Table 7: Comparison of token pooling methods. Hybrid Token Fusion achieves the best perfor- mance, demonstrating the benefit of leveraging both global and fine-grained local tokens.
Shored Evpoodrr w/jurt
Aocaracy Seore
Separate Docodres
Cheand-Awau Dinoderr (Ous)
700
+1.7
So2Sat
CHAMMI
Token Pooling Method
Full Partial
Pull
Partial
Full
Parlis
JUMPOP
So2Sat
CHAMMI
Avg Patch tokens
71.41
64.86 51.87
Figure 5: Different Decoders when using
[CLS) token
73.84
65.48 $1.40
with ChA-MAEVIT. Our Channel-Aware De- coder outperforms the best baseline by 1.1 3.2% all three datasets.
CCLS Avg Patch tokens
73.96
66.23 49.45
Hybrid Token Fusion (ours)
74.63
67.44 52.11
Channel-Aware Decoder analysis. Fig. 5 shows Channel-Awan Decoder in ChA-MAEVIT outperforms both Separate Decoders M and Shared Decoder W/out Channel Tokens by 1.1 3.2% across three datasets in full and partial settings. Channel-Aware Decoder is also more efficient than Separate Decoders thanks to its shared parameters, e.g., X fewer parameters on So2Sat.
Conclusion
In this paper, introduce ChA-MAEVIT, novel MAE-based method to enhance feature learning in Multi-Channel Imaging (MCI) ViTs. First, we introduce Dynamic Channel-Patch Masking, in which adaptively mask both image patches and channels and train the model reconstruct the masked patches. Additionally, we incorporate memory tokens to preserve global context across channels and Hybrid Token Fusion module combine features from local patch tokens and global class tokens. Furthermore, we propose Aitannel-Aware Decoder to efficiently reconstruct channel-specific details. Experiments conducted the CHAMMI, JUMP-CP, and So2Sat datasets demonstrate that ChA-MAEViT outperforms prior MCI-ViTs by 3.0 21.5%, highlighting the significance of enhanced cross-channel interactions. Future research could explore the adaptation of the channel-awan framework to additional complex modalities, such as volumetric medical imaging. which is composed of sequential evo-dimensional slices.
10

---

Acknowledgments and Disclosure of Funding
This study was supported, in part, by the National Science Foundation under NSF-DBI awards 2134695 and 2134696. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the supporting agencies.
References
[1] Oren Kraus, Kian Kenyon- Dean, Saber Saberian, Maryam Fallah, Peter McLean, Jess Leung, Vasudev Sharma. Ayla Khan, Jia Balakrishnan, Safiye Celik, et al. Masked autoencoders for microscopy are scalable learners of cellular biology. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11757-11768, 2024.
[2] Christian Szegedy. Wei Liu, Yangging Jia, Pierre Sermanet, and Scott Reed. Dragomiranguelov, dumitru erhan, vincent vanhoucke, and andrew rabinovich. 2015. going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[3] Xuelei Li, Liangkui Ding, Li Wang, and Fang Cao. Fpga accelerates deep residual learning for image recognition. In 2017 IEEE 2nd Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), pages 837-840. IEEE, 2017.
[4] Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6113111 6114. PMLR, 2019.
[5] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 704604 2017.
[6] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
(7) Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), 2022.
[8] Chaitanya Ryali, Yuan- Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, al. Hiera: A hierarchical vision transformer 2023.
without the gell-and-whistless arXiv preprint arXiv:2306.0090
[9] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu. Vmamba: Visual state space model. Advances in neural information processing systems, 37: 37:103031-103066 2025.
[10] Xiaoxiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Hossein Bagheri, Jian Kang, Hao Lichao Mou, Guicheng Zhang. Matthias Häberle, Shiyao Han, Yuansheng Hua, Rong Huang, Lloyd Hughes, Yao Sun, Michael Schmitt, and Yuanyuan Wang. New: So2sat lcz42, 2019. URL |RI.https http////aaaaaaa aediaatum.ce ub. tum
de/ 1483140
[11] Jian Zou, Tianyu Huang, Guanglei Yang, Zhenhua Guo, Tao Luo, Chun-Mei Feng, and Wangmeng Zuo. Unim 2 ae: Multi-modal masked autoencoders with unified 3d representation for 3d perception in autonomous driving. In European Conference on Computer Vision, pages 296-313. Springer, 2024.
[12] Zitong Chen, Chau Pham, Siqi Wang. Michael Doron, Nikita Moshkov, Bryan A. Plummer, and Juan C Caicedo. CHAMMI: A benchmark for channcl-adaptive models in microscopy imaging. In Thirry- Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL httts::/opperrviiet aet//forurPd--LnctbZeeee
[13] Nicolas Bourriez, Thab Bendidi, Cohen Fthan, Gabriel Watkinson, Maxime Sanchez, Guillaume Bollot, and Auguste Genovesio. Chada-vit Channel adaptive attention for joint representation learning of heterogeneous microscopy images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[14] Min Chen, Aaron Carass, Amod Jog. Junghoon Lee, Snehashis Roy, and Jerry L Prince. Cross contrast chlti-channel image registration using image synthesis for mr brain images. Medical image analysis, 36: 14, 2017.

---

[15] Kian Kenyon-Deal Dean. Zitong Jerry Wang. John Urbanik. Konstantin Donhauser. Jason Hartford, Saber Saberian, Nil Sahin, Ihab Bendidi, Safiye Celik. Juan Sebastián Rodríguez Vera, Marta Fay. Imran S Haque. and Oren Kraus. Vitally consistent: Scaling biological representation learning for cell microscopy. 2025.
URL
ntt.::/.eeee/ttoeeeeeeeee id ai
[16] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference Learning Representations, 2024, BtLLLttttp:///peeee..er
[17] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision, pages 348-367. Springer, 2022.
[18] Xiang Zhang, Huiyuan Yang, Taoyue Wang, Xiaotian Li, and Lijun Yin. Multimodal channel-mixing: Channel and spatial masked autoencoder on facial action unit detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6077-6086, 2024.
[19] Feng Liang, Yangguang Li, and Diana Marculescu. Supmae: Supervised masked autoencoders are efficient vision learners, arXiv preprint arXiv:2205.145400 2022.
[20] Yong Cheng, Wei Wang, Lu Jiang, and Wolfgang Macherey. Sele-superviedd and supervised joint training for reshurce-rich machine translation. International Conference on Machine Learning, pages 1825-1835. PMLR, 2021.
[21] Junwen Bai, Bo Li, Yu Zhang, Ankur Bapna, Nikhil Siddhartha, Khe Chai Sim, and Tara N Sainath. Joint unsupervised and supervised training for multilingual asr. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6402-6406. IEEE, 2022.
[22] Yifei Xin, Xiulian Peng, and Yan Lu. Masked audio modeling with clap and multi-ohjecttive learn ing. In INTERSPEECH, pages 2763-2767. 2023. URL /https://doi. org/10 21437/ 2ww.///Ineeee//ee Interspeech 2023- 2488
[23] Yu Gao, Xintong Han, Xun Wang, Weilin Huang, and Matthew Scott. Channel interaction networks for fine-grained image categorization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 10818 10825, 2020.
[24] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pages 27378-27394. PMLR, 2022.
[25] Jie Hu, Li Shen, and Gang Sun. Squecze-and-cxciing excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[26] Jianwei Yang, Zhile Ren, Chuang Gan, Hongyuan Zhu, and Devi Parikh. Cross-channel communication networks. Advances in Neural Information Processing Systems, 32, 2019.
[27] Srinivas Niranj Chandrasckaran, Beth A. Cimini, Amy Goodale, Lisa Miller, Maria Kost-Alimova, Nasim Jamali, John Doench, Briana Fritchman, Adam Skepner, Michelle Melanson, John Arevalo, Juan C. Caicedo, Daniel Kuhn, Desiree Hernandez, Jim Berstler. Hamdah Shafgat-Abbast David Root, Sussane Swalley, Shantanu Singh. and Anne E. Carpenter. Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations. bioRziv, 2022. doi: 10.1101/2022.01 05.4450090. URL (ttpss/p::/// /www. biorxiv. org/ /ttp:///wneeeeeee/////////////// 01.00..2.......................0 05. 475090
[28] Sorour Mohajerani and Parvaneh Sacedi. Cloud-net: An cnd-to to end cloud detection algorithm for landsat 8 imagery. In IGARSS 2019-2019 IEEE international geoscience and remote sensing symposium, pages 1029-1032. IEEE, 2019.
[29] Chau Pham and Bryan A. Plummer. Enhancing feature diversity boosts channel-adaptive vision transform- ers. In Advances in Neural Information Processing Systems (NeurIPS). 2024.
[30] Yujia Bao, Srinivasan Sivanandan, and Theofanis Karaletsos. Channel vision transformers: An image is worth 1 16 16 words. In The Twelfth International Conference on Learning Representations, 2024. URL http:://ee/eerrreeeeuuuu wwtp:ewwwee//trtrreeeeee - b5hBG
[31] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6000-16009 2022.
12

---

[32] Zhenda Xie. Zheng Zhang, Yue Cao. Yutong Lin. Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 9653-9663, 2022.
[33] Zachary Sims, Gordon B Mills, and Young Hwan Chang. Mim-cyeif: masked imaging modeling for enhancing eyclic immunoftuorescence (cycif) with panel reduction and imputation. bioRxiv, 2023.
[34] Junyan Lin, Feng Gao, Xiaochen Shi, Junyu Dong, and Qian Du. Ss-mae: Spatial-spectral masked autoencoder for multisource remote sensing image classification. IEEE Transactions Geoscience and Remote Sensing, 61:1-14, 14, 2023.
[35] Yue Wang. Ming Wen, Hailiang Zhang, Jinyu Sun, Qiong Yang. Zhimin Zhang, and Hongmei Lu. Hsimae: A unified masked autoencoder with large-scale pre-training for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing. 2024.
[36] Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum. cookbook of self-supervised learning. arXiv preprint arXiv:2304. 122/0, 2023.
[37] Ting Chen, Simon Korblithh Mohammad Norouzi. and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[38] Ting Chen, Simon Korbblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances neural information processing systems, 33:22243 22255,, 2020.
[39] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9588-9597, 2021.
[40] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17750-15758 2021.
[41] Baan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Flena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to superviedd leannigg learning. Advances neural information processing systems. 33:21271-21284, 2020.
[42] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IFEF/CVF international conference on computer vision, pages 9650-9660, 2021.
[43] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
[44] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Yao Huang, Shang-Waa Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnacve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOV2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URRLAtttts://ppenreeet net seet/foorumidd=aaStttee Featured Certification.
[45] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference on Computer Vision, pages 456-40 473. Springer, 2022.
[46] Alfie Roddan. Tobias Czempiel. Daniel S Elson, and Stamatia Giannarou. Calibration-jitter: Augmentation of hypersppetral data for improved surgical scene segmentation. Healthcare Technology Letters, 11(6): 345-354, 2024.
[47] Ankit Patnala, Scarlet Stadtler, Martin G Schultz, and Juergen Gall, Generating views using atmospheric correction for contrastive self-supervised learning of multispectral images. IEEE Geoscience and Remote Sensing Letters, 20:1-5, 2023.
[48] Vishal Nedungadi, Ankit Kariryaa, Stefan Oehmcke, Serge Belongie, Christian Igel, and Nico Lang. Mmearth: Exploring multi-mode pretext tasks for geospatial representation learning. In European Conference on Computer Vision, pages 164 182. Springer, 2024.
13

---

[49] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov. Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Weit Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[50] Dan Hendrycks and Kevin Cimpel. Gaussian error linear units (gelus), arXiv preprint arXiv:1606.0855 2016.
[51] David Ha, Andrew Dai, and Quoe Le. Hypernetworks. In International Conference on Learning Represen- tations (ICLR), 2016.
[52] Bryan A. Plummer, Nikoli Dryden. Julius Frost, Torsten Hoefler, and Kate Saenko, Neural parameter allocation search. In International Conference on Learning Representations (ICLR), 2022.
[53] Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter sharing. In International Conference on Learning Representations (ICLR), 2019.
[54] Chau Pham, Piotr Teterwak, Soren Nelson, and Bryan A. Plummer. Mixturegrowth: Growing neural networks by recombining learned parameters, In JEEE Winter Conference on Applications of Computer Vision (WACV), 2024.
[55] Kian Kenyon- Deann. Zitong Jerry Wang. John Urbanik, Konstantin Donhauser. Jason Hartford, Saber Saberian, Nil Sahin, Ihab Bendidi, Safiye Celik, Juan Scbastián Rodríguez Vera, Marta Fay, Imran S Haque, and Oren Kraus. Vitally consistent: Scaling biological representation learning for cell microscopy, 2025. URL /tttps://openrevieet nwt/foruuniiiieee -dd-nioosa5554
[56] Alexandre Eymaël, Renaud Vandeghen, Anthony Cioppa, Silvio Giancola. Bernard Ghanem, and Mare Van Droogenhroeck. Efficient image pre-training with siamese cropped masked autoencoders. Eumppean Conference on Computer Vision, pages 348-366. Springer, 2024.
[57] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre- pre-training of image transformers. In International Conference on Learning Representations, 2022. URL tttppttt:///opeeeeeeee
net/forum?
[58] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vectors for similarity scarch. In International Conference Learning Representations, 2019. URL https:// openreview wwt..///oorrrridddcca2222222.
[59] Gianluca Donato and Serge Belongie. Approximate thin plate spline mappings. In Computer Vision-ECCC 2002: 7th European Conference Computer Vision Copenhagen, Denmark, May 28-31, 2002 Proceed- ings, Part pages 21-31. Springer, 2002.
[60] Eu Wern Teh, Terrance DeVries, and Graham W Taylor. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16, pages 448-464. Springer, 2020.
[61] Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwer. Salman Khan, and Fahad Shahbaz Khan. Rethinking transformers pre-training for multi-spectral satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27811- 1- 27819, 2024.
[62] C Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp. S Candido, M Uyt- tenDAele, and T Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. 2023 ieee. CVF International Conference on Computer Vision (ICCV), pages 2022.
[63] Aamir Mustafa, Aliaksei Mikhailiuk, Dan Andrei Ilieseu, Varun Babbar, and Rafal Mantiuk. Training task-specific image reconstruction loss. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 23 2319-2328, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction reflect our main contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: While we demonstrate that ChA-MAEVIT can handle novel channels never seen during training a preliminary experiment, we leave systematic investigation of this setting for future work, as mentioned in the Broader Impacts and limitations.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our method is described clearly in Fig. and Section 3 We also provide the implementation details in Appendix [E] and release the code at https: /tte:://gittiie //tbthub.ttttg/gg//ggg com/
chaudatasciencee cha mae_vil vit
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We release the code with instructions download the datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "cipps.c/puulli//guu /tte:///uuddde/oooSuuubbmiibia ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implementation details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We provide error bars our resulting tables, such as in Table T and Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
17

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the implementation details with computer resources in Appendix P
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research conducted conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: include Broader Impacts and limitations.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original papers and provide a link to the codes. The license and terms are properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release the code with a document for people use.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: In this paper, we do not use crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLMs for editing the writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/: http://///////oofeeeeee/e/////// for what should or should not be described.
21

---

A Broader Impacts and limitations
The development of ChA-MAEViT represents a significant advancement in Multichannel Imaging. with benefits such as improved medical diagnostics and faster healthcare research. Its applications in satellite imaging also hold potential for environmental monitoring. However, there are risks, including the possibility of misuse for invasive surveillance systems, highlighting the need for ethical considerations and responsible deployment.
While our model is capable of handling unseen channels by leveraging relationships learned from known ones (Appendix F.TO), we have not thoroughly evaluated its performance in such scenarios. Adapting to novel channels requires mapping them existing ones, which becomes more challenging under domain shifts. We leave a systematic investigation of this setting to future work. Additionally, approach requires some extra apper-parameter tuning, which can increase computational resource demands.
B Image patching
Formally, let I Rbxwxe represent multi-channel input image, where (h, w/) denotes the dimen- sions of the image and is the number of channels. We begin by patchifying each channel into n 2-D patches of size p p in pixels. This process yields n x patches, denoted Xi,j RRPx,, where A:a is the number of patches per channel, indicates the spatial location of given patch, and p
denotes the j-th channel. Next, we flatten each patch x,,, E R and then pass into a shared linear xd This results in patch tokens t. (xx.) W) E IRd, leading to a sequence of c patch tokens [t ii. tn.e]. Note that we utilize shared projection across all channels, thus the number of parameters remains constant regardless of the number of input channels.
projection W E IRe²xd
C Combination of DiC with SSL
We adopt the following SSL methods with DiCha VIT 29 as baselines: SimCLR 37 38 Siam [40], MAE [31] iBOT [43), and DINOv2 44).
Sim-
utilizes augmentation strategies that include random cropping, color distortion, and Gaussian blur to generate the dual views of an input image. This approach requires twice the computational resources per sample compared to single-pati methods. Since hue and saturation not well defined multi-channel images, we apply the augmentation by only adjusting brightness and contrast.
SimCLR 37 38
SimSiam 40 employs similar augmentations as SimCLR, but eliminates the use of negative pairs by stop-gradient This operation helps prevent representation collapse by decoupling the optimization of the twin network branches, which simulates an alternating optimization process similar to the Expectation Doaatttion-Maimmzzationn (EM) approach. Additionally, SimSiam does not require momentum encoders.
MAE 31 involves randomly masking large portion of the input image and training the model to reconstruct the missing content. Unlike contrastive methods, MAE does not depend on augmentation
pipelines, which can be ineffective for heterogeneous channels in MCI [46 47 Additionally,
by processing only a small fraction the image, this method significantly lowers computational demands, thereby enhancing efficiency. Note that combining MAF with DiCha (29 results in
model very similar to CA-MAE [ an extension of MAE for multi-channel imaging. However, key components from DiCha VIT such diverse regularizers, channel tokens, and the diverse channel sampling strategy - are not present in CA-MAE, making the combined model more expressive for handling multi-channel imaging.
combines masked image modeling with self distillation by utilizing dual augmentation streams that include both masked and full views. This encourages the model to align representations across different views while reconstructing missing content. Particularly, iBOT uses clockwise masking introduced in BEiT [57], where a block of image patches is masked each time. To adapt this clockwise masking for multi-channel images, we generate a mask for each channel, ensuring that the masked blocks only cover the areas in the channel. Compared to MAE, iBOT introduces
iBOT 4
22

---

additional computational overhead due to its use of teacher-student distillation. Furthermore, the dual-stream processing increases the per-sample training cost, making iBOT more computationally demanding than single-view methods like MAFs.
DINOv2 4 employs multi-cropping alongside various augmentations, such as color jittering, Gaussian blur, and random solarization, well as self-distillation techniques. Notably, DINOv2 also incorporates additional loss functions, such as iBOT loss [43] and KoLeo loss [58]. During training, DINOv2 generates multiple crops from each image (e.g.. 2 global views and 8 local views) which leads to significantly higher computational demands compared to other SSL methods. For example, training DINOv2 on JUMP-CP dataset using two GPUs requires approximately 6 days, whereas SimCLR takes around 2 days and MAE only requires 1 day. Refer to Fig. 6 for FLOPs counts.
To integrate these SSL methods with ViC 29 we add the task loss and optimize it along with the SSL loss. Additionally, we explored another variant that included an additional branch utilizing standard augmentations recommended by the authors of the dataset optimize the task loss and return the predictions at test time (e.g.. only using random cropping, horizontal flipping, and thin-plate-spline transformation (59) for CHAMMI). We observed that relying exclusively on
complex augmentation pipelines, like those used in SimCLR, negatively impacts the performance
of the combined model, while incorporating at least one view with regular augmentation enhances the model's learning capability. For self-distillation methods such as DIN DINOv2, we found that the performance of the teacher network is slightly better than that of the student network, thus we report the performance on the teacher network.
D Dynamic Channel-Patch (DCP) Masking Algorithm
Algorithm| D Joutlnes the procedure of Dynamic Channel-Patch (DCP) Masking in Section of the main paper, where we combine both patch and channel masking strategies. In practice, setting
Ppatch = Pehanmel = 0 merges both patch and channel masks into a unified mask, and setting Ppatch- Pehannel 0.5 allows the model to switch between patch and channel masks. We adopt these two straightforward configurations for all our experiments. For Ppatch = Pehannel = 0 (i.e., DCP Combination), we found that works well with small value of patch mask ratio ry. and simply set Tp 0.25 for all the experiments. For Ppatch Prhannel = 0.5 (i.e., DCP Alterntee), we used a larger value of ry 0.75. Refer to Appendix F5 for more analysis on these hyperparameters.
E Implementation details
For most of the baseline models, such as HyperNetViT. ChanneIViT, ChAda-ViT, and For iBOT and DINOv2, we adapt the We employ ViT small architecture with 21M parameters as the backbone for all methods. To ensure that we maintain the same number of parameters as the baselines, we removed one layer from the encoder to incorporate it into the decoder for our method. For both CHAMMI and JUMP-CP, we use a patch size of 16, while for the So2Sat dataset, we opt for a patch size of 8. We trained the models using the AdamW W optimizer, aiming to minimize the cross-entropy loss for the JUMP-CP and So2Sat datasets, while employing proxy loss for CHAMMI.
DiChaViTI we utilize the implementation from 12] |
implementation from [44] 2
For the baselines, we utilize the [CLS] token from the final layer as the feature representation and train the model to minimize the proxy loss (60] We then evaluate the model on various tasks using the evaluation code provided by [12]. incorporating 1-Nearest Neighbour classifier to calculate the macro-average Fl-score for each task individually The channel-adaptive interfaces are adapted from the authors' implementation code In addition to the model, we apply the same data augmentation techniques recommended by the authors, including thin-plate-spline (TPS) transformations 59 Each model is trained for 100 epochs with learning rate of 0.0004 and a batch size of 64.
CHAMMI dataset 12
23

---

Algorithm 1: Dynamic Channel-Patch Masking
Input
Number of patches per channel Number of channels c; Patch mask ratio Tpi
Probability of using patch masking Ppatchi
Probability of using channel masking Pchanneli
(0 S Ppttch Pehannel I 1)
// Random Patch
P_ mask 0'xe // initiaalize patch mask
For = c.
p p_mask[[:]] generate_random
mask(n,
EndFor
// Dynamic Channel Mask
Uniformly sample number of channels to mask c-1.. 1}
Uniformly sample k masked channels C' = sa.mple((,....,
c_mask = 0 // initialize channel mask
For j C": (_mask((]]
// mask whole channel
10 EndFor
// Masks For Current Iteration
Sample ~ ((, E IR
If Ppatch
13 mask = 1p_mask///// // Using patch mask
14 Else s < Ppatch + Pchannell
15 mask = c_maak/ // Using channel mask
Else: // Combining both masks
mask = p_ p_mask| mask
Output :mask {0, 1)xx, where denotes mask
warm up the learning rate for the first 10 epochs, reaching a peak 0.0004. After this initial period, the learning rate gradually decays to 10 according cosine scheduler. To mitigate overfitting,
JUMP-CP [27] and So2Sat [10] datasets. Following (29 30
apply weight decay of 0.04 augmentation techniques outlined in [29]. For the final prediction,
to the weight parameters while excluding the bias and normalization terms. We use the same data
utilize [CL.S] token from the Transformer encoder into a classifier head to predict the probability of each class. The final model
checkpoint is selected based validation performance. Each model is trained for 100 epochs, with batch size of 64 on JUMP-CP and 128 on So2Sat.
Compute resources. Experiments conducted on So2Sat and CHAMMI utilize one NVIDIA RTX GPU with 48GB RAM, alongside three Intel(R) Xeon(R) Gold 6226R CPUs. For JUMP-CP experiments, two NVIDIA RTX GPUs and six Intel(R) Xeon(R) Gold 6226R CPUs were used.
F Additional Results and Analyses
F1 FLOPs Count for Combination of DiCha VIT with SSL
In Fig. we present the FLOPs for each of the SSL methods when combined with DiChaVIT, shown in Table 2 of the main paper. ChA-MAEViT achieves the lowest FLOPs, approximately one-sixth that of DINOv2.
F.2 CHAMMI Benchmark Results
Table presents the F] scores of various MCI-VIT models (VIT-S backbone) evaluated on CHAMMI benchmark [12] The evaluation consists of nine tasks, including six nut-of-distribution (OOD) tasks. ChA-MAEViT demonstrates superior overall performance, achieving the highest scores in six out of the nine tasks across different settings.

---

300
GFLOPs
200
100
ShnCLR
Sin
iBOT
DINOy2
ChA-MAEWT (ours)
CA-MAE
Figure 6: FLOPs Count for Combination of ViC with SSL. We show FLOPs for each method when trained on JUMP-CP. ChA-MAEVIT achieves the lowest FLOPs, x 1/6 that of DINOv2.
Table F1 Scores of MCI-VIT models on CHAMMI benchmark [2]. ChA-MAEViT demon-
strates better overall performance CHAMMI, achieving the highest scores in out of 9 tasks across various settings. "OOD" refers to out-of-distribution tasks. All models have the same number of parameters, except CA-MAE with 3X more parameters due to its use of separate decoders.
Average OOD
WTC
HPA
CP
Model Mean WTC
HPA
CP
Task1
Task2
Task1 Task2 Task3
Task1
Task2
Task3
Task4
VeyperNetViT [5]
47.17 45.78 67.61
28.11
58.83
45.78 88.78 82.70
52.52
82.13
53.74
23.16
7.42
DepthwiseViT
12
50.44
52.19 71.41
27.72
69.81
52.19
91.65 88.04
54.78
81.24
54.08
23.21
5.87
TempMixnngViI
[52
47.33
51.52 64.80
25.66
61.66
51.52
85.01 79.91
49.69
77.45
48.83
22.56
5.60
CA-MAE 1
Sup. Loss
48.82
61.85 56.45
28.15 77.57
61.85
84.45
71.99
41.01
76.49
56.52
19.43
8.49
ChAda-ViTT
IK
50.82
67.18 60.67
24.60
77.58
67.18
87.49
75.94
45.41
83.92
45.58
21.94
6.28
ChanneIViT
30
52.54
67.58 62.35
27.81
78.36
67.58
83.93
76.73
47.97
77.70
16
21.89
6.38
DiChaVil
29
55.36 75.18 64.36
26.53
80.87
75.18
88.08
79.26
49.45
84.08
53.03
20.95
5.60
ChA-MAEVIT (ours) 58.02
77.15
72.11 24.81
84.52
77.15
94.14
87.47
56.75
56.68
10.25
7.50
F3 Performance when only using SSL Objectives
We evaluate the effectiveness of ChA-MAEVIT in the SSL setting by comparing the performance of different SSL. methods. Specifically, all models were trained solely using SSL objectives, i.e., no
task-specific loss was incorporated. For the So2Sat [10] and JUMP-CP (27] datasets, reported the accuracy from linear probing over 20 epochs. As shown in Fig. 7 ChA-MAEViT achieves the highest score of 37.7%, surpassing other approaches by 1.0 8.5% on CHAMMI. In Fig. 8 our method consistently outperforms baseline models in both Full and Partial settings on So2Sat. A similar trend is observed for JUMP-CP in Fig. This demonstrates the effectiveness of our method in SSL settings for MCI, highlighting its ability to extract meaningful representations in scenarios where supervised labels are unavailable.
F.4 Leave-One-Channe-OOt at Test Time
In T we present the results of training the model using all eight channels of JUMP-CP and then testing it with various combinations of seven channels. This provides a detailed result for column "7" of Table in the main paper, which represents C different channel combinations. For each combination, we report the mean and standard deviation of the model's performance based on three runs. Our results demonstrate that ChA-MA EViT achieves an improvement of 17 23% for each combination compared to the bascline models DiChaVil 229 and ChannelViT [30]
25

---

Score
35-
SimCLR
ShinSia
iBOT
DINOV2
ChA-MAEVIT (ours)
CA-MAE
Figure 7: Comparison of SSL methods CHAMMI [12]. All models were trained solely using SSI. objectives, Le., without any task loss. ChA-MAEVIT achieves the highest score, outperforming other baselines by 1.0 8.5%, demonstrating the effectiveness of our approach over other methods SSL settings for MCI.
Acc. (Partial)
Acc. (Full)
60
20
40-
SimCLR
SimSSaa
iBOT
DINOV2
CAA-MAEVIT (ows)
CA-MAE
SimCLR SiniSaat
iBOT DINOv2
MAEVI (ours)
CA-MAE
Figure 8: Comparison of SSL methods on So2Sat [10] All models were trained solely using SSL objectives, i.e., without any task loss, then trained with linear probing for another 20 epochs. ChA-MAEViT outperforms other baselines both Full (left) and Partial channel settings (right), demonstrating its effectiveness for SSL in both scenarios.
Ace. (Partial)
Acc. (Full)
20.0
19
18
19.5
17
19.0
16
SimCLR
SimSiam
iBOT
MAEVTT (ours)
DINOv2
CA-MAE
SimCLR
SimSiam
iBOT DINOv2
ChA- (ours)
CA-MAE
Figure Comparison of SSL methods on JUMP-CP [27]. All models were trained solely using SSL objectives, Le., without any task loss then trained with linear probing for another 20 epochs. ChA-MAEViT achieves the highest score when tested on Full channels (left) and performs comparably to the pop-errfonnng baselines on Partial channel settings (right).
26

---

Table 9: Analysis of Seav--One-Channe--ne Testing. We present the detailed results for column "7" in Table of of the main paper. Each row shows the results obtained by leaving out one channel and testing the model the remaining seven channels. For all experiments conducted, the mean + standard deviation based on three runs. Our method, ChA-MAEVIT, consistently outperforms the best baselines ChannelViT (0) and DiChaViT tested combinations.
provide
29 by 17 23% across all
Missing channel at inference
ChannelViT 30
DiChaViT 29
ChA-MAEViT (ours)
61.72 + 0,48
63.48 = 0.20
86.29 + 0.37
61.21 0.41
62.72 0.28
86.16 = 0.32
2
61.90 + 0.48
63.28 0.31
87.13 + 0.24
37.70 0.60
38.83 0.46
61.79 0.58
58.52 + 0.63
59.61 0.17
82.67 = 0.53
67.28 0.53
69.12 = 0.16
89.24 + 0.35
6
67.20 + 0.59
69.06 0.20
87.54 + 0.20
67.37 0.60
69.21 0.19
86.03 + 0.32
E5 Dynamic Channel-Patch (DCP) Masking Analysis
DCP Combination with varying masking ratios rpp- Fig. shows the impact of varying random patch masking ratios (rp) on the performance of ChA-MAE
when trained with DCP Combina- tion (Ppatch PPchannel = 0). The blue and purple lines indicate the performance for full and partial channels on JUMP-CP, respectively. As the masking ratio rр increases, accuracy for both full and partial channels decreases due to excessive information loss, making reconstruction more difficult. Since DCP Combination masks both patches and channels, it is reasonable to use small rp, in contrast to previous studies that often utilize high ratio (e.g., 0.75). For example, rp of 0.25, when combined with dynamic channel masking, together mask = 0.65 patches in the original images. We found DCP Combination works well with small value of ry. thus use rp of 0.25 as default value for all the datasets for DCP Combination.
DCP Alternate with varying proportions Pehannel and Pptthh- Fig. TT evaluates the performance of DCP Alternate in ChA-MAENE by varying the proportions of channel- and patch-level masking.
We adjust Pehannel, while setting Ppatch = Pchannel to control the contribution cach masking
strategy. For example, Pchenne] = 1 indicates exclusive channel-level masking, and Pehannel 0 denotes only using patch-level masking. The blue and purple lines represent accuracy on JUMP-CP for Full and Partial channel settings, respectively, across different Pehannel values. We observed that using both channel and patch masks together improves performance. Additionally, a higher channel-level masking proportion (i.e., larger Pehannel) improves accuracy in the Partial setting (purple) but leads to a decline in the Full setting (blue), highlighting the trade-off between these masking strategies. For DCP Alternate, we found that rp 0.75 works well across experiments.
Patch Masking Strategies in DCP. Fig. 12 compares two patch masking strategies with DCP. The first strategy, Duplicate-Sponr Spatial Mask, creates a patch mask for one channel and then duplicates the mask across all other channels (e.g., 335 48). The second approach, generates a patch mask for each channel independently, resulting in varied patch positions. We set the same masking ratio for both strategies. Our findings indicate that using different mask for each channel performs better than the duplicating one, yielding an improvement of 1.2 5.5% across three datasets.
Seappendnn--ppattte Mask,
F.6 Impact of memory tokens on reconstruction loss
Fig. 13
shows the effect of memory tokens on reconstruction loss during training. Evaluated on the So2Sat dataset, the model using memory tokens (blue line) achieves lower and smoother loss compared to those without them (red). This indicates that memory tokens may improve the model's ability to capture global information, helping case the learning process.
27

---

Accuracy on JUMP-CP CP
80
70
60:
DCP Combination (Full)
50-
DCP Combination (Partial)
0.25
0.5
0.75
0.9
Random Patch Masking Ratio Tp
Figure 10: DCP Combination with varying masking ratios Tp. train ChA-MAEVIT us- ing combined mask integrating both channel- and patch-level masking, ie., DCP Combination (Pehannel = Ppatch = 0). The blue and purple lines depict accuracy on JUMP-CP for Full and Partial channel settings, respectively, across different patch masking ratios Tp-
92.5
70
Accuracy (Partial)
Accuracy (Full)
90.0
87.5
85.0
0
0.1
0.2
0.3
0.4 0.5 0.6 0.7 0.8 0.9
Pehannel
Figure 11: DCP Alternate with different proportions of Pehannel and Ppptch- To evaluate the effects of channel- and patch- level masks ChA-MAEViT during alternating masking, we vary Pehannel while setting Ppttch masking, whereas Pchannel 0 indicates only using patch-level masking. The blue and purple lines represent accuracy on JUMP-CP for Full and Partial channel settings respectively, across different values of Pchanuel
Pehannel- Pehannel = indicates only using channel-level
+5.5
90
Duplicate-Spatil Mask
Accuracy Score
Independent Spateettt---aallll Mask (Ours)
80-
-..
70
+1.2
+2.3
60
2.1
50
Full
Partial
Full
P Partial
JUMP-CP
So2Sat
CHAMMI
Figure 12: Patch Masking Strategies in DCP. We observed that using different mask for each channel outperforms duplicating the same mask across all channels (purple), resulting in improve- ments of 1.2 5.5% across three datasets.
28

---

w/ Memory Token
0.9
w/o Memory Token
Reconstrnction Loss
0.8
0.7
0.6
0.5
1000
2000
3000
4000
5000
Training Step
Figure 13: Impact of memory tokens on the reconstruction loss. Using memory tokens helps the reconstruction task, as demonstrated by the lower reconstruction loss observed on So2Sat.
67.5
Accuracy (Prtial))
Accuracy (Full)
52.0
67.0
51.5
66.5
51.0
8
Number Decoder Blocks
Figure 14: Number of Decoder Blocks of ChA-MAEViT. We show the performance of ChA- MAEVIT when trained on So2Sat with different numbers of blocks in the decoder. We observed that using 1 - blocks gives the best performance.
F.7 Number of Decoder Blocks of ChA-MAEVIT
Fig. 14 illustrates the performance of ChA-MAEViT when trained on the So2Sat dataset with varying numbers of decoder blocks. We found that configuration of to blocks yiclds the best performance.
F88 Scaling Model Size
Table 10 compares the performance of ChA-MAEViT with the two strongest supervised baselines:
Channe ViT 30 and ViT 29). To evaluate the impact of model scaling, each model is evaluated using two backbone architectures: ViT-S (21M parameters) and ViT-B (85M parameters). Larger models exhibit improved performance, particularly on CHAMMI and JUMP-CP. Our approach consistently outperforms the baselines across all three datasets, regardless of the backbone used. In ChA-MAEVIT, we reallocate one Transformer block from the encoder to the decoder, ensuring that all models maintain the same parameter count.
29

---

Table 10: Scaling Model Size. We compare ChA-MAEVIT with the two best baseline models, ChannelViT 30) and DiChaViT 129] Each model is evaluated using two backbones: VIT-S (21M parameters) and ViT-B (85M parameters). Increasing the model size boosts performance, particularly on CHAMMI and JUMP-CP. Our method outperforms others across three datasets with both backbones.
CHAMMI
12
JUMP-CP
27
So2Sat
10
Model
Backbone
Avg score
Full
Partial
Full
Partial
ChannelViT 30
ViT-S
64.97
67.51
56.49
61.03
46.16
ViT
29
ViT-S
69.77
69.19
57.98
63.36
47.76
ChA-MAEViT (ours)
VÝT-S
74.63
90.73
68.05
67.44
52.11
ChannelVil 30
ViT-B
67.72
67.92
56.97
62.19
46.20
ViT
29
ViT-B VIT-B
70.56
69.47
58.33
63.93
47.92
ChA-MAEViT (ours)
77.09
92.22
70.24
67.67
52.27
Origiaal
Masked
Recoustraction
Original
Recosstructien
Original
Masked
Reciostraction
Original
Masked
Recunstruction
Figure 15: Visualization of original, masked, and reconstructed images CHAMMI-HPA 12 which contains 4 channels. We show the reconstruction of ChA-MAEViT, utilizing a ViT-S/16 backbone. ChA-MAEVIT demonstrates its capability to reconstruct masked channels and patches under different channel masking settings (0 3 masked channels with fixed patch masking ratio rp 75%). Gray blocks indicate fully masked channels.
F.9 Reconstruction Images
Fig. 15 shows the original, masked, and reconstructed images from the CHAMMI-HPA dataset [2] which contains four channels. Each set of images shows the original image the left, followed by the masked image created using Dynamic Phanhel-Patch Masking (with patch masking ratio Tp = 75%), and finally, the reconstructed image produced by ChA-MAEVES (utilizing ViT-S/16 backbone) on the right. The gray blocks indicate the entire channel has been masked. ChA-MAEViT demonstrates its ability leverage cross-channel dependencies it reconstruct the entire channels using available patches from the other channels.
Note that the reconstructions lack some figh--freuency details, as the primary goal of ChA-MAEVIT to learn robust, high-quality representations rather than producing detailed reconstructions. Our approach prioritizes the encoder's ability to capture high-level semantic and cross-modal relation- ships, with a deliberately lightweight decoder designed validate the encoding of essential latent information. Additionally, following prior MAEs (e.g.. [ 31 61 62)), employ the squared error (MSE) loss for reconstruction, which often results blurry outputs by averaging plausible solu- tions 62, 63 This leads to reconstructions dominated by low-frequency components, expected outcome of the design choice.
F.10
Novel Channels at Inference
For the novel channels setting, we trained ChA-MAEVIT using the first three channels (i.e., channels 0, 1, and then tested on two other channels (i.e., channels 3, 4) of JUMP-CP, One of the main challenges is that new channels do not have learned channel tokens. To address this, assume that
30

---

11.5
Accuacy (Novel channels)
11.0
10.5
10.0
9.5
32 Number unlabeled test images
Figure 16: Novel Channels at Inference. Fine-tuning the channel tokens of novel channels with unlabeled test images boosted performance from 9 9.2% (without fine-tuning, denoted as "0") to 11.6% (with fine-tuning on 640 unlabeled test images).
some unlabeled test images are available at test time. We initialize the channel tokens for the new channels randomly and then fine-tuned these channel tokens the unlabeled test images using the reconstruction loss in Eq. (2). Throughout this process, we kept the entire model frozen and only fine-tuned the newly initialized channel tokens.
In Fig- 16. we illustrate the accuracy on novel channels with varying numbers of unlabeled test images. For example, "0" indicates that no fine-tuning was performed, while "8" represents that eight test images (each containing only the two novel channels) were used fine-tune the channel tokens before testing the model on the remaining test set. We observed that fine-tuning the channel tokens with some unlabeled test images improved performance from 9.2% (without fine-tuning) to 11.6% (with fine-tuning on 640 unlabeled test images). Additionally, to make upper performance bound, we trained ChA-MAEVES on the training set of the novel channels (i.e., channels and 4). This model achieved an accuracy of 46. 1 %. indicating there is still significant performance gap, which we leave for future work.
31


Reviews:
[{"review_id": "aqc7268hqW", "reviewer": "Reviewer_4uNy", "summary": "This paper introduces ChA-MAEViT, a new self-supervised training method for vision transformers, which addresses a flaw of random patch masking when applying masked autoencoders (MAE) to multi-channel imaging. Computer vision is typically developed for fixed channel configurations, but in multi-channel imaging, channels may provide unique complementary information for each image, and have low correlation with other channels. Random patch masking, does not explicitly optimise for learning information in one channel from another, which would be expected to provide robust representations for multi-channel images.\n\nThe authors highlight four key design choices for  ChA-MAEViT that address the challenge of applying MAEs to multi-channel imaging. The first is a new masking strategy. During training a subset of channels are masked out, in addition to masked patches from otherwise unmasked channels. All tokens from masked channels, and masked patches from unmasked channels, are reconstructed by the model via the model loss function. The second contribution are memory tokens. These are much like CLS or register tokens, learned embeddings appended to the input sequence, which are reported to act as long term memory. When channels are masked during training, these mask tokens may retain information about the missing channel from previous epochs/batches, that aid the reconstruction of a masked channel from the remaining others. Thirdly, ChA-MAEViT  uses a novel final representation, produced by aggregating the CLS token with all other patches, in what is referred to as hybrid token fusion. The last design choice that the authors highlight is a decoder which is shared across all channels. This is achieved by applying channel specific tokens to each patch embedding, to make the decoder aware of which channel each patch is derived from. This is crucial for training on datasets which contain large numbers of channels, as previous MAE approaches would consider a decoder-per channel which prevents scaling to datasets with many channels. \n\nThe paper compares ChA-MAEViT against a number of baselines across three multi-channel imaging datasets. For each dataset, it demonstrates superior performance in settings in which all channels are available at inference, and in which only some channels are available at inference. \n\nA number of additional experiments are included to justify i) the inclusion of memory tokens, ii) their novel masking strategy, and iii) to demonstrate the utility of the shared channel-aware decoder.", "questions": "- Line 175 - can you please clarify what channel tokens were used in the encoder? The paper would read better if a brief description was included here. \n- Could ChA-MAEViT be applied to consider OOD channels?\n- Are there error bars that could be included in Table 2, Figure 3 and 4, to give an idea of statistical significance?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "This is a strong paper and is notable for its clarity, quality and significance. The paper is written to a high standard. The technical details are well described, and this ranks the paper high in terms of its reproducibility. There are many architectural choices introduced in this paper, and each of them are discernible due to the clear text and the care with which mathematical expressions are used and explained. \n\nThe choice of baselines are fair, in my opinion, and the superior performance of ChA-MAEViT is clear across each of the datasets, in the full and partial channel setting, as evidenced in Table 1. The quality of the experiments and the convincing performance gain with ChA-MAEViT provide the paper with significance. It remains an open challenge to develop robust representation learning for multi-channel imaging. This work has shown that random patch masking and MAEs are not purposely designed for the specific challenges associated with multi-channel data, where correlation between channels is often low. Recent works have scaled MAEs for multi-channel imaging in fluorescence microscopy, and this work points towards future work in which ChA-MAEViT is scaled, potentially leading to more robust representations for transfer learning. For these reasons, I think the community will be interested in the results presented here. \n\nWhere the paper is weaker is in the quality of some of the other experiments included. For example, while experiments are included which justify the use of channel masking, memory tokens and the shared decoder, these results, in Table 2 and Figs 3 and 4, lack measures of statistical significance. They would be improved with the addition of error bars. I also think the significance of the paper would be improved had the model been used to evaluate generalisation to OOD channels, as this is a persistent issue when training models with fluorescence microscopy data. \n\n\nOverall, this paper is of high quality and I will recommend its acceptance.", "was_revised": true, "final_justification": "I initially felt that this paper should be accepted, due to the number of novel architectural choices introduced, the clarity with which they were explained, and the quality of the experiments to demonstrate their performance. \n\nI had some minor comments about some results presented without statistical significance. The author's rebuttal has not caused any concern and has addressed my comments. \n\nThe authors have detailed responses to all other reviewers and have not changed my opinion of this paper. I therefore still recommend its acceptance.", "rebuttal": "Thank you for your encouraging feedback! We address the reviewer’s questions below.\n> Are there error bars that could be included in Table 2, Figure 3 and 4, to give an idea of statistical significance?\n\nWe report the mean and standard deviation (std) over three runs for the Table and Figures as follows for those results we were able to complete in the time constraints of the rebuttal period.\n\n**Table 2**\n| DiChaViT +        |  So2Sat (Full)       | So2Sat (Partial)      |\n|-------------------|------------------|--------------------|\n| SimCLR [10, 11]   |         64.44 ± 0.58     | 49.42 ± 0.63        |\n| SimSiam [12]                   | 64.07 ± 0.31      | 48.52 ± 0.71       |\n| iBOT [55]                       | 63.11 ± 0.32     | 47.84 ±  0.54       |\n| DINOv2 [38]                    | 63.42 ±  0.27    | 49.20 ± 0.72        |\n| MAE [23]                          | 62.88 ± 0.49     | 47.76 ± 0.62         |\n| **DCP Masking**  (ours)  | **66.02** ± 0.22   | **50.52** ± 0.45   |\n\n**Fig. 3b**\n| Number of Memory Tokens | CHAMMI Score |\n|-------------------------|--------------------|\n| 0                        | 73.42 ± 0.53     |\n| **4**                  | **74.63** ± 0.54 |\n| 8                       | 74.25 ± 0.40     |\n| 16                      | 74.53 ± 0.64     |\n| 24                      | 74.47 ± 0.68     |\n\n**Fig. 4**\n| Decoder Type                                                   | CHAMMI Score |\n|---------------------------------------------------------------|--------------------|\n| Shared Decoder w/out channel tokens             | 71.533 ± 0.47     |\n| Separate Decoders                                           | 72.339 ± 0.53     |\n| **Channel-Aware Decoder (Ours)**                  |  **74.63** ± 0.54    |\n\n\nThe relatively low standard deviations (std) across experiments suggest the results are stable, supporting the robustness of our findings. We will include the corresponding error bars in the camera-ready version.\n\n>  Could ChA-MAEViT be applied to consider OOD channels, as this is a persistent issue when training models with fluorescence microscopy data. \n\nWe completely agree that OOD channel generalization is a critical and ongoing challenge in fluorescence microscopy. In Appendix F.11 we demonstrate that ChA‑MAEViT can handle novel channels never seen during training in a preliminary experiment by adjusting its weights via learning to reconstruct the new channels at test time. Specifically, we adapt to novel channels by mapping them to existing ones using a few unlabeled examples and our self-supervised objectives, resulting in a 2.5% boost using just 640 examples.\n\n> Line 175 - can you please clarify what channel tokens were used in the encoder? The paper would read better if a brief description was included here.\n\nThe channel tokens are learnable embeddings, each representing an input channel. Since all channels share a common projection layer, these tokens play a crucial role in capturing channel-specific features. They are concatenated with the spatial patch tokens and jointly processed by the transformer encoder and decoder. We will clarify this in the camera-ready version."}, {"review_id": "Yn37i34T8E", "reviewer": "Reviewer_7Rxw", "summary": "The authors present four architecture improvements to the Masked Autoencoder (MAE) aimed at improving performance in multi channel with each channel capturing a semantically different aspect of the data. Such data such as it is frequently collected in scientific imaging, e.g. in fluorescence microscopy where different channels show different parts of the cell.\n\n* Their new masking strategy does not only independently mask patches in different channels but additionally masks some channels completely. This is meant to encourage the model to capture inter-channel correlations as opposed to just intra-channel correlations.\n\n* The authors introduce memory tokens, which are learned tokens that are meant to encode information about the data.\n\n* The authors present an improved deocoder architecture.\n\n* The authors present a new way to ustilize the learned representations in a downstream task by combining information from a CLS tokaen with a pooling operation arocss patch tokens.", "questions": "* The reconstructions in Figure 14, are missing high frequency details. Does this mean the information from the high frequency details is not encoded and thus is also not available for the classification task?\n\n* Why is the \"Dynamic Channel-Patch Masking\" called \"Dynamic\"? This implies that the masking strategy somehow dynamically adapts to the data. I don't think this is the case. Am I missing something?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths ##\n\n* I believe the development of novel architectures for multichannel data as it used in science does not get the attention it deserves. So this paper is addressing an urgent problem.\n\n* I appreciate the wide range of different datasets the authors consider, showing the the method is generally applicable and not limited to a single application.\n\n*  Even though the paper introduces a collection of new architectural features, the ablation studies investigate their efficacy separately, which is I think exactly the way it should be done to provide useful insights for the reader. \n\n* The paper is well written.\n\n* Results are substantially better than the baselines.\n\n## Weaknesses ##\n\n* This is really an engineering paper that presents improved architectures, with reasonable arguments for them but without a corresponding theory.\n\n* Reconstruction quality in Figure 15 looks quite low. I understand that the goal of the paper is not the reconstruction but the learned representation; still I am not sure if this is to be expected or if related work produces higher quality results.", "was_revised": true, "final_justification": "I think this is a very good paper. I am satisfied with the comments made by the authors and will stick to my positive rating.", "rebuttal": "Thank you for your insightful feedback! We address the reviewer’s questions below.\n\n> This is really an engineering paper that presents improved architectures, with reasonable arguments for them but without a corresponding theory\n\nDynamic Channel-Patch (DCP) Masking involves removing both patches and entire channels from multi-channel inputs, where each channel typically contains distinct and complementary information, e.g., different sensor modalities or spectral bands. When these channels are combined, they often create emergent representations, which are features that arise solely from interactions between the channels. Masking disrupts not only the information within each channel but also the crucial synergy between channels.\n\nThis concept can be formalized using mutual information. Let \\$X_{\\text{full}}$ represent the complete multi-channel input, and \\$X_{\\text{observed}}$ denote the unmasked subset. The relationship can be expressed as follows:\n\n$$\nI(Z; X_{\\text{full}}) = I(Z; X_{\\text{observed}}) + I(Z; X_{\\text{masked}} \\mid X_{\\text{observed}})\n$$\n\nWhen part of the input is masked, the conditional mutual information term \\$I(Z; X_{\\text{masked}} \\mid X_{\\text{observed}})$ becomes zero, indicating that the model loses access to the synergistic cross-channel dependencies that could otherwise be recovered from the full input.\n\nReconstruction loss, such as Mean Square Error, compels the model to predict the missing patches and channels:\n\n$$\n\\mathcal{L}\\_{\\text{rec}} = \\mathbb{E}\\_{X} \\left[ \\sum_{c \\in \\mathcal{M}} \\sum_{p} \\left \\| X\\_{c,p} - \\hat{X}_{c,p} \\right \\|_2^2 \\right]\n$$\n\nHere, \\$\\mathcal{M}$ represents the set of masked channels, and \\$p$ indexes spatial patches.\nThis objective forces the encoder to capture both intra-channel structure and inter-channel dependencies, resulting in a more robust representation that retains emergent semantics and is resilient to missing modalities during inference.  We shall update these discussions in our camera ready.\n\n\n\n> Reconstruction quality in Figure 15 looks quite low. I understand that the goal of the paper is not the reconstruction but the learned representation; still I am not sure if this is to be expected or if related work produces higher quality results.\n\nThank you for the insightful comment! Indeed, the low-fidelity reconstructions are expected and align with observations from prior work on Masked Autoencoders (MAEs), e.g., [A,B,C,D].\n\nAs the reviewer noted, the primary objective of ChA-MAEViT is to learn high-quality representations, not to generate detailed reconstructions. Our design encourages the encoder to capture high-level semantic and cross-modal dependencies, while the decoder remains intentionally lightweight to verify whether essential latent information has been encoded. Additionally, following prior MAEs (e.g., [A,B,C,D]), we employ the mean squared error (MSE) loss for reconstruction, which often results in blurry outputs by averaging plausible solutions [D, E]. This leads to reconstructions dominated by low-frequency components, an expected outcome of the design choice.\n\nIn contrast, models designed to generate high-fidelity images (e.g., [F, G, H]) typically rely on deeper, more expressive decoders. In such models, scaling the decoder is critical [H], and specialized loss functions such as perceptual loss [I] are employed to enhance visual quality.\n\n[A] Masked Autoencoders for Microscopy are Scalable Learners of Cellular Biology, Krau et al., CVPR’24\n\n[B] Masked Autoencoders Are Scalable Vision Learners, He et al., CVPR’22\n\n[C] Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery, Noman et al., CVPR’24\n\n[D] Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning, Reed et aI., CCV’23\n\n[E] Training a Task-Specific Image Reconstruction Loss, Mustafa et al., WACV’22\n\n[F] Taming Transformers for High-Resolution Image Synthesis, Esser et al., CVPR’21\n\n[G] TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation, Qu et al., CVPR’25\n\n[H] GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation, Xiong et al., ICCV’25\n\n[I] The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, Zhang et al., CVPR’18\n\n> Why is the \"Dynamic Channel-Patch Masking\" called \"Dynamic\"? This implies that the masking strategy somehow dynamically adapts to the data. I don't think this is the case. Am I missing something?\n\nThe term \"Dynamic\" refers to the varying number of channels masked during training, rather than adaptation based on the specific input data. In other words, the masking strategy dynamically adjusts the number of masked channels and patches for each image, but this adjustment is independent of the actual content of the input samples. We will clarify this in the camera-ready version to avoid confusion."}, {"review_id": "9fknxjspLj", "reviewer": "Reviewer_La1T", "summary": "This paper tackles a key limitation of applying Masked Autoencoders (MAEs) to Multi-Channel Imaging (MCI), where, unlike in standard RGB images, channels often contain complementary rather than redundant information. The authors argue that conventional patch-masking MAEs consequently fail to learn crucial cross-channel interactions in the MCI setting. They propose ChA-MAEVIT, a Vision Transformer framework that enhances cross-channel learning through a combination of four techniques: (1) a novel Dynamic Channel-Patch (DCP) Masking strategy that masks both random patches and entire channels; (2) Memory Tokens to serve as a global information store; (3) an efficient, single Channel-Aware Decoder that leverages channel-specific tokens for reconstruction; and (4) a Hybrid Token Fusion module for downstream tasks. Through extensive experiments on three diverse MCI datasets (CHAMMI, JUMP-CP, and So2Sat), the authors demonstrate that ChA-MAEVIT significantly outperforms state-of-the-art methods, particularly in scenarios with missing channels at test time.", "questions": "I'd appreciate the authors' feedback on the weakness section.", "limitations": "Please see the weakness section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper clearly articulates a fundamental mismatch between the assumptions of standard MAEs and the realities of MCI. The observation that patches in MCI primarily attend to their own channel under a standard patch-masking scheme (nicely visualized in Fig. 1) provides a compelling motivation for the work. Addressing this is significant as it can unlock the potential of powerful self-supervised learning for important scientific and industrial domains like microscopy and remote sensing.\n\n\n2) The central idea of Dynamic Channel-Patch (DCP) Masking is both simple and highly effective. Forcing the model to reconstruct not just patches but entire channels using information from other available channels is a powerful pretext task that directly encourages the learning of cross-channel dependencies.\n\n\n3) The experimental validation is thorough. The method is tested on three distinct and challenging MCI datasets with varying numbers of channels. The authors compare against a strong and relevant set of recent baselines, including the current SOTA (DiChaViT). The performance gains are substantial and consistently reported across datasets and settings.\n\nWeaknesses:\n\n1) While the DCP masking strategy is a clear and novel contribution, the other components are largely effective adaptations of existing ideas. Memory tokens are directly inspired by Register Tokens , the Channel-Aware Decoder builds upon the channel token concept from prior work like ChannelViT and DiChaViT, and the hybrid fusion module is a relatively standard approach.\n\n\n2) Hyperparameter Complexity of DCP Masking: The proposed DCP masking introduces several new hyperparameters that appear sensitive. The appendix reveals that the two main configurations (\"Combination\" and \"Alternate\") require different settings for the patch ratio (0.25 vs 0.75, respectively). Furthermore, Figure 10 shows a non-trivial trade-off between performance on full vs. partial channel settings based on the balance between patch and channel masking. This suggests that applying ChA-MAEVIT to a new dataset might require careful and potentially expensive tuning of the masking strategy.", "was_revised": true, "final_justification": "After reading the authors' response and the other reviewers' comments, I've decided to maintain my current rating.", "rebuttal": "We appreciate the reviewer’s thoughtful comments and respond to the questions below.\n\n\n\n> While the DCP masking strategy is a clear and novel contribution, the other components are largely effective adaptations of existing ideas. Memory tokens are directly inspired by Register Tokens, the Channel-Aware Decoder builds upon the channel token concept from prior work like ChannelViT and DiChaViT, and the hybrid fusion module is a relatively standard approach.\n\nThank you for recognizing the novelty of our **DCP masking** strategy. We would like to clarify the contributions of the other components as follows.\n\nFirst, as stated throughout the paper, the **memory tokens** and **Hybrid Fusion module** are adapted from existing ideas but are employed for novel purposes. Specifically, we apply them in the context of multi-channel representation learning to enhance cross-channel interactions in a masked modeling framework. This constitutes a non-trivial shift in functionality: the memory tokens serve as global carriers of multi-channel semantics throughout the encoder-decoder pipeline. Hybrid Fusion module further contributes by making use of fine-grained patch tokens learned during reconstruction to enrich the final representation. \n\nSecond, **Channel-Aware Decoder** introduces a new architectural design by integrating channel tokens with a lightweight shared decoder. This contrasts with prior work such as CA-MAE, which does not use channel tokens and instead relies on separate decoders for each channel. By enabling a unified decoding process, our approach improves scalability to a larger number of channels and yields better performance.\n\nImportantly, there is no guarantee that just combining prior ideas will work at all. For example, in Table 2 of the main paper, we show that integrating the strongest baseline (DiChaViT) with CA-MAE yields only modest improvements and still significantly falls short of ChA-MAEViT by 3.5%-12.5% across three datasets. Our work investigates the role of all the components, both existing, newly proposed, and their integration. The paper presents insights on what matters to solve open problems in the field.\n\nWe will revise the camera-ready version to clarify these points.\n\n> Hyperparameter Complexity of DCP Masking: The proposed DCP masking introduces several new hyperparameters that appear sensitive. The appendix reveals that the two main configurations (\"Combination\" and \"Alternate\") require different settings for the patch ratio (0.25 vs 0.75, respectively). \n\nFirst, we would like to point out that the same masking ratios were used for every dataset in our evaluation, i.e., these hyperparameters are stable and generalize well to new settings, making them easy to use.  That said, these two settings use different hyperparameters, but result in the same amount of information being masked out in input images. In other words, the choice of patch ratio stems from ensuring the same amount of information is masked out, rather than the sensitivity of the hyperparameters.\n\n**Alternate** masking most closely resembles the prior work like MAE and CA-MAE and, thus,  we adopt a patch masking ratio of 0.75, consistent with standard practice for MAE and CA-MAE.\n\n**Combination** differs in that we mask entire channels, and, thus, we reduce the patch ratio to 0.25 due to the additional channel-level masking. As discussed in Appendix F.6 (line 652), a patch ratio of 0.25, when used with dynamic channel masking, results in approximately 65% of the spatial content being masked on average, which is comparable to the 75% masked in Alternate masking. Thus, applying a high patch ratio (e.g., 0.75) on top of channel masking would cause excessive information loss, making reconstruction extremely difficult and unstable. It is also reasonable to argue that masking entire channels is inherently more challenging than masking individual patches, as it removes all modality-specific information rather than local details, suggesting that a slightly smaller total masking ratio of 65% is warranted in this setting.\n\n> Furthermore, Figure 10 shows a non-trivial trade-off between performance on full vs. partial channel settings based on the balance between patch and channel masking. This suggests that applying ChA-MAEVIT to a new dataset might require careful and potentially expensive tuning of the masking strategy.\n\nAs noted above ChA-MAEViT does **not** require expensive hyperparameter tuning as all experiments across all three datasets were performed using the same masking hyperparameters.  As shown in Table 4 of the main paper, both configurations significantly and consistently outperform the SOTA in Table 1 for both full and partial channel settings. Specifically, the Alternate configuration achieves boosts of 2.5-21.0%, while the Combination improves by 3.5-16.5%. This highlights the robustness and generalizability of the hyperparameters in our model across both satellite and microscopy imaging domains.\n\nFig. 10 illustrates the effectiveness of combining both channels and patch masking. It shows that using both methods leads to better performance compared to using either one alone. Additionally, it provides insights into the trade-offs associated with each type of masking, which does not mean it requires costly tuning for new datasets.  Thus, we can recommend these masking ratios as reliable defaults for applying ChA-MAEViT to new datasets."}, {"review_id": "1kQHJXdsmV", "reviewer": "Reviewer_XYwZ", "summary": "ChA-MAEViT proposes a masked autoencoding framework for Multi-Channel Imaging that alternates between masking patches and entire channels (DCP masking), uses a small set of memory tokens to capture global context, fuses [CLS] and patch embeddings via cross-attention, and decodes masked patches with channel-aware embeddings.", "questions": "1. Can the authors make their core contributions clearer, particularly with respect to prior existing work?\n\n2. What is the performance drop if a known critical channel is masked?\n\n3. How does the model handle truly unseen channel types at test time?\n\n4. Does a frozen ChA-MAEViT encoder improve segmentation or regression compared to ViT or DiChaViT backbones?", "limitations": "1. If some channels are rare during training, DCP masking may not adequately teach reconstruction, leading to overfitting to “full” channel sets.\n\n2. Many hyperparameters (mask ratios, memory token count, reconstruction lambda) appear dataset-specific, reducing out-of-the-box usability.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 1, "strengths_and_weaknesses": "1. Incremental novelty: All core ideas (channel masking, memory tokens, hybrid fusion, channel tokens) have direct antecedents (CA-MAE, DiChaViT, ViT register tokens).\n\n2. Method is overly heuristic: Alternating between patch- and channel-masking lacks a principled basis; hyperparameters (ppatch, pchannel, mask ratios, reconstruction lambda etc.) require extensive tuning.\n\n3. Limited task scope: Evaluated only on classification; no evidence it extends to segmentation or regression in MCI.\n\n4. Unclear robustness: Experiments report averages over random channel removals, but worst‐case drops when critical channels are missing are not shown.\n\n5. Unreported compute costs: No training/inference time comparisons versus simpler supervised baselines or existing MAEs.", "was_revised": false, "final_justification": "", "rebuttal": "> Incremental novelty: All core ideas (channel masking, memory tokens, hybrid fusion, channel tokens) have direct antecedents (CA-MAE, DiChaViT, ViT register tokens).\n\nThank you for the opportunity to clarify our contributions.\n\nFirst, **DCP Masking** is not derived from CA-MAE but is a novel method, as noted by reviewers **4uNy** and **La1T**. Unlike prior MAEs that assume cross-channel redundancy, DCP Masking addresses the low redundancy in multi-channel data by masking both channels and spatial patches to promote stronger cross-channel interactions. As shown in Tables 2 and 4, it outperforms CA-MAE's patch masking by 1-6% and 3-21% across three datasets with different backbones.\n\nSecond, as stated throughout the paper, **memory tokens** and **Hybrid Fusion module** are adapted from existing ideas but are for novel purposes with a non-trivial shift in functionality: enhancing cross-channel interactions in masked modeling. Memory tokens capture global cross-channel semantics, while the Hybrid Fusion module leverages reconstructed patch tokens to enrich the final representation.\n\nThird, **Channel-Aware Decoder** introduces a scalable architecture that uses a shared lightweight decoder with channel tokens, unlike prior methods (e.g., CA-MAE) that require separate decoders per channel.\n\nImportantly, there is no guarantee combining prior ideas will work at all. Indeed, Table 2 shows merging the strongest baseline (DiChaViT) with CA-MAE yields only modest gains and still lags behind ChA-MAEViT by 3.5-12.5%. Our work investigates the role of all the components, both existing, newly proposed, and their integration, offering insights on what matters to solve open problems in the field.\nWe’ll revise the camera-ready version to clarify these points.\n\n> Method is overly heuristic: Alternating between patch- and channel-masking lacks a principled basis; \n\nAlternating between masking strategies helps mitigate the learning difficulties caused by excessive information loss when channels and patches are masked simultaneously. As discussed in Appendix F.6 (line 652), a patch ratio of 0.25 when used with dynamic channel masking, results in about 65% of the spatial content being masked, comparable to the 75% masked in Alternate masking. Since each channel carries distinct semantics, masking both at once can be difficult to recover. By alternating the two, we observed more stable learning and up to a 5% performance gain on the JUMP-CP dataset (Table 4). \n\n> hyperparameters require extensive tuning and appear dataset-specific, reducing out-of-the-box usability.\n\nChA-MAEViT does **not** require extensive or dataset-specific hyperparameter tuning. All our experiments used the same hyperparameters **without performing any separate tuning** for each dataset. Specifically:\n\n**Reconstruction Lambda**: fixed 0.99 for all experiments (line 308)\n\n**DCP Masking** has three hyperparameters (mask ratio, p_patch, p_channel). As detailed in Sec. 3.1 (line 156),  we used two fixed configurations: Alternate (0.75, 0, 0) and Combination (0.25, 0.5, 0.5), across all experiments. Table 4 shows both consistently outperform the best baseline in Table 1 by 2.5-21.0% and 3.5-16.5%, showing strong generalizability.\n\n**Memory Token**: As shown in Fig. 3, a small number of tokens (e.g., 4) boosts performance, with diminishing returns beyond that. Results are stable, showing low sensitivity to this hyperparameter.\n\nThese results show that our **default settings are effective across datasets**, and can likely be used in new settings directly.\n\nWe'd like to emphasize that additional hyperparameters do not necessarily diminish the method’s out-of-the-box usability, e.g., DINOv2 has over 20 and is widely adopted for its stability and performance. Ours are stable, transferable, and require no per-dataset tuning.\n\nTo further support reproducibility and enhance understanding, we provide extensive analysis (Table 4, Fig. 3, and Appendix Figs. 9, 10, 12, 13), confirming ChA-MAEViT’s robustness and effective generalization across tasks.\n\n> Limited task scope: Evaluated only on classification; no evidence it extends to segmentation or regression in MCI. \n\nOnly JUMP-CP and So2Sat are classification tasks. In CHAMMI, methods are evaluated based on their ability to learn useful cellular representations across various axes of generalization (e.g., new plates, new cell lines), using nearest neighbor protocols based on feature sets from the CHAMMI’s authors. Thus, CHAMMI assesses **representation learning, not a classification task**. ChA-MAEViT performs well on both tasks (3-21.5% boost across three datasets), showing a broad applicability that merits publication in its own right.  In addition, this is the same set of experimental settings of prior work [1,12,13,28], suggesting the community finds these two evaluation task types are effective.\n\nThat said, to further demonstrate generalization, we evaluate segmentation performance on the 38-Cloud dataset [A], which includes four-channel (RGB + NIR) satellite images. Below are the average scores and std from three runs:\n|Model|Accuracy|IoU|Precision|Recall|F1|\n|-|-|-|-|-|-|\n|ChannelViT[13]|0.945±0.003|0.843±0.012|0.919±0.015|0.911±0.004|0.915±0.003|\n|DiChaViT[28]|0.951±0.004|0.857±0.011|0.924±0.007|0.922±0.006|0.923±0.003|\n|CA-MAE[1]|0.946±0.002|0.845±0.003|0.917±0.005|0.916±0.003|0.916±0.002|\n|DiChaViT[28]+CA-MAE[1]|0.959±0.001|0.886±0.004|0.939±0.005|0.943±0.004|0.941±0.002|\n|**ChA-MAEViT**(ours)|**0.964**±0.001|**0.894**±0.002|**0.943**±0.002|**0.945**±0.001|**0.944**±0.001|\n\nChA-MAEViT outperforms all baselines across all evaluation metrics, demonstrating its effectiveness in enhancing multi-channel representation learning.\n\n[A] Cloud-Net: An End-To-End Cloud Detection Algorithm for Landsat 8 Imagery, Mohajerani et al., IGARSS’19\n\n> Unclear robustness: Experiments report averages over random channel removals, but worst‐case drops when critical channels are missing are not shown. \n\nResults in Table 1 follow established evaluation protocols from prior work, **not random channel removals**. CHAMMI’s varying channels stem naturally from differences in its three source datasets, i.e., we do not remove any channels. For So2Sat and JUMP-CP, we follow the established evaluation protocols of [2,3]. This can result in some critical channels never being masked, which is why we provided a more complete evaluation.\n\nSpecifically, in Table 3, we report the average over **all possible channel combinations** as stated in its caption and L279, not random masking. E.g., column \"4\" shows the mean and std across all 70 four-channel combinations (8C4 = 70). Due to space constraints, only the mean and std values are shown in the main paper. However, detailed result in Appendix Table 8 **does show the worst-case drop**, when channel 3 is removed. Even in this worst-case setting, our method achieves the best result, outperforming the SOTA by 23%.\n\nTo leave no further doubt, we extend Table 3 by including the minimum performance reported across all channel combinations below:\n|Method|8 channels|7 channels|6 channels|5 channels|4 channels|3 channels|2 channels|1 channel|\n|-|-|-|-|-|-|-|-|-|\n|DiChaViT[28]–avg|69.19±0.26|61.91±0.28|54.49±0.23|46.35±0.18|38.00±0.15|30.09±0.14|23.97±0.10|20.90±0.04|\n|ChA-MAEViT(ours)–avg|90.73±0.14|83.36±0.28|74.55±0.33|63.46±0.35|50.85±0.29|38.13±0.19|27.62±0.06|21.59±0.04|\n|DiChaViT[28]–min|69.19±0.26|38.80±0.45|29.39±0.25|23.49±0.21|20.42±0.26|19.76±0.03|19.75±0.02|19.73±0.01|\n|ChA-MAEViT(ours)–min|90.73±0.14|61.79±0.53|46.71±1.21|36.46±0.62|26.67±0.76|22.05±0.28|20.61±0.06|19.95±0.05|\n\n“avg” indicates the average across all possible combinations, while “min” represents the worst-case performance. Note that different from Table 3, here we report the average of three runs and the std for model variance.  As seen above, our approach consistently outperforms the prior state-of-the-art.\n\n> Unreported compute costs: No training/inference time comparisons versus simpler supervised baselines or existing MAEs.\n\nWe report **training runtimes** in Appendix C (line 561), and **FLOPs comparisons** in Figure 5 of the Appendix. Specifically, on JUMP-CP, DINOv2 takes 6 days, SimCLR 2 days, and CAMAE/ChA-MAEViT 1 day. Figure 5 shows ChA-MAEViT has the lowest FLOPs, about 1/6 of DINOv2, while outperforming it. \n\n> How does the model handle truly unseen channel types at test time?\n\nIn Appendix F.11, we show that ChA-MAEViT can handle novel channels unseen during training. We initialize new channel tokens for these channels and fine-tune only these tokens using a small set of unlabeled test images with DCP, while freezing the rest of the model. This approach improves performance as more unlabeled images are used (Fig. 16).\n\n> If some channels are rare during training, DCP masking may not adequately teach reconstruction, leading to overfitting to “full” channel sets.\n\nRare channels do **not** lead to overfitting to full channel settings. DCP masking consistently uses a subset of channels during training (line 138), forcing the model to be robust to missing modalities and mitigating reliance on any specific channel configuration. Also, DCP masking outperforms patch masking by 4% (Table 4) on CHAMMI, in which some channels are always missing due to the nature of the dataset, meaning it never sees “full” channel sets.\nRare channels, instead, reflect the natural data distribution and do **not** imply overfitting. We demonstrate robustness under these conditions in CHAMMI, where the “membrane” channel occurs only 31% as often as “nucleus.” Yet, our model outperforms the best baseline by 2% on images with the “membrane” channel (Table 7, Appendix), showing strong generalization under uneven channel distributions. Additionally, techniques like channel-aware sampling, oversampling, or importance-weighted loss can be easily integrated into our method to enhance learning if needed."}]

Author rebuttal:

Meta-review: The paper proposes ChA-MAEViT, which augments MAEs for multi-channel imaging via dynamic channel-patch masking, memory tokens, hybrid token fusion, and a channel-aware shared decoder. Reviewers are largely positive (two Accepts, one BA), highlighting clear writing, strong results across CHAMMI, JUMP-CP, and So2Sat, and practical value when channels are missing; reported gains over prior MCI-ViTs are sizable. One reviewer remains concerned about conceptual incrementality and heuristic design, but the authors provided additional analyses that improve confidence: fixed default hyperparameters used across datasets (no per-dataset tuning) with consistent improvements, plus added experiments demonstrating segmentation performance and robustness under worst-case channel removal. Taken together, the method is sound, broadly applicable, and empirically strong; the AC recommends accept.
Decision: Accept (poster)