Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers
Authors: Chau Pham
Bryan A. Plummer
Venue: NeurIPS 2024 poster (2024)
Keywords: vision transformer
representation learning
multi-channel imaging
Primary area: machine_vision
TL;DR: We enhance Vision Transformers for multi-channel imaging by improving diverse representations.

Abstract: Multi-Channel Imaging (MCI) contains an array of challenges for encoding useful feature representations not present in traditional images. For example, images from two different satellites may both contain RGB channels, but the remaining channels can be different for each imaging source. Thus, MCI models must support a variety of channel configurations at test time. Recent work has extended traditional visual encoders for MCI, such as Vision Transformers (ViT), by supplementing pixel information with an encoding representing the channel configuration. However, these methods treat each channel equally, i.e., they do not consider the unique properties of each channel type, which can result in needless and potentially harmful redundancies in the learned features. For example, if RGB channels are always present, the other channels can focus on extracting information that cannot be captured by the RGB channels. To this end, we propose DiChaViT, which aims to enhance the diversity in the learned features of MCI-ViT models. This is achieved through a novel channel sampling strategy that encourages the selection of more distinct channel sets for training. Additionally, we employ regularization and initialization techniques to increase the likelihood that new information is learned from each channel. Many of our improvements are architecture agnostic and can be incorporated into new architectures as they are developed. Experiments on both satellite and cell microscopy datasets, CHAMMI, JUMP-CP, and So2Sat, report DiChaViT yields a 1.5 - 5.0% gain over the state-of-the-art. Our code is publicly available at https://github.com/chaudatascience/diverse_channel_vit.

Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers
Chau Pham Boston University Boston, MA chaupham@bu. edu
Bryan A. Plummer Boston University Boston, MA
uplum@bu.edi
Abstract
Multi-Channel Imaging (MCT) contains an array of challenges for encoding useful feature representations present in traditional images. For example, images from two different satellites may both contain RGB channels, but the remaining channels can be different for each imaging source. Thus, MCI models must support variety of channel configurations at test time. Recent work has extended traditional visual encoders for MCI, such Vision Transformers (ViT), by supplementing pixel in- formation with an encoding representing the channel configuration. However, these methods treat each channel equally, i.e., they do not consider the unique properties of each channel type, which can result in needless and potentially harmful redun- dancies in the learned features. For example, if RGB channels always present, the other channels can focus on extracting information that cannot be captured by the RGB channels. To this end, we propose DiCha ViT, which aims to enhance the diversity in the learned features of MCI-ViT models. This is achieved through novel channel sampling strategy that encourages the selection of more distinct channel sets for training. Additionally, we employ regularization and initialization techniques to increase the likelihood that new information is learned from each channel. Many of our improvements are architecture agnostic and can be incorpo- rated into new architectures as they are developed. Experiments on both satellite and cell microscopy datasets, CHAMMI, JUMP-CP, and So2Sat, report DiChaVii yields a 1.5 5.0% gain over the tate-off-hhe-att Our code publicly available at https: / / githhb..b... com chw.datasccaacaccaccccaaaa wwweccieennceeeceeeeee wiinnc//dvieeeeeeeeoe chapecchaanlecceeeeeeeeeeee vit
Introduction
Most visual encoders assume they are provided with a fixed-channel representation as input (e.g., they take RGB inputs input at train and test time) [1110]. However, many applications find variety of imaging techniques beyond just the traditional RGB channels beneficial. For example, satellite images or sensors onboard robot often contain infrared camera in addition traditional RGB, and microscopes can also host a significant range of potential imaging channels 17 Thus, Multi-Channel Imaging (MCT) models aim to learn good feature representations from datasets with heterogeneous channels, where the number and type of channels can vary for each input at test time. Training a model that is robust to changes in channel configurations can save time and resources as only a single model needs to be learned, while also helping to prevent overfitting in small datasets through transfer learning [4]] Prior work proposed methods to make MCI models robust missing channels by randomly masking them during training [18] As shown in Fig. [a] left and (b) top, this results redundancies being learned across channels during training rather than encoding new information. A consequence of this repetition is a model focused on learning strong cues that are easy identify, making less capable of learning unique and/or challenging cues within each channel.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

merooutules
microtubutes
protein ChannelVIT
proteh
rudius
nucwia
monsutoll
prided
nasless
midretokaied
Prodes
microlubuies protein
DiChaVIT
nucluus
er
ChannelViT
DiChaVIT
(b) Aggregated patch token attention scores by channel
(a) Mutual information of channel tokens
Figure 1: Comparison of the redundant information learned by different models the HPA
dataset in CHAMMI (14). (a) Measures the mutual information between the channel tokens, which captured the configuration of channels in image. Note we gray out the diagonal for better visualization. We find ChannelViT tokens have high mutual information, which suggests significant redundancy exists across channels (34 35 In contrast, DiCha ViT has little mutual information as
each channel is encouraged to learn different features. (b) We compute attention scores of the [CLS] token to the patch tokens in the penultimate layers and aggregate them by channel. ChannelVil (top) relies on certain channels (e.g.. microtubules and nucleus) to make predictions and less on other channels (e.g.. protein and er). In contrast, DiChaViT demonstrates more evenly distributed attention scores across channels, suggesting that each channel contributes more to the model's predictions.
To address this limitation, we propose Diverse Channel Vision Transformer (DiCha ViT) that aims to balance the robustness in different MCI configurations, which may cause redundancy, with a need to learning diverse and informative features. First, we include Channel Diversification Loss (CDL), a regularization term that encourages a special channel token, which represents the presence of a channel in the input data, be distinct from the other channel tokens. As shown in Fig. [ right, this reduces repeated information across our model's 's channels. However, this can still result in similar features being encoded for each image patch. Thus, our Token Diversification Loss (TDL) aims to directly diversify the features learned for each patch token as shown in the bottom of Fig. bb) by encouraging that each patch token is orthogonal to the others. Finally, rather than a uniform random channel masking strategy as used in prior work [18 19], we introduce Diverse Channel Sampling (DCS), which we select channels based on their dissimilarity, further promoting feature diversity. We observe that promoting more diverse representation enables each channel to contribute more to the final prediction, leading to a performance boost of up to 5.0% in downstream MCI tasks. Fig. 2 provides an overview of our approach.
The work that is closest in spirit ours are methods that are designed learn disentangled rep- resentations 27, e.g., learning features aligned to a given set of attributes [28)30] These methods have shown trade-off between the strength of the disentanglement and the downstream tasks performance (3). This is due, in part, to the fact that many attributes these methods aim to disentangle are correlated with each other, making it challenging to know what features relate individually to each attribute. However, unlike these tasks, MCI methods do not focus only on disentangling features across channels. Instead, they must capture some redundant information to be robust to missing channels while simultaneously learning features that may only arise in subset (or even a single) channel. In other words, in MCI some redundancy is desirable across channels even we could learn perfectly disentangled representations. In addition, many methods in disentangled representation learning assume the attributes to separate are labeled, but there are no labeled attributes in MCI. Instead, DiCha ViT must automatically decide what to capture in multiple channels while still learning important -speciff information.
We summarize our contributions below:
We propose DiCha ViT solution enhance feature diversity and robustness in MCI-ViTs, boosting classification accuracy by 1.5 5.0% over the state-of-the-art on three diverse MCI
datasets: CHAMMI [14) JUMP-CP [12), and So2Sat [17].
We introduce new channel sampling strategy to encourage the selection of more distinct channel sets during training, thereby enhancing feature diversity in MCI models.
We introduce regularization and initialization techniques that better balance robustness to different
configurations MCI and facilitate learning diverse and informative features.

---

Patch token
paby Positional embeed
Chanael token
DiChaViT (ours)
(a) Channel
Classifior
Transformer Encoder
Diversification
Separate channel Wiokens
Loss
(b) Token
CLS
phitig
diversify takens scross charnely
Diversification
Loss
diversify tokens within channel
Shared Patch Projection
(c) Diverse Channel Sampling
Inpul (mullt--chunnl)
palch
Sample a
diteciilar chamnels
Figure 2: An overview of DiChaVil We introduce two regularization methods the features and a channel sampling strategy to promote diversity in feature representations. We apply (a) Channel Diversification Loss (CDL) (Sec. 3.1 for channel tokens and (b) Token Diversification Loss (TDL) (Sec. 3.2) on the patch tokens ). Additionally, we (c) sample a subset of dissimilar channels using Diverse Channel Sampling (DCS) (Sec. 3.3
2 Related Work
Convolutional-based models for multi-channel imaging. Researchers have been developing convolutional- based models to keep pace with the evolving landscape of multi-channel imaging data. Bhattacharyya el al. [36] introduced IRFacExNet, which utilizes depth-wies convolutions to merge channel-wise features from infrared thermal images. Jiang et al. [37] introduced double-channel CNN that takes into account the correlation between input channels in aerial images. This approach employs separate sub-network for each group of channels and then performs feature fusion to aggregate features across channels. Siegismund et al. (38] presented DCMIX to work with images with many channels based on imaging blending concepts. While these methods can be used for MCI, they are not designed to work on varying input channels. In recent study, Chen et al. [14] introduced and adapted channel-adaptive models based Depthwise convolutions, TemplateMixxng (3941] 41], and HyperNets [42]. These models incorporate their adaptive interface in the first layer of an otherwise shared ConvNeXt model [8]. While these methods provide strong baseline, they find settings where some channels are missing during inference challenging. In our work, we aim to improve MCI model robustness by improving the diversity of learned features.
Vision Transformers for multi-channel imaging. Vision transformers (ViT) 43 have natural advantages when dealing with multiple channels, especially when the number of channels varies. ViTs treat image modeling as tooenccc----eeeeeeee problems, allowing them to be flcxible handling different numbers of image tokens. Nguyen et al. [44] introduced variable tokenization and variable aggregation, in which they divided each input channel independently into patches and then aggregated the patch features across channels using learnable queries. Tarasiou al. [45] proposed TSVIT, which incorporates a tokenization scheme and temporal position encodings to process Satellite Image Time Series. In relevant work, Zhou el al. [46) introduced FAN, a channel reweighting design aimed at adjusting channel features based on the observation that some channels capture more significant information than others. In the medical domain, Hatamizadeh al. [47 proposed UNETR that utilized a transformer encoder followed by a skip-connected decoder for 3-D medical image segmentation. Recently, Bao al. [18] proposed ChannelViT that processes each input channel independently via a shared linear projection and incorporates a learnable channel embedding for preserving channel-speeific features. In addition, the authors proposed Hierarchical Channel Sampling (HCS), a regularization technique applied to the input channels to boost robustness and reduce training time. ChannelViT outperforms standard ViTs classification tasks and demonstrates its generalization ability when only subset of the trained channels is available during inference. In similar work, Bourriez el al. [5] introduced ChAda- ViT, a channel adaptive attention technique for handling heterogeneous microscope images. However, these methods do not adequately model the unique properties of each channel type, resulting harmful redundancies, whereas we boost the diversity of features across channels to enhance the robustness of MCI-VE models.

---

3 Encouraging Diverse Representations in multi-channel ViTs
Given a multi-channel image (MCT) containing channels G; E Cx, our goal is to train a model M that takes our input image X as input to make its predictions. Following [5 18 we consider the MCI setting where has seen all the channels we expect to see during inference, i.e., Cx C Cm- We leave the exploration of handling novel channels during inference for future work, as it presents significant challenges, including establishing meaningful connections between existing and new channels, and identifying informative channel weights in the presence of domain shifts. In our setting, since we do not know what Cx we may see during inference, prior work has focused primarily on exploring methods that are robust to different choices of Cx by encouraging M to redundancies across channels (e.g., [4 18) Specifically, they begin with a base ViT encoder [48] that uses each channel-specific image patch p; as input. Each image patch is passed through shared patch projection layer and concatenated with its corresponding channel token ch;. Hierarchical Channel Sampling (HCS) [88] encourages robustness to missing channels by randomly masking some channels during training to ensure key information can be captured in multiple channels. However, as noted in the Introduction, this can be harmful when M does not balance this repetitive feature learning also capture distinctive channel-specifer information.
As illustrated in Fig. D DiChaViT aims to better balance repetitive and distinct feature learning through three major components. First, we use a Channel Diversification Loss (CDL) to learn diverse representations to help prevent feature collapse in the channel tokens (Sec. (.1). Second, our Token Diversification Loss (TDL) encourages patch tokens to also learn distinct features (Sec. (32]. Finally, Diverse Channel Sampling (DCS) promotes robustness missing channels while also encouraging that new features are also learned during training (Sec. (.3.. These components enable our approach to balance repetitive and channel-specific feature learning (overview in Fig. 2
3.1 Enhancing channel token separation
Recall that in Fig. [a), learned channel tokens ch; from prior work show high mutual information,
indicating these tokens are not well-separated. Following [41 49-51), we partly mitigate this issue by
replacing the random initialization of ch; used by prior work [15 18 with an orthogonal initialization.
To further encourage the diversity in the features, we introduce Channel Diversification Loss (CDL) for increased separation between the channel tokens (Fig- (a)). Inspired by ProxyNCA ++ [52].
the idea is to use a learnable vector (i.e., an orthogonally initialized channel anchor) to represent each channel in the input image during training. We promote diversity in the channel tokens by pulling channel features toward their corresponding anchors while pushing them away from all other anchors. A key benefit of this approach is that the anchors prevent channel tokens from collapsing while still allowing for flexibility learning useful representations.
Formally, we denote A as the set of all channel anchors, LCDL, as the temperature, and || |la as the L2-Norm. We start by initializing the channel tokens ch;, and their channel anchors orthogonally. Then, we apply CDL as follows:
(dd
ch: Tch,ll, :
g(ch.)
exp
[I
İODL
LCDL = = log
(I)
Eg(a)EA exp
(-d TTebella
ch. o(a) Tata)
TCDL
where g(ch:) is function that returns a corresponding channel anchor for channel token ch,, and d(ch;,g g()) is the squared Euclidean distance between channel token ch, and an anchor. In Eq. [ the numerator calculates the distance of channel token to its anchor, while the denominator computes all these distance pairs of the channel token to all the channel anchors. When the temperature value LCDL. is set 1, we get a standard Softmax function. Lowering the temperature can lead more focused and sharp probability distribution, but we found that the results are not very sensitive to the value of TCDL. Thus, we simply use a fixed temperature tCDL of 1/14 0.07.
3.2 Enhancing feature diversity for patch tokens
use shared linear projection to extract features independently from each input channel in the image rather than using separate projections for each channel. With the shared projection, only the common features across channels are retained,
MCI-VIT models like ChannelViT 18 ChAda-Vii VIT 1.

---

while other channel-specinn information is filtered out, which helps to reduce overfitting, However, this design can also produce similar representations for all patch tokens. This is not ideal because each patch may contaín unique information that would be ignored. In our approach, we also leverage this shared projection, but we enhance it with Token Diversification Loss (TDL), regularization applied to the patch token features to enhance the diversity of features learned by each patch in the input image (see Fig. 22bb for an overview). Specifically, we enforce an orthogonality constraint on the tokens to ensure that each token is orthogonal to the others. Additionally, we take into account the token type information to differentiate between tokens from the same channels and across channels. The main idea is to make features from different channels more distinct while allowing for certain level of similarity among features within the same channel.
Let P: be the input patch at position i, and W proj be the shared linear projection at the first layer. We denote t; W proj Pi as the patch feature token of p.. T {t²} i,,,,.. as the set containing all patch feature tokens in the input image, and h(t;) as function that returns the corresponding channel for input patch p.. We devise a unified loss function for each input image as follows:
i,2....
C.
Σ
(ts. tz)
(2)
N₂
La =
Σ
(t,,
(3)
Na
((t))tht
LTDL = A. 14,1 + Xa |Ca
(4)
where ( represents the cosine similarity, |- denotes an absolute value, and Ns. Na are the numbers of patch token pairs in the two equations respectively. Eq. calculates the average cosine similarity of all feature token pairs the same channels, while Eq calculates the average of all feature token pairs from different channels. The two losses are combined with weights A and Xa to balance the constraint of tokens belonging to the same channels (first term) and tokens belonging to different channels (second term), to form the final loss LTDL in Eq. + Our goal is to encourage each patch token to be orthogonal to each other to promote the diversity of patch tokens.
3.3 Diverse Channel Sampling (DCS)
Bao et al. 18 introduced HCS to reduce the training time and improve the robustness of the model. The main concept is to randomly drop some input channels and train the model only the remaining channels. In the same spirit, we propose a novel method, Diverse Channel Sampling (DCS), to sample more diverse subset of channels during training (Fig. ()). Similar to HCS, we start by randomly sampling a number k, which is the size of subset of channels to train on. However, while HCS samples channels randomly, DCS first samples an anchor channel cj. Then, we select other k- - 1 channels that are dissimilar to the anchor channel. This idea shares similarity with Channel DropBlock [53] where a set of similar channels in CNN layer is masked out to disrupt co- adapted features. However, instead of keeping fixed number of feature map channels as in Channel DropBlock, DCS selects a flexible number of input channels for each sampling. The procedure of DCS is outlined in Algorithm
In practice, Algorithm [ can be applied batch of images for faster sampling. We use channel token Ch. to represent the channel feature fi. Refer to Sec. [4d Tab. for more discussion on choices of f. The temperature IDCS controls the sharpness of the probability distribution. With large tDC,, DCS reduces to HCS, while with a small tpcs, DCS selects a random subset of channels that are the least similar to the anchor channel.
3.4 Training Objective
The final loss consists of the primary loss for the specific task (e.g., cross-entropy for classification), Channel Diversification Loss (CDL) applied to channel tokens, and Token Diversification Loss (TDL) used on patch tokens. These terms work together to promote diversity in channel and patch token features, resulting in more robust model, as shown in Eq. 5
Linal = Lrask + ACDL LODL + LTDL
(5)
where ACDL is a weight to balance CDL. Note that TDL is balanced by X and A in Eq. 4
5

---

Algorithm 1: Diverse Channel Sampling (DCS)
Input Image X with channels 61, Cyn
Channel feature ft for each input channel Ci
Temperature fDcs
Sample random variable & uniformly from the set m...
Sample an anchor channel eą uniformly from all m channels
3 Compute the cosine similarity between channel Cg and the other m 1 channels:
s s= [[f,. fi) ....]. Vi o k (s Rm--)
Convert 1
probability using softmax with temperature tpas: p softmax s) s)/tDcs
(p E Rm
Sample A: distinct channels from m - 1 channels with probability
Combine the k- channels with channel cą to create set of & sampled channels.
Output: Image X with only l sampled channels
4 Experiments
4.1 Experimental Setup
Baseline methods. We adopt the following baseline methods.
DepthwiseViT 14 utilizes depthwise convolution layer independently filter each input channel. The resulting features are averaged to create new feature representation, which is then fed into ViT backbone.
TemplateMixing ViT 43 generates weights for each channel by learning linear combination of shared, learnable parameter templates. These weights are formed into patch project layer, followed by a ViT backhone.
HyperNet VIT 42 employs neural network (e.g., MLP) to independently generate weights for each channel, which are then concatenated to form patch projection layer. This patch projection layer subsequently used in a ViT backbone.
ChAda-ViT [15 uses shared projection layer to extract features from each channel separately, then feeds these tokens, together with their corresponding positional embeddings and channel embeddings, into VIT backbone.
ChannelViT [8] is the same general architecture as ChAda- ViT, but also employs Hierarchical Channel Sampling (HCS) during training.
Implementation details. As HCS proves robust in multi-channel imaging 18 we incorporate this technique for DepthwiseViT, TemplateMixxng in these adaptive baselines used by Chen et al. 14 For ChannelViT and ChAda- ViT, due to their similarity (primarily a difference in whether HCS included), we use the implementation from [18] for both m All baselines utilize ViT small architecture (21M parameters) implemented in (55) to train the models, minimizing cross-entropy loss on JUMP- CP and So2Sat, and proxy loss on CHAMMI For the learning rate, we
and HyperNet VTT to ensure fair comparison
DINOv2 [54] as the backbone 3] We use AdamW optimizer
scheduler with linear warmup and cosine decay. Refer to Appendix Sec. Sec.[Afor details.
Metrics. We evaluated the methods by calculating their top-1 classification accuracy on the So2Sat (17 and JUMP-CP [12] datasets. For CHAMMI [14), we used the evaluation coderIroovide by the authors, in which 1-Nearest Neighbour classifier used to predict the macro-average Fl-score for each task separately. We report the average score on WTC and HPA, and present the detailed results in Tab. of the Appendix.
4.2 Datasets
CHAMMI [14] consists of varying-channel channel images from three sources: WTC-11 11 hiPSC dataset (WTC-11, three channels), Human Protein Atlas (HPA, four channels), and Cell Painting datasets
http:://hhhh..cooeemaaaaaaaaaeed httpes///gthbbbooo////stttt//ttp

---

Table 1: Comparison of test accuracy of channel adaptive models. "Full" refers to inference on all channels, while "Partial" means testing on a subset of channels (Sentinel-1 channels for So2Sat, fluorescence channels for JUMP-CP). We find our model outperforms other baselines, with a 5.0% boost on CHAMMI and a 1.5- 2.5% point improvement on JUMP-CP and So2Sat.
CHAMMI [4]
JUMP-CP 12]
So2Sat
17
Model
Avg score
Full
Partial
Full
Partial
HyperNetViI 42
54.54
47.07
42.43
60.73
41.88
DepthwiseViT 14
60.94
49.86
44.98
60.41
43,41
Mexpla ViT [39
40
57.02
52.48
43.85
55.86
37.28
ChAda- ViT
[5]
63.88
65.03
42.15
56.98
12.38
ChannelViT
[18
64.90
67.51
56.49
61.03
46.16
DiCha ViT (ours)
69.68
69.19
57.98
63.36
47.76
Table 2: Test accuracy of DiCha VIT and ChannelViT on partial channels of JUMP-CP [12]. Each column represents mean for all combinations when tested on partial channels. For example, column "7" indicates testing on 7 out of 8 channels, and, thus, the reported variance is due to the pres- ence absence of channel. See to Tab. in the Appendix for detailed results for each combination for column "7" with model variance. DiChaViT consistently exhibits improved robustness in the presence of missing channels during inference.
Number of channels for evaluation
Method
5
3
ChannelViT 18 67.51 60.3649.1 52.74 44.89+132 36.88+123 29.36193 23.70=5.0 20.78
DiCha (ours) 69.19 61.91493 54.49 46.35+13.4 38.00 30.09 23.97=4.9 20.90+1.6
(CP, five channels). The three sub-datasets contain total of 220K microscopy images, of which 100K images are for training and the rest for testing across various tasks. The models are trained to learn feature representation and then evaluated on domain generalization tasks.
JUMP-CP [1] comprises images and profiles of cells that were individually perturbed using chemical and genetic methods. Our experiments focus the compound perturbation plate BR001 16991, which contains 127K training images, 45K validation images, and 45K test images. Each image has eight channels, with the first five being fluorescence and the remaining three containing brightfield information. The dataset consists of 161 classes, including 160 perturbations and a control treatment.
So2Sat [17 contains synthetic aperture radar and multispectral optical image patches from remote sensing satellites. Each image the dataset has 18 channels, of which eight Sentinel-1 and 10 Sentinel-2 channels. The dataset consists of 17 classes, each representing distinct climate zone. We use the city-split version of the dataset, which includes 352K training images and 24K test images.
4.3 Results
Tab. shows that DiCha ViT outperforms the state-of-the-art ChannelViT by up to 5.0% points on all three datasets: CHAMMI [14, JUMP-CP [12, and So2Sat [17). For JUMP-CP and So2Sat, we consider two scenarios: tested on all training channels (denoted as "Full") and tested on subset of channels (denoted as "Partial"). In the full channels setting, our model shows a 1.5 - 2.5% point improvement compared with other basclines JUMP-CP and So2Sat. When tested on partial channels, DiCha ViT demonstrates its robustness by achieving a 1.5% improvement compared with the baselines. This demonstrates that diversifying feature representations in MCI-VIT models boosts both performance and robustness.
Tab. 2 presents a detailed evaluation of DiC and the best baseline model, ChannelViT, when tested on partial channels of the JUMP-CP dataset (with a total of eight channels). For the partial channel evaluation, we exclude some of the channels that the models were trained on and only test the model on the remaining channels. Then, we calculate the average accuracy across all combinations, e.g., testing on seven channels, shown in column "7", involves averaging the results of Cg 8
7

---

Table 3: Model ablations of DiChaViT. Removing any component in DiChaViT has a negative impact on overall performance, with significant decreases observed on the Partial setting when DCS removed. Including all components improves performance across all three datasets.
CHAMMI [14]
JUMP-CP [22
So2Sat 17
Model
Avg score
Full
Partial
Full
Partial
DiCha ViT
69.66
69.19
57.98
63.36
47.76
w/o CDL
68.07
67.66
56.87
62.20
45.74
w/o TDL
67.61
68.12
56.62
62.39
46.87
w/o DCS
65.32
66.03
42.37
59.20
17.88
Figure 3:
Performance of
70
Wiout Channel Takenss
DiCha VIT JUMP-CP and CHAMMI with and without
Accuracy
65
Channel Tökens
Channel Tokens Ortha Init.
channel tokens. Using channel to-
55-
kens with orthogonal initialization (green) improves performance.
JUMP CP
CHAMMI
combinations (refer to Tab. in the Appendix for detailed results). Our findings consistently show that ViT demonstrates improved robustness when some input channels missing-
To provide more insight into the contribution of each component of DiChaVil: Tab, presents the model's performance when a component is removed. The results highlight the critical role of the DCS component, as its removal has the most detrimental effect on performance, particularly in the Partial setting, with decrease of 16% and 30% points JUMP-CP and So2Sat, respectively. The absence CDL and TDL results in similar performance drops across all datasets. The highest scores integrated, indicating that each component plays crucial role
achieved when all components in the model's design. Refer to Tab. g in the Appendix for comprehensive analysis.
4.4 Analysis and Discussion
4.4.1 Role of Channel Tokens in MCI-VIT Models
The role of channel tokens. In MCI-VTT models such as ChannelViT [18] and ChAda-Vii [15] channel tokens play a crucial role in learning channel-specifie features, particularly when dealing with multiple channels where each contains unique information. To assess the impact of channel tokens, we compared the performance of DiCha ViT on JUMP-CP and CHAMMI with (orange bars) and without channel tokens (blue bars), shown in Fig. 3 The results indicate that DiCha ViT demonstrates significant improvements with channel tokens, resulting in 8.0% and 15.0% point increases JUMP-CP and CHAMMI, respectively, highlighting their importance.
Orthogonal initialization of channel tokens boosts performance. As shown in Fig. 3 using orthogonal initialization (green) provides 1.0% gain on JUMP-CL CP and CHAMMI. This may suggest that by initializing the weights orthogonally, the model can more effectively capture diverse patterns within the data, resulting in boosting its overall performance.
4.4.2 Ablation on Feature Diversification Losses (CDL and TDL)
Impact of ACDL 5 in CDL. Fig, 4(a) and (b) show the performance of DiCha VIT (mean and std) across different values of ACDL on So2Sat and CHAMMI datasets. We can observe that selecting value that is too large is not beneficial to the performance. It is worth finding suitable value for ACDL. On the So2Sat, the best performance is achieved with \CDL = 0.001, while the suitable value for CHAMMI is 0.1.
Ablation on TDL (Eq. 4). Fig. C) reports the performance of our model across different ratios of Xa and X, in TDL, We set fixed value of X, at 0.05 and vary Ag- We observe that using larger da compared with X2 leads to better performance for DiChaViT. This suggests that knowing which channel token comes from, ie., the same or different channel, is necessary. The results indicate

---

70
69
Accuracy
63.0
Accuracy
Accuracy
68
63.0
62.5
67
62.5
62.0
66
61.5
62.0
0.001
0.01
0.1
0
0.01
0.1
1
0.5
8
ACDL
ĀCDL
Aa/As
(a) So2Sat
(b) CHAMMI
(c) TDL
Figure 4: Impact of CDL and TDL on DiChaViT" performance. (a) (b) We demonstrate the average top-1 test accuracy and standard deviation over three runs for different values of ACDI. on So2Sat and CHAMMI. (e) Performance with different ratios of Xa and )g TDL on So2Sat.
Table Ablation on the two components of TDL. Only L, indicates using only within channel tokens (L.e,, Xa= = 0), while Only La indicates the use of only tokens from different channels in Eq. Incorporating both components in TDL gives the best performance.
So2Sat
CHAMMI
Only C, Only La Both
61.43
65.47
62.50
68.
63.36
69.66
Table 5: Different choices of channel feature f in DCS [Algorithm]] We compare the performance when using the channel tokens (ch:) and patch tokens (i.e., image patches after passing through the projection layer) to compute the similarity score for sampling.
So2Sat [17
CHAMMI [14]
Patch tokens
63,00
65.57
Channel tokens
63.36
69.68
Table Effect temperature tDss on DCS (Algorithm 1). The first column (= 0) indicates the use of very small value of (DCS, which is reduced to selecting the lowest similarity channels. The last column indicates a large value (Dcs, which is reduced to HCS [18. Using IDcs 0.1 obtain
the best results on So2Sat and CHAMMI datasets.
Temperature /DCS
RJ 0
0.001
0.01
0.1
0.2
HCS
So2Sat 17
62.51
63.21
63.30
63.36
61.92
62.15
CHAMMI 67.22
66.91
68.96
69.66
66.07
66.30
imposing stricter constraints on tokens from different channels compared with tokens from the same channel obtains the best performance. Tab. 4 shows the impact of each component in TDL. We see that considering only tokens within the same channels (denoted by "Only Z,") is insufficient, resulting in significant drop in performance. In contrast, using both L, and La in TDL yields the best performance of DiCha ViT.
4.4.3 Ablations for Diverse Channel Sampling (DCS)
Channel feature f in DCS. Tab. compares the performance of using channel tokens (ch,) and patch tokens (i.e., image patches after passing through the projection layer) to compute the similarity score for sampling in Algorithm Il (line 3). We observe that using channel tokens gains better performance on So2Sat and CHAMMI datasets. Note that while channel tokens are shared across all input images, patch tokens differ for each input image.
Impact temperature on DCS. Tab, shows the effect of temperature (DCS used Algorithm 1 on DCS. When (DCS is set to very small value, reported in the first column (denoted as "~ 0"), DCS selects channels with the lowest similarity scores to the anchor channel. Conversely, when Ipcs assigned large value, denoted "HCS" in the last column, DCS is reduced to HCS (18), meaning

---

sampling freq (%)
8
6
HCS
4
0
band 68a
band B12
band 87
band B11
band BB
band B6
band 85
imaginary WV
B2 (blue)
real VV
B4 (red)
B3 (green)
Imaginary VH
imaginary Lee. Cov
real VH
intensity Lee- VV
intensity Lee. VH
real Lee- Cov
Channel
Figure 5: Comparison of DCS and HCS [18] in terms of the frequency (%) each channel is sampled during training on So2Sat. Unlike HCS, which provides uniform distribution for all channels (red dashed line), some channels in DCS are trained much more than others (blue bars). For example, Real Lee-Cov channel (rightmost) is sampled twice as much as Band B8a (first bar).
that it selects the subset of channels randomly. We find that always selecting the lowest similar channels (= 0) does not yield the best performance. Instead, setting the temperature to tpcs 0.1 produces favorable results for both So2Sat and CHAMMI.
DCS and HCS on the distribution of number sampling of the channels. Fig. 5 compares the number of times each channel sampled during training with DCS (blue bars) and HCS [8] (red dashed line). DCS offers a different distribution for its channels compared with HCS, with some channels receiving more training than others. For example, Real part of Lee-filtered covariance matrix (Real Cov- in the last bar, is sampled twice frequently Band B8a channel (first bar).
5 Conclusion
In this paper, we present DiChaViT, model aimed at enhancing feature diversity and robustness in Multi-Channel Imaging (MCT) ViTs. First, we introduce Diverse Channel Sampling, novel channel sampling strategy that encourages the selection of more distinct channel sets during training. thereby promoting feature diversity. Additionally. DiCha incorporates Token Diversification Loss on the patch tokens and Channel Diversification Loss for channel tokens to further diversify the features learned in MCI-ViTs. Our experiments demonstrate a 1.5 5.0% point improvement over state-of-tte -f-the--rttttt- methods on satellite and microscopy imaging datasets. Many of our enhancements are not tied to any specific architecture and can be incorporated into new architectures as they are developed. DiCha ViT represents promising advancement addressing the challenges associated with MCI, paving the way for more effective MCI-ViT models.
Broader Impacts and limitations. The development of DiChaViT represents advancement in MCL, with potential positive impacts such as improved medical diagnosis and accelerated healthcare research. Additionally, its versatility in satellite imaging holds promise for environmental monitoring. However, there are also potential negative impacts, including the risk of bad actors using this research develop harmful applications, such invasive surveillance systems. This highlights the importance of ethical considerations and responsible deployment. One of the limitations of our work that it is not designed handle novel channels. Generalizing to unseen channels is challenging because it requires establishing connection between existing and new channels. This is further complicated in the presence domain shifts, which makes finding the informative channel weights even more difficult. Thus, investigating techniques to adapt to new channels at test time is promising research direction in MCI. In addition, our approach requires extra hyperparameter tuning, which may necessitate additional compute resources.
10

---

Acknowledgments and Disclosure of Funding
This material is based upon work supported, in part, by the National Science Foundation under award DBI-2134696. Any opinions, findings, and conclusions recommendations those of the author(s) and do not necessarily reflect the views of the supporting agencies.
References
[1] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing syxtems, 27, 2014.
[2] Christian Szegedy. Wei Liu, Yangqing Jia, Pierre Sermanet, and Scott Reed. Dragomiranguelov. dumitru erhan, vincent vanhoucke, and andrew rabinovich. 2015. going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[3] Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference muchine learning, pages 6105-6114. PMLR, 2019.
[4] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500 1500, 2017.
[5] Xuelei Li, Liangkui Ding, Li Wang, and Fang Cao. Fpga accelerates deep residual learning for image recognition. In 2017 IEEE 2nd Information Technology, Networking, Electronic and Automation Control Conference ([TNEC), pages 837- 840. IEEE, 2017.
[6] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arxiv: (exxx:::4.... 770404861, 2017.
[7] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[8] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[9] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[10] Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaced, Judy Hoffman, et al. Hiera: A hierarchical vision transformer without the bells-and-whistles arXiv preprint arXiv:2306. 00999, 2023.
[11] Henry Pinkard, Cherry Liu, Fanice Nyatigo, Daniel Fletcher, and Laura Waller. The berkeley single cell computational microscopy (bseem) dataset. arXiv preprint wwwix:2402.0019999 2024.
[12] Srinivas Niranj Chandrasekaran, Beth A. Cimini, Amy Goodale, Lisa Miller, Maria Kost-Alimova, Nasim Jamali, John Doench, Briana Fritchman, Adam Skepner, Michelle Melanson, John Arevalo, Juan C. Caicedo, Daniel Kuhn, Desiree Hernandez, Jim Berstler, Hamdah Shafqat-Abii Swalley, Shantanu Singh, and Anne E. Carpenter. Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations. bioRxiv. 2022. doi: 10.12000/22000000 05.0099......... URL (tRLahttpa:/// //www. biorziv. org/ Con.tenteeaalll//2////////////// 202022 01. 05.07090
David Root, Sussane
[13] Yury Goltsev, Nikolay Samusik, Julia Sennedy-Darling Salil Bhate, Matthew Hale, Gustavo Vazquez, Sarah Black, and Garry P Nolan. Deep profiling of mouse splenic architecture with codex multiplexed imaging. Cell, 174(4):968-981, 2018
[14] Zitong Chen, Chau Pham, Siqi Wang. Michael Doron, Nikita Moshkov, Bryan A. Plummer, and Juan Caicedo. CHAMMI: A benchmark for Shannel-adaptive models in microscopy imaging. In Thirty-eeventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreviev het//EorumI..eiuucblllleelt

---

[15] Nicolas Bourriez, Ihab Bendidi, Cohen Ethan. Gabriel Watkinson, Maxime Sanchez. Guillaume Bollot, and Auguste Genovesio. Chada-vit Channel adaptive attention for joint representation learning of heterogeneous microscopy images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[16] Xiao Xiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Jian Kang, Lichao Mou, Hossein Bagheri, Matthias Haberle, Yuansheng Hua, Rong Huang, Lloyd Hughes, Hao Li, Yao Sun, Guichen Zhang, Shiyao Han, Michael Schmitt, and Yuanyuan Wang. So2sat lcz42: A benchmark data set for the classification of global local climate zones [software and data sets]. IEEE Geoscience and Remote Sensing Magazine, 8(3);
76-89, 2020. doi: N..1109/MRS.2022 2020.2964708.
[17] Xiaoxiang Zhu, Jingliang Hu, Chunping Qiu. Yilei Shi, Hossein Bagheri, Jian Kang. Hao Li. Lichao Mou, Guicheng Zhang, Matthias Häberle, Shiyao Han, Yuansheng Hua, Rong Huang, Lloyd Hughes. Yao Sun. Michacl Schmitt, and Yuanyuan Wang. New: So2sat le742, 2019. URL https: atum. tum
de/ 1483140
[18] Yujia Bao, Srinivasan Sivanandan, and Theofanis Karaletsos. Channel vision transformers: An image worth 1 16 x words. In The Twelfth International Conference on Learning Representations, 2024. URL (ttp:ttp::// h/tp:::://ppeee http:///ppeeeeeeeeeeeeeee net/ fuet/forure om inruummmmiiii CK-CK5Hf b5hBG
[19] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 2014.
[20] Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Scholkopf, and Stefan Bauer. On disentangled representations Icarned from correlated data. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10401-10412. PMLR, 18-24 2021. URL https::/
mlr. press v/1v/v11111/ tttp//rrrbbbeeea...aaaaaa html
[21] Xiao Liu, Pedro Sanchez, Spyridon Thermos, Alison Q. O'Neil, and Sotirios A. Tsaftaris. Learning disentangled representations the imaging domain. Medical Image Analysis, 80:102516, 2022. ISSN 1361-8415 doi: httpss///oo..rrg/// org/11. 1016 media 2022. 102516. URL htttps//www/ sttpp::/www.scieeneeirree com/ http://cenncearricclee////e/ /tt..///...1111111111200111111
[22] Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang. and Wenwu Zhu. Learning disentangled represen- tations for recommendation. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Buc, E. Fox, and R. Garnett, editors. Advances in Neural Information Processing Systems. volume 32, Curran As- sociates, Inc., 2019. URL https: pesttss:/proceeeteee http://pppeeee...ee/eepe// 22186a7770885444444433111 www.o//ee22aaaa-b-ee Paper.pdf
[23] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. In 16th Eurppean Conference on Computer Vision-ECCV 2020, volume 12367, pages 205-221. Springer, 2020.
[24] Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N Siddharth, Brooks Paige. Dana H. Brooks, Jennifer Dy, and Jan- Willem van Meent. Structured disentangled representations. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Second Second International Conference on Artificial Intelligence and Statisties. volume 89 of Proceedings of Machine Learning Research, pages 2525 2534. PMLR, 16 18 Apr 2019. WRRLitttps//prroeeddngg
mlr press/ v89/ eem./emmmm............aaaaammm.m html
[25] Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gauray Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. In International Conference Learning Representations, 2021. UtL./ptta:://
[26] Kanchana Ranasinghe. Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan. Orthogonal projection loss. In Proceedings of the IEEE/CVF international conference computer vision. pages 12333- 2021.
[27] Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 6348-6359. PMLR, 13-18 Jul 2020. URL https wwtppp/eeeee/peeeeeeeppeeeeeeeee
//proceedings.o mlr. prwss/ wttp//1111//1/// w119/ooaaeelll0aaaa.c html
[28] Jian Jia, Naiyu Gao, Fei He, Xiaotang Chen, and Kaiqi Huang. Learning disentangled attribute represen- tations for robust pedestrian attribute recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1069-1077, 2022.
12

---

[29] Jungsoo Lee. Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased represen- tation via disentangled feature augmentation. Advances in Neural Information Processing Systems, 34: 25123-25133, 2021.
[30] Pierre Colombo, Guillaume Staerman, Nathan Noiry, and Pablo Piantanida. Learning disentangled textual representations via statistical measures of similarity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2614-2630, 2022.
[31] Andrea Burns, Aaron Sarna, Dilipp Krishnan, and Aaron Maschinot. Unsupervised disentanglement without autoencoding: Pitfalls and future directions. arXiv preprint arXiv:2108.06613, 2021.
[32] Ruiqian Nai, Zixin Wen, Ji Li, Yuanzhi Li, and Yang Gao. Revisiting disentanglement in downstream tasks: study on necessity for abstract visual reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, (13)) 14405 14413, Mar. 2024. doi: 10. 10.1609/aaai 13.134.9934. URL
https://ojj aaai www.:aaai...aannaaaea index. php/ http/AAA//artille///coaa /vt///iii//ee//e /23344
[33] Andrea Valenti and Davide Bacciu. Leveraging relational information for learning weakly disentangled representations. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1-8, 2022. doi: (LL////ICNNNS004222000099999
[34] Man Zhou, Keyu Yan, Jie Huang, Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual information-driven pan-sharpeming. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1798-1808, 2022.
[35] Jing Zhang. Deng-Ping Fan, Yuchao Dai, Xin Yu, Yiran Zhong, Nick Barnes, and Ling Shao. Rgb-d saliency detection via cascaded mutual information minimization. In Proceedings of the IEEE/CVF international conference computer vision, pages 4338 4347, 2021.
[36] Ankan Bhattacharyya, Sommath Chatterjee, Shibaprasad Sen, Aleksandr Sinitca, Dmitrii Kaplun, and Ram Sarkar. A deep learning model for classifying human facial expressions from infrared thermal images. Scientific reports, 11 11(1):20696, 2021.
[37] Jionghui Jiang, Fen Liu, Yingying Xu. Hui Huang, et al. Multi-spectral rgb-nir image classification using double-channel cnn. IEEE Access, 7:20607-20613, 2019.
[38] Daniel Siegismund. Mario Wieser, Stephan Heyse, and Stephan Steigele. Learning channel importance for high content imaging with interpretable deep input channel mixing. In DAGM German Conference on Pattern Recognition, pages 335-347. Springer, 2023.
[39] Bryan A. Plummer, Nikoli Dryden, Julius Frost, Torsten Hoefler, and Kate Saenko. Neural parameter allocation search. In International Conference on Learning Representations (ICLR), 2022.
[40] Pedro Savarese and Michael Maire. Learning implicitly recurrent CNNs through parameter sharing. In International Conference on Learning Representations (ICLR), 2019.
[41] Chau Pham, Piotr Teterwak, Soren Nelson, and Bryan A. Plummer. Mixturegrowth: Growing neural networks by recombining learned parameters. In IEFE Winter Conference on Applications of Computer Vision (WACV), 2024.
[42] David Ha. Andrew Dai, and Quoe Le. Hypernetworks. In International Conference on Learning Represen- tations (ICLR), 2016.
[43] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL URL htttps://openreviie..eeee id=YicbFaRTly
[44] Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: foundation model for weather and climate. In International Conference on Machine Learning, pages 25904- 25938, PMLR, 2023.
[45] Michail Tarasiou, Erik Chavez, and Stefanos Zafeiriou. Vits for sits: Vision transformers for satellite image time series. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 10418-10428, 2023.
[46] Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pages 27378 273944 PMLR, 2022.
13

---

[47] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of IEEE/CVF winter conference on applications of computer vision, pages 574-584, 2022.
[48] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11920 2020.
[49] Wei Hu, Lechao Xiao, and Jeffrey Pennington. Provable benefit orthogonal initialization in optimizing deep linear networks. International Conference on Learning Representations, 2020. URL https
/ www.eeeoonoddd
[50] Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X. Yu. Orthogonal convolutional neural networks, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[51] Mario Lezcano- Casado and David Hartunez-Rubiq Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group. In International Conference Machine Learning, pages 3794-3803, PMLR, 2019.
[52] Eu Wern Teh, Terrance DeVries, and Graham W Taylor. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16, pages 448-464, Springer, 2020.
[53] Yifeng Ding, Shuwei Dong, Yujun Tong, Zhanyu Ma, Bo Xiao, and Haibin Ling. Channel dropblock: An improved regularization method for fine-grained visual classification. In British Machine Vision Conference (BMVC), 2021.
[54] Maxime Oquah, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Wer Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Learning robust visual features without supervision. 2023.
Dinov2:
[55] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2019. URL/ htt.eeee...et.ooooooodddd/gdddd.
[56] Gianluca Donato and Serge Belongie. Approximate thin plate spline mappings. In Computer Esion-ECCV 2002: 7th European Conference on Computer Vision Copenhagen, Denmark, May 28-31, 2002 Proceed- ings, Part III 7, pages 21 1-31. Springer, 2002.
14

---

A Implementation details
We utilize ViT small architecture (21M parameters) implemented in D1NOv2 [54] the backbone for all the baselines and ViT-S/8 (patch size of 8) on So2Sat. The Adam optimizer (55] is used train the models, minimizing cross-entrope loss on JUMP-CP and So2Sat, and proxy loss on CHAMMI.
5)
Specifically, we use ViT-S/16 (patch size of 16) on CHAMMI and JUMP-CP,
CHAMMI dataset [14]. The goal of CHAMMI is train model to learn the feature representation for the input image. Thus, we use the [CLS] token at the final layer as the feature representation and train the model to minimize the proxy loss (52) We then evaluate the model on various tasks following the evaluation code provided by the authors, in which a -Neerestt Neighbour classifier is
used to predict the macro-average Fl-score for each task separately [ The mhannel-adaptin interfaces are adapted from the author's 's implementation code Besides the model, we incorporate the same data augmentation as introduced by the authors, such as thin-plate-spline (TPS) transformations [56]. We train each model for 60 epochs with a learning rate of 0.00004, and batch size of 64.
JUMP-CP [12] and So2Sat [17 datasets. Following Bao et al. [8], the learning rate is warmed up for the initial 10 epochs, peaking at 0.0005 after which it will gradually decay to 10-6 following a cosine scheduler. We also apply a weight decay of 0.04 the weight parameters, excluding the bias and normalization terms to mitigate overfitting. Additionally, we use the same data augmentation as used in the code provided by the authors. To get the final prediction, we pass the Transformer encoder's s representation for the [CLS] token into a classifier head to predict the probability of each class. We train each model for 100 epochs, with batch size of 64 on JUMP-CP. and 128 on So2Sat. We adapt the code provided by the authors [8] for the baselines in our work
Compute resources. In this study, experiments were conducted So2Sat and CHAMMI using single NVIDIA RTX (48GB RAM) and three Intel(R) Xeon(R) Gold 6226R CPUs 2.90GHz. For experiments on JUMP-CP, two NVIDIA RTX A6000 GPUs and six Intel(R) Xeon(R) Gold 6226R CPUs @ 2.90GHz were utilized.
B Additional experimental results
Extended main results. Tab. 7 shows an extension of the main resulting table in the main paper (Tab. [ where we include CNN-based (ConvNeXi backbone ( models from [14]. To ensure a fair comparison, we adjust the number of layers in these cased models that all models in Tab. Thave approximately 21M parameters. We can observe that in general, DiChaViT outperforms
CNN-based and ViT-based models on the three datasets.
Extensive ablation results on DiCha Tab. exxends Tab. 3 in the main paper to have a better understanding of the individual effects and contributions of each of the losses. We observe that adding DCS helps improve the performance (e.g.. by 4% on CHAMMI), and robustness of the model, especially when tested on partial channels (a boost of 35% So2Sat Partial). Similarly, TDL and CDL also show improvement across the three datasets. For example, TDL improves the performance by 2.5% on CHAMMI and 1.7% on So2Sat on full channels.
Effect of CDL on channel token distributions. Fig. 6 illustrates the distributions of channel tokens with (blue) and without (red) CDL. Each subplot presents the distribution of trained channel token on the CHAMMI dataset. We observe that CDL results in more flattened distributions with more non-zero values in the channel tokens.
Attention scores of the [CLS] token the patch tokens at different layers. Fig. 7 shows an extended version of Fig. [b) in the main paper, where we calculate the attention scores of the [CLS] token to the patch tokens at layers 4, 8, and 12 (the penultimate layer), and then aggregate them by channel. This indicates that ChannelViT (top) relies more heavily on specific channels (e.g., microtubules and nucleus) for making predictions, while other channels (e.g., protein and er) are less considered. In contrast, DiChaViT (bottom) displays more evenly distributed attention scores across channels, indicating that each channel contributes more significantly to the model's 's predictions.
htt.ep///httaaaoo/////eeeerrrrrr httpp: httegp///ggaaabb.oo/////oaasssts
15

---

Table 7: Test accuracy of channel-adaptive models across multi-channel datasets. DiCha VIT performs better than other CNN- and ViT-based baselines. It shows overall better performance on Allen and CP. and 1.5 2.5% improvement JUMP-CP and So2Sat. "Full" refers to lesting on all channels, while "Partial" means testing on subset of channels. We use Sentinel-1 channels for So2Sat, and fluorescence channels for JUMP-CP.
CHAMMI, especially
Architecture
CHAMMI [4]
JUMP-CP 121
So2Sat 17
Model
Allen
HPA
CP
Full
Partial
Full
Partial
HyperNet
42
ConvNeXt
58.43
65.93
26.53
53.48
10.58
58.97
41.54
Depthwise
ConvNeXt
58.76
57.60
27.39
49.34
39.88
58.60
38.87
TemplateMixing
39
ConvNeXt
60.21
63.44
25.98
49.74
43.74
60.79
40.61
HyperNet
42
ViT
45.17
63.90
26.23
47.07
42.43
60.73
41.88
Depthwise
ViT
50.35
71.52
27.74
49.86
44.98
60.41
43.41
Template.Mixxing [39
40
ViT
49.51
64.52
25.65
52.48
43.85
55.86
37.28
ChAda- ViT
15
VIT
67.08
60.67
24.60
65.03
42.15
56.98
12.38
ChannelViT
[8
VIT
67.66
62.14
27.62
67.51
56.49
61.03
46.16
DiCha VIT (ours)
VIT
75.69
63.67
28.98 69.19
57.98
63.36
47.76
Table Extensive Ablation Studies on DiChaViT. We expanded Tab. 3 in the main paper to show the performance improvements achieved with different combinations of our components, offering more insights into the roles of each component, We report meanistd over three runs.
CHAMMI
JUMP-CP
So2Sat
Exp. Model
Avg Score
Full
Partial
Full
Partial
1.
ChannelViT w/o HCS (ChAda-ViT)
03.88 65.03 42.15512.33 56.98
12.38-12.03
2. HCS (ChannelViT)
64.90.10.75 67.51 56.49:1.53
61.03:1.117
46. 16.46
3. DCS
67.741:0.33 67.90:1..37
56.6110.43
62.1740.23
47.30-10.43
4. + TDL.
66.27:10.38 65.77:1..88
43.89+1.89
50.68
15.63+45.01
5. CDL
64.241:.554 66.75:20.57 42.74
57.70+0.11
15.0814.00
6. DCS TDL
68.0710.44 67.66110.28 56.8710.78
62.20110.18
45.74=1042
TDL + CDL
65.3210.448 66.03:10.39 42.37=1.16
59.20110.43 17.88:3.14
8.
DCS + CDL
67.61+00.44
12.10.0.6 56.62+0.78
62.39 46.87+4024
9. + TDD + CDL+ HCS
67.46+0.39
67.50+#0.00 57.90 62.05+0.099 45.08++0600
10.
TDL + CDL+ DCS (DiChaViT)
69.66+0043 69.19=0.47 57.98+0.41
63.36+0.11 47.76++0.23
-0.2
-0.1
0.2
-0.1
0.0
ChannelViT
(b) DiChaViT
Figure 6: The effect of Channel Diversification Loss (CDL) channel embedding distributions. Each subplot shows the distributions of a channel token after training on the CHAMMI dataset. (a)
ChannelViT's features (red) more concentrated around O. (b) In contrast, DiCha shows more flattened distributions with more non-zero values (blue).
16

---

Microtudules pridait
Misulabuha
Menmutules
nudleus ChannelVIT
ChannefVIT
Microtubules
Micrenublles
Merchabulee
protein
DIChaVIT
DIChaViT
DiChaVIT
(a) Layer
(b) Layer 8
(c) Layer 12 (last layer)
Aggregated patch token attention scores by channel
Figure 7: Attention scores HPA channels (CHAMMI) across different layers. We compute attention scores of the [CLS] token to the patch tokens in a layer (layers 4, 8, and 12) and aggregate them by channel. ChannelViT (top) relies on certain channels (e.g., microtubules and nucleus) to make predictions and less on other channels (e.g., protein and er). In contrast, DiCha ViT demonstrates more evenly distributed attention scores across channels, suggesting that each channel contributes more to the model's predictions.
Table Detailed performances ChannelViT and DiCha VIT on JUMP-CP in the leave-one- channel-out at test time setting. We present the details of column "7" in Tab, of the main paper. DiChaVit ViT achieves 2% better performance on each combination compared with ChannelViT.
Channels at inference
ChannelVil
DiCha VIT (ours)
{0, 1,,,,,33,4 3,,,, 4,
67.3710.60
69.21:10.19
00,1,2334,,577
67.20=10.59
69.06:10.20
40,1,2.3,4.6,77
67.28=0.53
69.12::0116
7
58.52=0.63
59.611:.117
20,1,2,4.5,6,77
37.70=0.60
38.83:0.46
70,1,3
61.90+0.48
63.28+
0,,,,,
61.21+0.41
62.72++.228
3,2,3,455,677
61.72=00.48
63.48++0.0
Leave-one-channel-out at test time. In Tab. 9. we provide individual channel combination results when using seven channels (of eight) of the JUMP-CP dataset for inference. This corresponds to the details in column "7" from Tab. 2 in the main paper, representing Cg 8 different channel combinations. For each combination, we report the mean and std the models computed over three runs. Our results demonstrate that DiChaViT gets 1 2% better performance for each combination while also providing more stable results (i.e., smaller model variance) than baseline ChannelViT.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, our abstract and introduction reflect our main contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We include limitation section the end of our paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We also provide the implementation details in Section 4. and Section in the appendix. We release the code and checkpoints at https::///// ttt.ttttttttttttt//g////t com http://haaaaainenneeeeee wtwpeeeeeeeeeeeeeeeeeeeeaaa chwnnelaaaiaeaeeeeeee
Justification: Our method is described clearly in Fig. 2 and Section 3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We release the code and instructions on downloading the datasets at httts://giihhe com/ cww.daaasciiecceeee wtaptttacciinncccieeeeeeeeee diverse_
vit
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//ddeeeSbmimiia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details in Section 4.1 and Section A in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
We provide error bars in our results, such as in Table 2 and Fig.
4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the implementation details with computer resources Section A in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
The research conducted conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We have broader impact statement.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We cite the original papers and provide the links to their codes. The license and terms properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We plan to release the code with a document on how to use it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
This paper does not use crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "Sv8FYFoBOs", "reviewer": "Reviewer_q1Xa", "summary": "Multiple channel imaging (MCI) is widely used in different application domains, ranging from medical image analysis to satellite imagery. Each channel can contain information that is orthogonal compared to other channels, which can be useful in downstream tasks. Obtaining such orthogonal signals from individual channels without extracting repetitive information is challenging. In this work, three contributions are made to improve the channel diversity in vision transformer (ViT) class of models. Firstly, to enhance separation between channel specific tokens a contrastive loss, known as channel diversification loss (CDL) is used. Next, patch-level token diversification loss (TDL) ensures extraction of patch-level features that are not dependent on the channels. Finally, masked-training of sorts across channels is performed using diverse channel sampling (DCS). These components result in a composite loss function that is used to demonstrate competitive performance on three MCI datasets. The proposed diverse channel ViT (DiChaViT) method outperforms relevant baseline methods, yielding considerable performance improvements.", "questions": "See points under weaknesses.", "limitations": "The authors have addressed the main limitations in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* MCI data and meaningfully combining channel information, more so, in the presence of missing channels at inference is a challenging task. The proposed contributions that use CDL, TDL, DCS are well-reasoned, and appear to be useful in mitigating some of the known problems within this literature.   \n\n* Experiments are comprehensive, with the proposed method showing competitive performance across all three datasets. In some settings, even yielding up to 5% improvements. \n\n* Ablation studies cover a large combination of settings which help understand the contribution of the key contributions. \n\n* The paper is well-structured, performs a thorough literature review of relevant works, and nicely written.", "weaknesses": "* **Unseen channels during training**: One of the key limitations of this work, which is also briefly acknowledged by the authors, is that the method is only focused on dealing with missing channels only at inference. Meaning, all channels have to be seen by the model during training. This might not always be the case. I would like the authors to elaborate more on why this is not considered in this work. I would be happy with an explanation in terms of which of the contributions would breakdown for unseen channels. Just to be clear, I am not seeking additional experiments. \n\n* **Balancing loss function**:The composite loss function has multiple hyperparameters: $\\lambda_{CDL}, \\lambda_s,\\lambda_d$, and temperature, $t$. The authors describe the tuning of these parameters individually with discussions on the ranges without elaborating on how to obtain these for other datasets. For instance, $\\lambda_{CDL}$ is set to 0.001 for So2Sat and 0.1 for CHAMMI. How were these obtained? Fig. 4 (a,b) show the sensitivity of test accuracy wrt these parameters. And do these magnitudes mean anything? Why the log-scale? \n\n* **Channel anchors**: The discussion on using channel anchors is missing some key information. This could be because I was not aware of the ProxyNCA++ loss referenced in [52]. How are the channel anchors initialised and then selected for each channel? What would happen if instead of learning these vectors, one used a one-hot encoding to specify the channels? Perhaps some clarification here can be useful.  \n\n* **Temperature, t**: In L. 148 authors say that results are not sensitive to the choice of $t$ in Eq. 1. However, they set it to a specific (and a peculiar) value of $t=0.07$. And, there is another temperature parameter (?) in Sec. 3.3 when describing DCS which is later reported in Table 5. They are different softmax temperatures, as far as I can see. This is very confusing. And which of the temperatures is more important?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Unseen channels during training**: One of the key limitations of this work, which is also briefly acknowledged by the authors, is that the method is only focused on dealing with missing channels only at inference. Meaning, all channels have to be seen by the model during training. This might not always be the case. I would like the authors to elaborate more on why this is not considered in this work. I would be happy with an explanation in terms of which of the contributions would breakdown for unseen channels. Just to be clear, I am not seeking additional experiments.\n\n\nOur contributions should still benefit from a setting requiring inferring unseen channels, but taking advantage of it is not straightforward. A key challenge when generalizing to unseen channels is establishing a connection between existing and new channels.  This is not challenging, as simply identifying a similar channel may not be sufficient.  For example, using the weights for a similar channel may result in simply extracting the same features as that channel, resulting in redundancy that does not improve performance.  Instead, unseen channels require extracting the most informative features, which may not be learned by the most similar channel.  This is further complicated in the presence of domain shifts, which makes finding the move informative channel weights even more difficult.  Thus, this requires a more focused effort to create this mapping that goes beyond the scope of our paper.\n\n\n\n> **Balancing loss function**: The composite loss function has multiple hyperparameters: 𝜆𝐶𝐷𝐿, 𝜆𝑠, 𝜆𝑑, and temperature, 𝑡. The authors describe the tuning of these parameters individually with discussions on the ranges without elaborating on how to obtain these for other datasets. For example, 𝜆𝐶𝐷𝐿 is set to 0.001 for So2Sat and 0.1 for CHAMMI. How were these obtained? \n\nHyperparameters are set using grid search over a validation set.  We shall update our plot to include both the test and validation results in our revised paper.\n\n\n\n\n\n\n> Fig. 4 (a,b) show the sensitivity of test accuracy wrt these parameters. And do these magnitudes mean anything? Why the log-scale?\n\nThe magnitudes show how these parameters affect the model performance. For example, a very high $\\lambda_{𝐶𝐷𝐿}$ value could cause the model to excessively prioritize diversifying the channel token features, which may negatively affect the performance on downstream tasks. In contrast, if $\\lambda_{𝐶𝐷𝐿}$ is too small, it may not effectively diversify the learned channel token features, leading to underutilization of its benefits.\n\nSimilarly, $\\lambda_{s}$ and $\\lambda_{d}$ control the constraint of tokens belonging to the same channels and different channels in Eq. 4. \n\nUsing a log scale allows us to visualize a wide range of parameters to observe the impact of these parameters on the final performance.\n\n\n\n\n\n\n> **Channel anchors**: The discussion on using channel anchors is missing some key information. This could be because I was not aware of the ProxyNCA++ loss referenced in [52]. How are the channel anchors initialised and then selected for each channel? \n\nAs noted in L132 of our paper, each channel has its own anchor initialized orthogonally to the other channel anchors.  There is no need for them to be “selected” as they are trained as their anchor. I.e., in So2Sat there are 18 channels, so we would orthogonally initialize 18 channel anchors, each trained to correspond to one specific channel.\n\n> What would happen if instead of learning these vectors, one used a one-hot encoding to specify the channels? Perhaps some clarification here can be useful.\n\nEach channel anchor is the same dimension as the channel token so that we could compute similarities between them.  For the 384-D dimension channel tokens used in our experiments, we would have a 384-D channel anchor.  A one-hot encoding on the 18 channel So2Sat dataset would effectively ignore 384-18=366 of the channel token features (as their loss contribution would always be 0 in a one-hot encoding of the anchors).  Thus, this would, in large part, make our CDL loss completely ineffective in practice.\n\n> **Clarification on two Temperatures**:  In L. 148 authors say that results are not sensitive to the choice of 𝑡 in Eq. 1. And, there is another temperature parameter (?) in Sec. 3.3 when describing DCS which is later reported in Table 5. They are different softmax temperatures, as far as I can see. This is very confusing. And which of the temperatures is more important?\n\nThank you for carefully reading our paper and spotting out the confusion regarding the temperatures! In line 148, the temperature $t$ controls the sharpness of the probability distribution when using CDL. We observed that the model is not sensitive to the value of $t$, and simply fixing it consistently yields good results across datasets.\n\nThe second temperature $t$ mentioned in Section 3.3 (Algorithm 1) is used to control the distribution of CDS. This $t$ is the one presented in Table 5 and is more sensitive in our experiments.\n\nWe will revise the manuscript by using $t_{CDL}$ and $t_{CDS}$ for these temperatures to make them clear.\n\n\n\n>  However, they set it to a specific (and a peculiar) value of 𝑡=0.07. \n\nIn Eq. 1 of our paper the temperature $t$ is represented as the denominator of a fraction.  We used grid search centered around the denominator value of $t=1/9$ used by ProxyNCA [52].  None of the results were statistically different, but we used $t=1/14=0.07$ as it had the best average result.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHIdwXb7Ql", "reviewer": "Reviewer_gjzo", "summary": "In this paper, the authors present an improved methodology for modeling hyper-spectral multi channel images that can support a variety of channel configurations at test time. The authors reason that the current baselines treat all channels equally and do not consider the diverse qualities of each channel type. Hence they introduce 3 modifications to the training procedure that improves the performance of multi-channel image evaluations over state-of-the-art baseline methods.\n\nThe key contributions of the paper include:\n1. Orthogonal initialization of Channel Embeddings and Channel Diversification Loss $L_{CDL}$\n2. Token Diversification Loss $L_{TDL}$ with different weighting for tokens from the same channel ($\\lambda_s$) and different channels ($\\lambda_d$).\n3. Diverse Channel Sampling strategy that samples diverse set of channels during training\n\nUsing the above modification to loss function and training procedure, the authors demonstrate an improvement of 1.5-5% in classification accuracy over the best performing baseline methodology (ChannelViT) on microscopy benchmarks CHAMMI and JUMP-CP and a satellite imaging benchmark So2Sat. The authors have performed further ablation studies to compare the usefulness and contribution of the loss terms.", "questions": "* Can the authors provide a more intuitive explanation for why patch token orthogonality (especially within channel patch token orthogonality) is necessary or might be useful? As the patch tokens are generated using the same set of weights for all patches of images, an identical patch of pixels would produce an identical set of tokens. In such a case, patch token diversification would be unreasonable. The results from Fig 4c further indicates that imposing weaker constraints on tokens from same channels compared with tokens from different channels obtains best performance. So, how is the patch token diversification loss on same channels helpful?", "limitations": "* The authors have shown results on limited datasets with smaller dataset sizes. The general applicability of this methodology to large data settings is unknown and would require further experimentation.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths of the Paper:\n* The overall motivation and idea of studying the impact of diversity of channel representations, patch tokens and sampled channels on learned representations and classification performance is interesting. The observations and contributions from the paper will be useful for further exploration and research in this direction.\n* The paper is written with sufficient details and clarity of the contributions. \n* Well presented introduction and motivation to the problem with sufficient discussion of relevant methods and baselines.\n* The authors have provided all the details of their methodology, training parameters, settings and hardware details for reproducibility. They've also agreed to make the code available.\n* The authors have chosen relevant strong baselines for comparison and show improvement in classification performance over the baselines with their methodology.\n* The authors have performed ablation studies over the new loss terms introduced and demonstrate the performance improvements with each of those loss terms.", "weaknesses": "While the overall methodology is well presented and has reasonable intuitive motivation, there are some weaknesses to the proposed methodology. \n\n1. One of the key problems addressed by the channel adaptive image models are generalization to settings where only a subset of channels available. This requires the models to learn both redundant and non-redundant information across channels for generalization to settings where some of the channels are missing or unavailable. While, the authors show channel diversification improving performance of the model, it is important to investigate and interpret where the new setup is improving the performance. Additional experiments on interpretations and comparisons of predictions between baseline methods and proposed methodology might help address this weakness.\n\n2. In Fig 5, the authors compare the distributions of channels sampled during training between HCS (hierarchical channel sampling) and DCS (diverse channel sampling) methodologies. As that DCS samples some channels more compared to other during training, it is important to compare the performance between the baselines and proposed methodologies on those individual channel combination settings. Given the observation above and performance improvements in Table 2 being quite small with large standard deviation, the data is insufficient to the readers to demonstrate strict performance gains.\n\n3. In Table 3, the authors have performed ablation studies by leaving out each of the losses or sampling methods. However, it is also important to show performance improvements with inclusion of each of the losses independently to understand the independent effects and contributions of each of the modifications", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. It is important to investigate and interpret where the new setup is improving the performance. Additional experiments on interpretations and comparisons of predictions between baseline methods and proposed methodology might help address this weakness.\n\nThank you for your valuable suggestions! To investigate and interpret where the new setup is improving the performance, we conduct additional analysis in responses 1, 2 and 3 of the Global Response, and Fig. 1&2 in the rebuttal PDF. Additionally, Fig. 7 in the supplementary also gives some insight on the distribution of channel tokens.\n\n> 2.  Given the observation above and performance improvements in Table 2 being quite small with large standard deviation, the data is insufficient to the readers to demonstrate strict performance gains.\n\nThank you for the question.  We believe that this comment stems from a misunderstanding of what the standard deviations pertain to in Tab. 2.  Specifically, these refer to variances due to the presence or absence of a channel during inference (not model variance). The relatively large standard deviations show some channels are very informative, and removing them results in large drops for both models, but they *do not* suggest our gains over ChannelViT are not significant.\nTo illustrate, below we provide individual channel combination results when using 7 channels (of 8), i.e., the “7” column from Table 2, representing 7C8 = 8 channel combinations.  However, for each combination we report the mean and standard deviation of the models computed over 3 runs.  Our results demonstrate that our approach gets 1-2% better performance for each combination while also providing more stable results (i.e., smaller model variance) than ChannelViT.  Other experiments in Tab. 2 reported similar behavior, clearly demonstrating our benefits over prior work. \n\n| Channels at Inference        | ChannelViT           | **DiChaViT (ours)**       |\n|---|----|----|\n| [0, 1, 2, 3, 4, 5, 6]        | 67.37±0.60           | **69.21±0.19**     |\n| [0, 1, 2, 3, 4, 5, 7]        | 67.2±0.59            | **69.06±0.20**     |\n| [0, 1, 2, 3, 4, 6, 7]        | 67.28±0.53           | **69.12±0.16**     |\n| [0, 1, 2, 3, 5, 6, 7]        | 58.52±0.63           | **59.61±0.17**     |\n| [0, 1, 2, 4, 5, 6, 7]        | 37.7±0.60            | **38.81±0.46**     |\n| [0, 1, 3, 4, 5, 6, 7]        | 61.9±0.48            | **63.28±0.31**     |\n| [0, 2, 3, 4, 5, 6, 7]        | 61.21±0.41           | **62.72±0.28**     |\n| [1, 2, 3, 4, 5, 6, 7]        | 61.72±0.48           | **63.48±0.20**     |\n\n> 3. Table 3: It is also important to show performance improvements with inclusion of each of the losses independently to understand the independent effects and contributions of each of the modifications\n\nWe thank the reviewer for their suggestions! In Table 1 of the rebuttal PDF, we extended Table 3 in the main paper to have a better understanding of the individual effects and contributions of each of the losses. We observe that adding DCS helps improve the performance (e.g., by ~4% on CHAMMI), and robustness of the model, especially when tested on partial channels (a boost of ~35% on So2Sat Partial). Similarly, TDL and CDL also show improvement across the three datasets. For example, TDL improves the performance by 2.5% on CHAMMI and 2% on So2Sat on full channels.\n\n>  Can the authors provide a more intuitive explanation for why patch token orthogonality (especially within channel patch token orthogonality) is necessary or might be useful?\n\nPatch token orthogonality encourages the tokens generated from different image patches are distinct from each other, helping in preserving and enhancing the discriminative features of the patches. Patches within the same channel can still contain significant variations due to imaging different image regions, and our model helps encourage that they encode different features, if possible. \n\n> An identical patch of pixels would produce an identical set of tokens. Is patch token diversification unreasonable? \n\nIn this case the model can not further optimize the diversification loss (it’s at the minimum). That said, like any regularizer, it encourages a bias whose contribution must be carefully tuned.\n\n> The results from Fig 4c further indicates that imposing weaker constraints on tokens from same channels compared with tokens from different channels obtains best performance. So, how is the patch token diversification loss on same channels helpful?\n\nFig. 4C implies that one may want to apply a larger penalty for the similarity between patches from different channels, compared to within channel. That said, patch token diversification loss on same channel is still helpful, as shown in Table 4 of the main paper. \nPatches within the same channel can still contain significant variations due to different image regions. By enforcing some level of diversification within patches from the same channel, the model can capture finer details and nuances that might be missed.\n\n> Limitations. Experiments are conducted with limited datasets with smaller dataset sizes.\n\nWe appreciate the reviewer's feedback. We begin by noting that our experiments are already more extensive than prior work in MCI, as we combine the benchmarks used by two different recent MCI papers [14,18].  In addition, these are similar sizes to many widely used benchmarks (e.g., COCO has 330K images while CHAMMI, JUMP-CP, and So2Sat contain 220K, 217K, and 376K images, respectively).  While they are not suitable for large-scale pretraining, many (if not most) of the applications in MCI simply cannot do so due to lack of data availability and the expertise required to collect new data.  For example, collecting new data and annotations for CHAMMI requires a biologist to perform a time consuming experiment in a lab. As such, we would argue that our experiments are more representative of the real-world applications of MCI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XyDhjhFiRU", "reviewer": "Reviewer_4kDP", "summary": "The authors provided the plug-and-play framework for MCI analysis, considering the classification as a downstream task. To do so, they introduced two-channel and token diversification strategies, and in addition, they proposed a new channel sampling strategy for converging the training faster.", "questions": "Please see the Weaknesses.", "limitations": "Please see the Weaknesses.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Fair and complete ablation studies\n- The manuscript is well-written, constructed, and interesting.", "weaknesses": "- The authors repeatedly stress that each modification in DCS and TDL helps ensure the model's robustness. What ablation study did you apply to endorses it? Did you do any statistical testing on your experiments?\n\n- The method, as stated, is plug-and-play. Did the authors apply their strategy to other efficient transformer structures with token sampling strategies [1], window-based [2], or linear attentions [3,4]? Because such methods don't have such redundancy exist in the standard Transformer.\n\nThe paper needs proofreading. As instances: \n- In line 37, the authors used “to” two times repeatedly. Omit one.\n- In line 120, it is better to say “as noted in Section 1” rather than “as noted in the Introduction”.\n- When citing a work, cite it after the author's name. For example, in line 98, reformat it as “ Recently, Bao et al. [18] …”.\n\n[1] Fayyaz, Mohsen, et al. \"Adaptive token sampling for efficient vision transformers.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Liu, Ze, et al. \"Swin transformer: Hierarchical vision transformer using shifted windows.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[3] Ali, Alaaeldin, et al. \"Xcit: Cross-covariance image transformers.\" Advances in neural information processing systems 34 (2021): 20014-20027.\n\n[4] Shen, Zhuoran, et al. \"Efficient attention: Attention with linear complexities.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors repeatedly stress that each modification in DCS and TDL helps ensure the model's robustness. What ablation study did you apply to endorses it? Did you do any statistical testing on your experiments?\n\nTable 3 of our paper presents a leave-one-out ablation study, highlighting that best performance is obtained by incorporating each component.  We have expanded upon that study in the rebuttal PDF Table 1 (including the standard deviation from model variance), where we clearly demonstrate the benefits of each model component.  Finally, Tables 1 and 2 of our main paper, Table 1 of our rebuttal PDF, and our response to the second question of Reviewer gjzo also discusses experiments using “partial” sets of channels, i.e., evaluating our robustness to missing channels.  Note that we do not report a separate “partial” result for CHAMMI, as the different image sources within this dataset only contain “partial” sets of channels (so no “full” channel results are possible).\n\n\n\n\n> The method, as stated, is plug-and-play. Did the authors apply their strategy to other efficient transformer structures with token sampling strategies [1], window-based [2], or linear attentions [3,4]? Because such methods don't have such redundancy exist in the standard Transformer.\n\nWe appreciate the reviewer’s suggestion to apply our method to efficient transformer structures. Our method can be easily incorporated as a plug-and-play module into various vision transformer models. Following the reviewer's suggestion, we conducted experiments to test our strategy on Adaptive Token Sampler (ATS) [A]. The results, shown in the table below, indicate that using our method on top of ATS results in a 2% performance boost on So2Sat when testing on all training channels. The improvement is especially significant when only partial channels are available at test time, with a performance boost of up to 38%. In the case of CHAMMI, the average score sees a 3% improvement when incorporating our method.\n\n|\t\t\t\t| So2Sat Full\t| \tSo2Sat Partial |  \tCHAMMI\t|\n|----------------------------------------|--------------------|------------------------------|-------------------------------|\n| ATS\t\t\t\t| 58.46±0.59\t\t| \t9.30±1.89 | \t63.02±0.43\t\t|\n|**ATS + (TDL + CDL + DCS)**| **61.52±0.19**\t| \t**45.34±0.27**\t| **66.18±0.42**\t|\n\n[A] Fayyaz, Mohsen, et al. \"Adaptive token sampling for efficient vision transformers.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n\n\n\n\n\n> The paper needs proofreading. As instances:\nIn line 37, the authors used “to” two times repeatedly. Omit one.\nIn line 120, it is better to say “as noted in Section 1” rather than “as noted in the Introduction”.\nWhen citing a work, cite it after the author's name. For example, in line 98, reformat it as “ Recently, Bao et al. [18] …”.\n\nWe thank the reviewer for carefully reading our manuscript and providing suggestions to enhance the clarity of our paper! We will make the necessary revisions to address these points.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TR56DGBT3y", "reviewer": "Reviewer_K1xF", "summary": "This manuscript addresses the challenge of learning robust feature representations from datasets with heterogeneous channels, where the number and type of channels vary during training and testing. The authors propose a new DiChaViT, a model for Multi-Channel Imaging (MCI) based on the Vision Transformer (ViT) backbone. \n\nDiChaViT is proposed to improve existing MCI-ViT models like ChannelViT (*Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words, Bao et al., 2023*). Indeed, it re-elaborates ideas such as channel-wise patches and channel sampling, which contribute to the model's robustness in handling partial channel configurations at test time. Specifically, DiChaViT addresses previous models' tendency to learn redundant features across channels and patches, limiting their performance and generalizability.\n\nDiChaViT introduces three novelties to promote feature diversity and improve generalization across different channel configurations:\n\n1. **Channel Diversification Loss (CDL):** An additional loss term that encourages learning of distinct channel tokens by maximizing their Euclidean distance.\n2. **Token Diversification Loss (TDL):** An additional loss term that aims to diversify the features extracted from each patch token by promoting orthogonality between them (especially between patches belonging to different channels).\n3. **Diverse Channel Sampling (DCS):** A novel channel sampling strategy that selects channels based on their dissimilarity.\n\nOverall, they enhance the diversity of learned features, both between different channels and within individual tokens, situating the work in the broader context of learning disentangled representations.\n\nThese novel methods, combined with an MCI-ViT model, improve the classification accuracy on multi-channel fluorescence microscopy and satellite image data by 2-5 percentage points, making DiChaViT SOTA on this specific task.", "questions": "1. Regarding CDL:\n    - The orthogonal initialization of channel tokens introduces a strong prior/bias, especially considering that, in MCI, neighboring channels may contain very similar information. Have you tried to see what is the impact of imposing this initialization on tokens belonging to very similar channels? How do you expect the resulting tokens to be once the model is trained?\n    - Why did you use the Euclidean distance in the loss? Have you also tried other distances that are more appropriate for multi-dimensional objects?\n    - How are the anchors initialized? I believe it would be nice to briefly mention it since it is a key aspect of understanding CDL’s behavior.\n\n2. Why is cosine similarity between channel tokens used in DCS, whereas in CDL euclidean distance is employed instead? I could see potential reasons for that, but I would appreciate it if the authors included a brief note about that in the manuscript.\n\n3. In the first paragraph of section 4.4. the role of channel tokens is analyzed. In this context, why is the evaluation performed on ChannelViT instead of DiChaViT? Indeed, although CDL and the DCS are not applicable without channel tokens, TDL could still be used.\n\n4. The methods seem very flexible to different kinds of data and channel configurations. Have you tried to measure their effectiveness on hyperspectral images with several tens of channels?", "limitations": "- I expect the proposed losses CDL and TDL to struggle in the specific case of MCI data where groups of channels share similar semantic information. An example is spectral imaging, where neighboring (wavelength) bands often contain similar information. \nIn this case, I believe the following:\n    1. Regarding the Token Diversity Loss, the choice of introducing a larger penalty for the similarity between patches of different channels is counterintuitive. Indeed, this choice implicitly assumes that the channel information is more important than the spatial one. However, in the context of such images, I would argue that the opposite is instead true in most situations. Likely, two patches referring to the same spatial location but belonging to different channels usually exhibit more similar semantic features than two patches in different locations for the same channel.\n    2. Moreover, as already mentioned above, the approach proposed with CDL channel tokens introduces quite a strong prior on the channel token structure, which can be limiting for the datasets mentioned here.\nOverall, I see these methods as effective only on MC datasets in which channels contain sufficiently diverse information (but I’m happy to read more results and be proven wrong).\n\n- It would be interesting and meaningful to compare the learned features at different levels of the model (e.g., by visualizing the attention maps) to explicitly assess the disentangling effect brought by these methods.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The methods proposed are flexible and generalizable. They work independently of the channel configurations and have proven effective in different data domains (microscopy and satellite images). Moreover, they are also agnostic on the specific ViT architecture as they act at the input level. All three proposed novelties outline solid yet intuitive and elegant mathematical formulations.\n\nIn particular, Diversity Channel Sampling (DCS) provides a tangible improvement over the existing Channel Sampling approaches often used in MCI-ViT models. The proposed idea is also simple and effective, enabling content-aware channel sampling. \n\nOverall, the novel method proposed by the paper enhances the performance in classifying MC images (with a variable number of channels), making DiChaViT SOTA on this specific task.", "weaknesses": "- Despite improving classification accuracy over previous SOTA methods, the paper lacks consistent experiments (or even hypotheses) demonstrating how and why the introduced novelties lead to such performance (e.g., in-depth analysis on the effect of losses on the distribution and content of channel and patch tokens).  In this context, the analyses reported in Appendix B are a good starting point, but the current analyses are too superficial and do not provide enough insights. \n\n- Comparing the accuracy of ChannelViT and DiChaViT for different subsets of channels (as shown in Table 2) is an interesting way to assess the robustness of the model to different channel configurations. However, the conclusion that “DiChaViT consistently demonstrates improved robustness when some input channels are missing” is rather weak and probably not statistically significant due to the large standard deviations. Similarly, the ablation study is extensive and clear, but the assertions about the positive impact of CDL and TDL (see Fig. 4) are not backed up by sufficient statistical evidence. Indeed, \n    1. the number of runs is not sufficient to produce reliable uncertainty estimates; \n    2. the differences in performance for varying hyperparameters are not remarkable. \nFinally, it could be useful to introduce more metrics than simple accuracy.\nFor these reasons, I would suggest readjusting the emphasis of the claims to what can comortably be claimed.\n\n- There are some points in which the notation and naming is quite misleading:\n    1. f() is used both at line 141 to identify a map from a channel to its anchor, and at line 166 to identify a map from a patch to its corresponding channel. \n    2. In the CDL paragraph, the authors used c_i to identify the channel token, while in the DCS paragraph c_i is an alias for the channel, and f_i is instead the channel token. \n    3. In Fig.2 patch embeddings are referred to as p_i, which is the same notation used for the input patches in the paragraph about TDL. \n    4. In algorithm 1 mirror, I would not call the first sampled channel as c_n, since it seems the last channel sampled over a total of n. Something like c_k or $\\bar c_i$ would be more coherent.  \n    5. The name “channel tokens” for the vectors c_i that are concatenated to the patch tokens is a bit unfortunate. Indeed, as far as I am concerned, their role is closer to the one of “patch embeddings” p_i. As a result, I believe that referring to them as “channel embedding” would be a better choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Can you provide an in-depth analysis on the effect of losses on the distribution and content of channel and patch tokens?\n\nThank you for your valuable suggestions! Please refer to Fig. 1a for the effect on channel token content and Fig. 7 the effect on channel token distribution.  Additionally, the first two questions of the global response we further analyze model performance, and Fig. 1 of the rebuttal PDF reports the effect on patch tokens. In summary, our contributions result in less redundant information being captured, boosting performance across the board. Additional discussion in the global response.\n\n> Are the gains in Tab. 2 significant (they have large standard deviations)? \n\nWe believe that this comment stems from a misunderstanding of what the standard deviations pertain to in Tab. 2.  See Response 2 to Reviewer *gjzo*.\n\n> Are the gains of CDL and TDL significant?\n\nTab. 1 of the rebuttal PDF shows when including CDL in our full model (exp. 6 & 10) we boost performance by 1-2% with standard deviations from model variance of 0.2-0.4 in most settings, clearly demonstrating significant gains. The gains from TDL are similar (exp. 8 & 10).\n\n> The differences in performance for varying hyperparameters are not remarkable.\n\nThis is a benefit of our model as it helps show our model does not need extensive tuning when applied to new datasets.\n\n> Can you use metrics other than accuracy?\n\nClassification accuracy (So2Sat and Jump-CP) and F1 score (CHAMMI) were selected by prior work due to dataset restrictions [14,18].  E.g., CHAMMI’s tasks reveal differences and similarities among cellular phenotypes with a carefully designed evaluation protocol.  Changing this protocol to include other metrics would require collecting a new dataset.  \n\n> Notation clarifications/suggestions\n\nWe thank the reviewer for carefully reading our manuscript and providing useful suggestions! We will revise our manuscript to make these points clear.\n\n> 1. Regarding CDL: Have you tried to see what is the impact of imposing orthogonal initialization on tokens belonging to very similar channels? \n> Overall, I see these methods as effective only on MC datasets in which channels contain sufficiently diverse information\n\nOur datasets do contain very similar channels, yet models still report improved performance when using orthogonal initialization. E.g., So2Sat (18 channels) contains Red, Green, and Blue channels, which are known to be similar, and even show stronger correlation among Bands B6 - B8a (Fig. 2 of the rebuttal PDF). Yet, we still obtain improvements with CDL (see Fig. 3 main paper, and Tab. 1 of the rebuttal PDF).  We find that some of these similar channels do result in similar tokens in a trained model, i.e., if there is important information the model can overcome an orthogonal initialization.  Additionally, if two channels are similar, then the likelihood they end up learning the same features is high, especially if they are informative features.  Thus, a model may miss learning any unique information contained in only a single channel (resulting in high mutual information in Fig. 1a of the main paper).  As such, in the case of similar channels, it is even more important to encourage diversification so that more than just the redundant information is learned.\n\n> Regarding CDL: Why did you use the Euclidean distance in the loss?\n\nEq. 1 and L143 note that we use L2-norm + Euclidean distance following ProxyNCA++ [52]. This combination is equivalent to common alternatives like cosine similarity (e.g., as used in [A]). That said, we are happy to compare to other alternatives the reviewer would suggest, but also argue these implementation details do not significantly affect any conclusions we make.\n[A]  Learning transferable visual models from natural language supervision. Radford et al., ICML, 2021.\n\n> How are the anchors initialized?\n\nThe trainable channel anchors are also orthogonally initialized, encouraging diversity.\n\n> 2. Why is cosine similarity between channel tokens used in DCS, whereas in CDL euclidean distance is employed instead?\n\nIn CDL, we utilize the Euclidean distance with L2-Norm, as mentioned in Line 143, to be consistent with [52]. However, this is equivalent to using Cosine similarity as noted earlier. We will revise our manuscript to clarify this point.\n\n> 3. Can you use DiChaViT instead of ChannelViT when analyzing the role of channel tokens in section 4.4?\n\nWe conducted the same experiments using DiChaViT and presented the results in Table 2 of the rebuttal PDF. The conclusion remains the same as ChannelViT’s.\n\n> 4. Have you tried to measure their effectiveness on hyperspectral images with several tens of channels?\n\nWe combined the evaluation protocols used by prior work on MCI [14,18], resulting in a more comprehensive study than prior work. That said, we would be happy to explore additional suitable datasets suggested by the reviewer.\n\n> Don’t two patches referring to the same spatial location from different channels exhibit similar semantic features? \n\nWe found empirically that a larger penalty for patches from different channels compared to patches within the same channel improves performance. However, in Eq. 4  $\\lambda_s$ and $\\lambda_d$ balance the loss of tokens belonging to the same channels (first term) and different channels (second term). Thus, one can adjust these hyperparameters according to the dataset if needed.  That said, our experiments already show that similar channels can still benefit.\n\n> It would be interesting and meaningful to compare the learned features at different levels of the model (e.g., by visualizing the attention maps) to explicitly assess the disentangling effect brought by these methods.\n\nWe compare the learned features at different layers by visualizing the attention maps in Fig.1 of the rebuttal PDF.  We found that DiChaViT has more evenly distributed attention scores across channels, indicating each channel contributes more to the prediction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive comments and suggestions on the result analysis!
To get more insights, we conducted some additional analyses and attached the figures and tables in the PDF for the rebuttal. 

## 1. How Do Channel-Specific Attention Distributions Differ Between ChannelViT and DiChaViT? 
To investigate and interpret where the new setup is improving the performance, we look into the attention map of each channel and observe that ChannelViT relies heavily on a subset of channels to make a prediction, while DiChaViT pays attention more evenly across channels. 

Fig.1 of the rebuttal PDF, we calculate the attention scores of the [CLS] token to the patch tokens in a layer, such as layers 4, 8, and 12 (last layer), and then aggregate them by channel. This indicates that ChannelViT (top) relies more heavily on specific channels (e.g., *microtubules* and *nucleus*) for making predictions, while other channels (e.g., *protein* and *er*) are less considered. In contrast, DiChaViT (bottom) displays more evenly distributed attention scores across channels, indicating that each channel contributes more significantly to the model's predictions.

## 2. What is the benefit of having evenly distributed attention scores across channels?
To better understand why having more uniform attention scores across channels can enhance robustness and performance, we conducted a test where we removed one channel at test time. We observed that ChannelViT's performance dropped more than DiChaViT's.

For this setting, we trained ChannelViT and DiChaViT on CHAMMI, and then removed the *microtubules* channel during inference on HPA. This means we used only the remaining three channels (*protein*, *nucleus*, and *er*) for testing. As discussed in Section 1 above, ChannelViT relies on this channel to make predictions. As expected, we observed that ChannelViT's performance dropped more compared to DiChaViT (0.38 vs. 0.22), shown in the table below. This suggests that DiChaViT has a better ability to integrate and extract information from all the channels, making it less reliant on any single channel for accurate predictions.


|                    |Full Channels      |  Removing microtubules |   Avg Performance Drop    |
|-------------------|--------------------|------------------------|------------------------|
|ChannelViT         |   62.17±0.10       |   61.79±0.11           |   0.38                 | 
|DiChaViT           | 	63.59±0.12      |   63.28±0.13            |   0.22                 |


## 3. Where does DiChaViT improve performance?
We observe that the performance gets improved across classes, such as *Endoplasmic Reticulum* (from 910 to 1070 correct predictions) as shown in the confusion matrices below.

In this setting, we trained ChannelViT and DiChaViT on CHAMMI, and tested the models on 3 novel classes (*Cytosol*, *Endoplasmic Reticulum*, and *Nucleoplasm*). 

**ChannelViT’s Confusion Matrix**
|               | Cytosol | Endoplasmic Reticulum | Nucleoplasm |
|---------------|---------|-----------------------|------------|
| **Cytosol**                | 2047    | 962                   | 256         |
| **Endoplasmic Reticulum**  | 1126    | 910                   | 216         |
| **Nucleoplasm**            | 241     | 179                   | 3234        |


**DiChaViT’s Confusion Matrix**
|                         | Cytosol | Endoplasmic Reticulum | Nucleoplasm |
|-------------------------|---------|-----------------------|-------------|
| **Cytosol**             | **2109**    | 848                   | 285         |
| **Endoplasmic Reticulum** | 996     | **1070**                  | 186         |
| **Nucleoplasm**         | 265     | 158                   | **3393**        |



In addition,  their classification reports are shown below. We observe that DiChaViT gains improvement across all metrics (precision, recall, and f1-score) for all classes. This indicates that the model has generally become better when working on novel classes at test time. The most significant improvements are seen at *Endoplasmic Reticulum*.

**ChannelViT’s Classification Report**
| Class                    | Precision | Recall | F1-Score | Support |
|--------------------------|-----------|--------|----------|---------|
| Cytosol                  | 0.60      | 0.38   | 0.47     | 5318    |
| Endoplasmic Reticulum    | 0.44      | 0.24   | 0.31     | 3830    |
| Nucleoplasm              | 0.87      | 0.54   | 0.67     | 6018    |
| **Micro Avg**            | 0.68      | 0.41   | 0.51     | 15166   |
| **Macro Avg**            | 0.64      | 0.39   | 0.48     | 15166   |
| **Weighted Avg**         | 0.67      | 0.41   | 0.51     | 15166   |




**DiChaViT’s Classification Report**
| Class                    | Precision | Recall | F1-Score | Support |
|--------------------------|-----------|--------|----------|---------|
| Cytosol                  | 0.63      | 0.40   | 0.49     | 5318    |
| Endoplasmic Reticulum    | 0.52      | 0.28   | 0.36     | 3830    |
| Nucleoplasm              | 0.88      | 0.56   | **0.69**     | 6018    |
| **Micro Avg**            | 0.71      | 0.43   | **0.54**     | 15166   |
| **Macro Avg**            | 0.67      | 0.41   | **0.51**     | 15166   |
| **Weighted Avg**         | 0.70      | 0.43   | **0.53**     | 15166   |

Meta-review: The paper introduces DiChaViT an enhanced modeling of hyper-spectral images by increasing the diversity of multiple channels using orthogonal channel embedding, token diversification, and diverse channel sampling strategy. The proposed pipeline boosts the classification performance on multi-channel image settings such as satellite and microscopy imaging by encoding MCI into a more robust embedded feature representation. The paper has received all accept reviews. There is consensus among reviewers that the utility of DiChaViT is novel and it is well compared with several baseline approaches and can generalize to different data domains. Overall the paper is well reasoned, motivated and presented which opens a new direction on MCI. Further, the details of the proposed DiChaViT are provided in comprehensive manner and thorough experiments with ablation studies are demonstrated. Some concerns were raised by reviewers on (a) limitation of the proposed method where all channels are needed for training which raised the question about channels that are not seen during training, (b) sensitivity of composite loss hyper-parameters and their transferability to other dataset distribution, (c)  Initialization of channel anchors, (d) sensitivity on temperature parameter, (e) the rational and interpretability of DiChaViT’ design and how it compares with baseline approaches, (f) ablation on each losses individually, (g) intuition behind patch token orthogonality, and (h) notation typos and naming of variables can be miss leading. Authors have addressed most of reviewers concerns during rebuttal. The AC finds strong support from reviewers to merit the paper for publication. It is highly encouraged for the authors to take the advantage on the discussions raised by reviewers as highlighted above for their final revision using from both pre-/post-rebuttal phase comments.
Decision: Accept (poster)