Enhancing Feature Diversity Boosts Channel-Adaptive Vision Transformers

Authors: chau  pham
bryan a plummer
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 05:46:20

Abstract: Multi-Channel Imaging (MCI) contains an array of challenges for encoding useful feature representations not present in traditional images. For example, images from two different satellites may both contain RGB channels, but the remaining channels can be different for each imaging source. Thus, MCI models must support a variety of channel configurations at test time. Recent work has extended traditional visual encoders for MCI, such as Vision Transformers (ViT), by supplementing pixel information with an encoding representing the channel configuration. However, these methods treat each channel equally, i.e., they do not consider the unique properties of each channel type, which can result in needless and potentially harmful redundancies in the learned features. For example, if RGB channels are always present, the other channels can focus on extracting information that cannot be captured by the RGB channels. To this end, we propose DiChaViT, which aims to enhance the diversity in the learned features of MCI-ViT models. This is achieved through a novel channel sampling strategy that encourages the selection of more distinct channel sets for training. Additionally, we employ regularization and initialization techniques to increase the likelihood that new information is learned from each channel. Many of our improvements are architecture agnostic and can be incorporated into new architectures as they are developed. Experiments on both satellite and cell microscopy datasets, CHAMMI, JUMP-CP, and So2Sat, report DiChaViT yields a 1.5 - 5.0% gain over the state-of-the-art. Our code is publicly available at https://github.com/chaudatascience/diverse_channel_vit.

Main paper: 
 introduction: 
 Most visual encoders assume they are provided with a fixed-channel representation as input (e.g., they take RGB inputs as input at train and test time) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] . However, many applications find a variety of imaging techniques beyond just the traditional RGB channels beneficial. For example, satellite images or sensors onboard a robot often contain an infrared camera in addition to traditional RGB, and microscopes can also host a significant range of potential imaging channels [11] [12] [13] [14] [15] [16] [17] . Thus, Multi-Channel Imaging (MCI) models aim to learn good feature representations from datasets with heterogeneous channels, where the number and type of channels can vary for each input at test time. Training a model that is robust to changes in channel configurations can save time and resources as only a single model needs to be learned, while also helping to prevent overfitting in small datasets through transfer learning [14] . Prior work proposed methods to make MCI models robust to missing channels by randomly masking them during training [18] . As shown in Fig. 1 (a) left and (b) top, this results in redundancies being learned across channels during training rather than encoding new information. A consequence of this repetition is a model focused on learning strong cues that are easy to identify, making it less capable of learning unique and/or challenging cues within each channel. Figure 1 : Comparison of the redundant information learned by different models on the HPA dataset in CHAMMI [14] ). (a) Measures the mutual information between the channel tokens, which captured the configuration of channels in an image. Note we gray out the diagonal for better visualization. We find ChannelViT tokens have high mutual information, which suggests significant redundancy exists across channels [34, 35] . In contrast, DiChaViT has little mutual information as each channel is encouraged to learn different features. (b) We compute attention scores of the [CLS] token to the patch tokens in the penultimate layers and aggregate them by channel. ChannelViT (top) relies on certain channels (e.g., microtubules and nucleus) to make predictions and less on other channels (e.g., protein and er). In contrast, DiChaViT demonstrates more evenly distributed attention scores across channels, suggesting that each channel contributes more to the model's predictions.To address this limitation, we propose a Diverse Channel Vision Transformer (DiChaViT) that aims to balance the robustness in different MCI configurations, which may cause redundancy, with a need to learning diverse and informative features. First, we include a Channel Diversification Loss (CDL), a regularization term that encourages a special channel token, which represents the presence of a channel in the input data, to be distinct from the other channel tokens. As shown in Fig. 1 (a) right, this reduces repeated information across our model's channels. However, this can still result in similar features being encoded for each image patch. Thus, our Token Diversification Loss (TDL) aims to directly diversify the features learned for each patch token as shown in the bottom of Fig. 1(b ) by encouraging that each patch token is orthogonal to the others. Finally, rather than a uniform random channel masking strategy as used in prior work [18, 19] , we introduce Diverse Channel Sampling (DCS), in which we select channels based on their dissimilarity, further promoting feature diversity. We observe that promoting a more diverse representation enables each channel to contribute more to the final prediction, leading to a performance boost of up to 5.0% in downstream MCI tasks. Fig. 2 provides an overview of our approach.The work that is closest in spirit to ours are methods that are designed to learn disentangled representations [20] [21] [22] [23] [24] [25] [26] [27] , e.g., learning features aligned to a given set of attributes [28] [29] [30] . These methods have shown a trade-off between the strength of the disentanglement and the downstream tasks performance [31] [32] [33] . This is due, in part, to the fact that many attributes these methods aim to disentangle are correlated with each other, making it challenging to know what features relate individually to each attribute. However, unlike these tasks, MCI methods do not focus only on disentangling features across channels. Instead, they must capture some redundant information to be robust to missing channels while simultaneously learning features that may only arise in a subset (or even a single) channel. In other words, in MCI some redundancy is desirable across channels even if we could learn perfectly disentangled representations. In addition, many methods in disentangled representation learning assume the attributes to separate are labeled, but there are no labeled attributes in MCI. Instead, DiChaViT must automatically decide what to capture in multiple channels while still learning important channel-specific information.We summarize our contributions below:• We propose DiChaViT as a solution to enhance feature diversity and robustness in MCI-ViTs, boosting classification accuracy by 1.5 -5.0% over the state-of-the-art on three diverse MCI datasets: CHAMMI [14] , JUMP-CP [12] , and So2Sat [17] .  
related work: 
 Convolutional-based models for multi-channel imaging. Researchers have been developing convolutional-based models to keep pace with the evolving landscape of multi-channel imaging data. Bhattacharyya et al. [36] introduced IRFacExNet, which utilizes depth-wise convolutions to merge channel-wise features from infrared thermal images. Jiang et al. [37] introduced a double-channel CNN that takes into account the correlation between input channels in aerial images. This approach employs a separate sub-network for each group of channels and then performs feature fusion to aggregate features across channels. Siegismund et al. [38] presented DCMIX to work with images with many channels based on imaging blending concepts. While these methods can be used for MCI, they are not designed to work on varying input channels. In a recent study, Chen et al. [14] introduced and adapted channel-adaptive models based Depthwise convolutions, TemplateMixing [39] [40] [41] ,and HyperNets [42] . These models incorporate their adaptive interface in the first layer of an otherwise shared ConvNeXt model [8] . While these methods provide a strong baseline, they find settings where some channels are missing during inference challenging. In our work, we aim to improve MCI model robustness by improving the diversity of learned features.Vision Transformers for multi-channel imaging. Vision transformers (ViT) [43] have natural advantages when dealing with multiple channels, especially when the number of channels varies. ViTs treat image modeling as sequence-to-sequence problems, allowing them to be flexible in handling different numbers of image tokens. Nguyen et al. [44] introduced variable tokenization and variable aggregation, in which they divided each input channel independently into patches and then aggregated the patch features across channels using learnable queries. Tarasiou et al. [45] proposed TSViT, which incorporates a tokenization scheme and temporal position encodings to process Satellite Image Time Series. In a relevant work, Zhou et al. [46] introduced FAN, a channel reweighting design aimed at adjusting channel features based on the observation that some channels capture more significant information than others. In the medical domain, Hatamizadeh et al. [47] proposed UNETR that utilized a transformer encoder followed by a skip-connected decoder for 3-D medical image segmentation. Recently, Bao et al. [18] proposed ChannelViT that processes each input channel independently via a shared linear projection and incorporates a learnable channel embedding for preserving channel-specific features. In addition, the authors proposed Hierarchical Channel Sampling (HCS), a regularization technique applied to the input channels to boost robustness and reduce training time. ChannelViT outperforms standard ViTs in classification tasks and demonstrates its generalization ability when only a subset of the trained channels is available during inference. In a similar work, Bourriez et al. [15] introduced ChAda-ViT, a channel adaptive attention technique for handling heterogeneous microscope images. However, these methods do not adequately model the unique properties of each channel type, resulting in harmful redundancies, whereas we boost the diversity of features across channels to enhance the robustness of MCI-ViT models. 
encouraging diverse representations in multi-channel vits: 
 Given a multi-channel image (MCI) X containing channels c i ∈ C X , our goal is to train a model M that takes our input image X as input to make its predictions. Following [15, 18] , we consider the MCI setting where M has seen all the channels we expect to see during inference, i.e., C X ⊆ C M . We leave the exploration of handling novel channels during inference for future work, as it presents significant challenges, including establishing meaningful connections between existing and new channels, and identifying informative channel weights in the presence of domain shifts. In our setting, since we do not know what C X we may see during inference, prior work has focused primarily on exploring methods that are robust to different choices of C X by encouraging M to redundancies across channels (e.g., [14, 15, 18] ). Specifically, they begin with a base ViT encoder [48] that uses each channel-specific image patch p i as input. Each image patch is passed through a shared patch projection layer and concatenated with its corresponding channel token ch i . Hierarchical Channel Sampling (HCS) [18] encourages robustness to missing channels by randomly masking some channels during training to ensure key information can be captured in multiple channels. However, as noted in the Introduction, this can be harmful when M does not balance this repetitive feature learning to also capture distinctive channel-specific information.As illustrated in Fig. 1 , DiChaViT aims to better balance repetitive and distinct feature learning through three major components. First, we use a Channel Diversification Loss (CDL) to learn diverse representations to help prevent feature collapse in the channel tokens (Sec. 3.1). Second, our Token Diversification Loss (TDL) encourages patch tokens to also learn distinct features (Sec. 3.2). Finally, Diverse Channel Sampling (DCS) promotes robustness to missing channels while also encouraging that new features are also learned during training (Sec. 3.3). These components enable our approach to balance repetitive and channel-specific feature learning (overview in Fig. 2 ).Recall that in Fig. 1 (a), learned channel tokens ch i from prior work show high mutual information, indicating these tokens are not well-separated. Following [41, [49] [50] [51] , we partly mitigate this issue by replacing the random initialization of ch i used by prior work [15, 18] with an orthogonal initialization.To further encourage the diversity in the features, we introduce Channel Diversification Loss (CDL) for increased separation between the channel tokens (Fig. 2(a) ). Inspired by ProxyNCA++ [52] , the idea is to use a learnable vector (i.e., an orthogonally initialized channel anchor) to represent each channel in the input image during training. We promote diversity in the channel tokens by pulling channel features toward their corresponding anchors while pushing them away from all other anchors.A key benefit of this approach is that the anchors prevent channel tokens from collapsing while still allowing for flexibility in learning useful representations.Formally, we denote A as the set of all channel anchors, t CDL as the temperature, and ∥ • ∥ 2 as the L2-Norm. We start by initializing the channel tokens ch i and their channel anchors orthogonally. Then, we apply CDL as follows:EQUATIONwhere g(ch i ) is a function that returns a corresponding channel anchor for channel token ch i , and d(ch i , g(•)) is the squared Euclidean distance between channel token ch i and an anchor. In Eq. 1, the numerator calculates the distance of a channel token to its anchor, while the denominator computes all these distance pairs of the channel token to all the channel anchors. When the temperature value t CDL is set to 1, we get a standard Softmax function. Lowering the temperature can lead to a more focused and sharp probability distribution, but we found that the results are not very sensitive to the value of t CDL . Thus, we simply use a fixed temperature t CDL of 1/14 ≈ 0.07.MCI-ViT models like ChannelViT [18] , ChAda-ViT [15] use a shared linear projection to extract features independently from each input channel in the image rather than using separate projections for each channel. With the shared projection, only the common features across channels are retained, while other channel-specific information is filtered out, which helps to reduce overfitting. However, this design can also produce similar representations for all patch tokens. This is not ideal because each patch may contain unique information that would be ignored. In our approach, we also leverage this shared projection, but we enhance it with Token Diversification Loss (TDL), a regularization applied to the patch token features to enhance the diversity of features learned by each patch in the input image (see Fig. 2 (b) for an overview). Specifically, we enforce an orthogonality constraint on the tokens to ensure that each token is orthogonal to the others. Additionally, we take into account the token type information to differentiate between tokens from the same channels and across channels.The main idea is to make features from different channels more distinct while allowing for a certain level of similarity among features within the same channel.Let p i be the input patch at position i, and W proj be the shared linear projection at the first layer. We denote t i = W proj • p i as the patch feature token of p i , T = t i i=1,2,... as the set containing all patch feature tokens in the input image, and h(t i ) as a function that returns the corresponding channel for input patch p i . We devise a unified loss function for each input image as follows:EQUATIONEQUATIONEQUATIONwhere ⟨•, •⟩ represents the cosine similarity, | • | denotes an absolute value, and N s , N d are the numbers of patch token pairs in the two equations respectively. Eq. 2 calculates the average cosine similarity of all feature token pairs in the same channels, while Eq. 3 calculates the average of all feature token pairs from different channels. The two losses are combined with weights λ s and λ d to balance the constraint of tokens belonging to the same channels (first term) and tokens belonging to different channels (second term), to form the final loss L TDL in Eq. 4. Our goal is to encourage each patch token to be orthogonal to each other to promote the diversity of patch tokens.Bao et al. [18] introduced HCS to reduce the training time and improve the robustness of the model. The main concept is to randomly drop some input channels and train the model only on the remaining channels. In the same spirit, we propose a novel method, Diverse Channel Sampling (DCS), to sample a more diverse subset of channels during training (Fig. 2(c) ). Similar to HCS, we start by randomly sampling a number k, which is the size of a subset of channels to train on. However, while HCS samples k channels randomly, DCS first samples an anchor channel c k . Then, we select other k -1 channels that are dissimilar to the anchor channel. This idea shares similarity with Channel DropBlock [53] , where a set of similar channels in a CNN layer is masked out to disrupt coadapted features. However, instead of keeping a fixed number of feature map channels as in Channel DropBlock, DCS selects a flexible number of input channels for each sampling. The procedure of DCS is outlined in Algorithm 1.In practice, Algorithm 1 can be applied to a batch of images for faster sampling. We use channel token ch i to represent the channel feature f i . Refer to Sec. 4.4 and Tab. 5 for more discussion on choices of f . The temperature t DCS controls the sharpness of the probability distribution. With a large t DCS , DCS reduces to HCS, while with a small t DCS , DCS selects a random subset of channels that are the least similar to the anchor channel.The final loss consists of the primary loss for the specific task (e.g., cross-entropy for classification), Channel Diversification Loss (CDL) applied to channel tokens, and Token Diversification Loss (TDL) used on patch tokens. These terms work together to promote diversity in channel and patch token features, resulting in a more robust model, as shown in Eq. 5:EQUATIONwhere λ CDL is a weight to balance CDL. Note that TDL is balanced by λ s and λ d in Eq. 4. = [⟨f k , f i ⟩ ,...], ∀i ̸ = k (s ∈ R m-1) Convert 1 -s to probability using softmax with temperature t DCS : p = softmax((1 -s)/t DCS ) (p ∈ R m-1 ) Sample k -1 distinct channels from m -1 channels with probability p Combine the k -1 channels with channel c k to create a set of k sampled channels. Output : Image X with only k sampled channels 4 Experiments 
experimental setup: 
 Baseline methods. We adopt the following baseline methods.• DepthwiseViT [14] utilizes a depthwise convolution layer to independently filter each input channel. The resulting features are averaged to create a new feature representation, which is then fed into a ViT backbone. • TemplateMixingViT [39, 40] generates weights for each channel by learning a linear combination of shared, learnable parameter templates. These weights are formed into a patch project layer, followed by a ViT backbone. • HyperNetViT [42] employs a neural network (e.g., MLP) to independently generate weights for each channel, which are then concatenated to form a patch projection layer. This patch projection layer is subsequently used in a ViT backbone. • ChAda-ViT [15] uses a shared projection layer to extract features from each channel separately, then feeds these tokens, together with their corresponding positional embeddings and channel embeddings, into a ViT backbone. • ChannelViT [18] is the same general architecture as ChAda-ViT, but also employs Hierarchical Channel Sampling (HCS) during training.Implementation details. As HCS proves robust in multi-channel imaging [18] , we incorporate this technique for DepthwiseViT, TemplateMixingViT, and HyperNetViT to ensure a fair comparison in these adaptive baselines used by Chen et al. [14] 1 . For ChannelViT and ChAda-ViT, due to their similarity (primarily a difference in whether HCS is included), we use the implementation from [18] for both methods 2 . All baselines utilize a ViT small architecture (21M parameters) implemented in DINOv2 [54] as the backbone 3 . We use AdamW optimizer [55] to train the models, minimizing cross-entropy loss on JUMP-CP and So2Sat, and proxy loss on CHAMMI. For the learning rate, we use a scheduler with linear warmup and cosine decay. Refer to Appendix Sec. A for details.Metrics. We evaluated the methods by calculating their top-1 classification accuracy on the So2Sat [17] and JUMP-CP [12] datasets. For CHAMMI [14] , we used the evaluation code 4 provided by the authors, in which a 1-Nearest Neighbour classifier is used to predict the macro-average F1-score for each task separately. We report the average score on WTC and HPA, and present the detailed results in Tab. 7 of the Appendix.CHAMMI [14] consists of varying-channel images from three sources: WTC-11 hiPSC dataset (WTC-11, three channels), Human Protein Atlas (HPA, four channels), and Cell Painting datasets (CP, five channels). The three sub-datasets contain a total of 220K microscopy images, of which 100K images are for training and the rest for testing across various tasks. The models are trained to learn feature representation and then evaluated on domain generalization tasks.JUMP-CP [12] comprises images and profiles of cells that were individually perturbed using chemical and genetic methods. Our experiments focus on the compound perturbation plate BR00116991, which contains 127K training images, 45K validation images, and 45K test images. Each image has eight channels, with the first five being fluorescence and the remaining three containing brightfield information. The dataset consists of 161 classes, including 160 perturbations and a control treatment.So2Sat [17] contains synthetic aperture radar and multispectral optical image patches from remote sensing satellites. Each image in the dataset has 18 channels, of which eight Sentinel-1 and 10 Sentinel-2 channels. The dataset consists of 17 classes, each representing a distinct climate zone. We use the city-split version of the dataset, which includes 352K training images and 24K test images.Tab. 1 shows that DiChaViT outperforms the state-of-the-art ChannelViT by up to 5.0% points on all three datasets: CHAMMI [14] , JUMP-CP [12] , and So2Sat [17] . For JUMP-CP and So2Sat, we consider two scenarios: tested on all training channels (denoted as "Full") and tested on a subset of channels (denoted as "Partial"). In the full channels setting, our model shows a 1.5 -2.5% point improvement compared with other baselines on JUMP-CP and So2Sat. When tested on partial channels, DiChaViT demonstrates its robustness by achieving a 1.5% improvement compared with the baselines. This demonstrates that diversifying feature representations in MCI-ViT models boosts both performance and robustness.Tab. 2 presents a detailed evaluation of DiChaViT and the best baseline model, ChannelViT, when tested on partial channels of the JUMP-CP dataset (with a total of eight channels). For the partial channel evaluation, we exclude some of the channels that the models were trained on and only test the model on the remaining channels. Then, we calculate the average accuracy across all combinations, e.g., testing on seven channels, as shown in column "7", involves averaging the results of C 7 8 = 8 combinations (refer to Tab. 9 in the Appendix for detailed results). Our findings consistently show that DiChaViT demonstrates improved robustness when some input channels are missing.To provide more insight into the contribution of each component of DiChaViT, Tab. 3 presents the model's performance when a component is removed. The results highlight the critical role of the DCS component, as its removal has the most detrimental effect on performance, particularly in the Partial setting, with a decrease of 16% and 30% points on JUMP-CP and So2Sat, respectively. The absence of CDL and TDL results in similar performance drops across all datasets. The highest scores are achieved when all components are integrated, indicating that each component plays a crucial role in the model's design. Refer to Tab. 8 in the Appendix for a comprehensive analysis.The role of channel tokens. In MCI-ViT models such as ChannelViT [18] and ChAda-ViT [15] , channel tokens play a crucial role in learning channel-specific features, particularly when dealing with multiple channels where each contains unique information. To assess the impact of channel tokens, we compared the performance of DiChaViT on JUMP-CP and CHAMMI with (orange bars) and without channel tokens (blue bars), as shown in Fig. 3 . The results indicate that DiChaViT demonstrates significant improvements with channel tokens, resulting in 8.0% and 15.0% point increases on JUMP-CP and CHAMMI, respectively, highlighting their importance.Orthogonal initialization of channel tokens boosts performance. As shown in Fig. 3 , using orthogonal initialization (green) provides a 1.0% gain on JUMP-CP and CHAMMI. This may suggest that by initializing the weights orthogonally, the model can more effectively capture diverse patterns within the data, resulting in boosting its overall performance.Impact of λ CDL (Eq. 5) in CDL. Fig. 4 (a) and (b) show the performance of DiChaViT (mean and std) across different values of λ CDL on So2Sat and CHAMMI datasets. We can observe that selecting a value that is too large is not beneficial to the performance. It is worth finding a suitable value for λ CDL . On the So2Sat, the best performance is achieved with λ CDL = 0.001, while the suitable value for CHAMMI is 0.1.Ablation on TDL (Eq. 4). Fig. 4 (c) reports the performance of our model across different ratios of λ d and λ s in TDL. We set a fixed value of λ s at 0.05 and vary λ d . We observe that using a larger λ d compared with λ s leads to better performance for DiChaViT. This suggests that knowing which channel a token comes from, i.e., the same or different channel, is necessary. The results indicate imposing stricter constraints on tokens from different channels compared with tokens from the same channel obtains the best performance. Tab. 4 shows the impact of each component in TDL. We see that considering only tokens within the same channels (denoted by "Only L s ") is insufficient, resulting in a significant drop in performance. In contrast, using both L s and L d in TDL yields the best performance of DiChaViT.Channel feature f in DCS. Tab. 5 compares the performance of using channel tokens (ch i ) and patch tokens (i.e., image patches after passing through the projection layer) to compute the similarity score for sampling in Algorithm 1 (line 3). We observe that using channel tokens gains better performance on So2Sat and CHAMMI datasets. Note that while channel tokens are shared across all input images, patch tokens differ for each input image.Impact of temperature on DCS. Tab. 6 shows the effect of temperature t DCS used in Algorithm 1 on DCS. When t DCS is set to a very small value, as reported in the first column (denoted as "≈ 0"), DCS selects channels with the lowest similarity scores to the anchor channel. Conversely, when t DCS is assigned a large value, denoted as "HCS" in the last column, DCS is reduced to HCS [18] , meaning Figure 5 : Comparison of DCS and HCS [18] in terms of the frequency (%) each channel is sampled during training on So2Sat. Unlike HCS, which provides a uniform distribution for all channels (red dashed line), some channels in DCS are trained much more than others (blue bars). For example, Real Lee-Cov channel (rightmost) is sampled twice as much as Band B8a (first bar). that it selects the subset of channels randomly. We find that always selecting the lowest similar channels (≈ 0) does not yield the best performance. Instead, setting the temperature to t DCS = 0.1 produces favorable results for both So2Sat and CHAMMI.DCS and HCS on the distribution of number sampling of the channels. Fig. 5 compares the number of times each channel is sampled during training with DCS (blue bars) and HCS [18] (red dashed line). DCS offers a different distribution for its channels compared with HCS, with some channels receiving more training than others. For example, Real part of Lee-filtered covariance matrix (Real Lee-Cov) in the last bar, is sampled twice as frequently as Band B8a channel (first bar). 
conclusion: 
 In this paper, we present DiChaViT, a model aimed at enhancing feature diversity and robustness in Multi-Channel Imaging (MCI) ViTs. First, we introduce Diverse Channel Sampling, a novel channel sampling strategy that encourages the selection of more distinct channel sets during training, thereby promoting feature diversity. Additionally, DiChaViT incorporates Token Diversification Loss on the patch tokens and Channel Diversification Loss for channel tokens to further diversify the features learned in MCI-ViTs. Our experiments demonstrate a 1.5 -5.0% point improvement over state-of-the-art methods on satellite and microscopy imaging datasets. Many of our enhancements are not tied to any specific architecture and can be incorporated into new architectures as they are developed. DiChaViT represents a promising advancement in addressing the challenges associated with MCI, paving the way for more effective MCI-ViT models.Broader Impacts and limitations. The development of DiChaViT represents an advancement in MCI, with potential positive impacts such as improved medical diagnosis and accelerated healthcare research. Additionally, its versatility in satellite imaging holds promise for environmental monitoring. However, there are also potential negative impacts, including the risk of bad actors using this research to develop harmful applications, such as invasive surveillance systems. This highlights the importance of ethical considerations and responsible deployment. One of the limitations of our work is that it is not designed to handle novel channels. Generalizing to unseen channels is challenging because it requires establishing a connection between existing and new channels. This is further complicated in the presence of domain shifts, which makes finding the informative channel weights even more difficult. Thus, investigating techniques to adapt to new channels at test time is a promising research direction in MCI. In addition, our approach requires extra hyperparameter tuning, which may necessitate additional compute resources. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]We have a broader impact statement.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]We cite the original papers and provide the links to their codes. The license and terms are properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]We plan to release the code with a document on how to use it.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] This paper does not use crowdsourcing.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We utilize a ViT small architecture (21M parameters) implemented in DINOv2 [54] as the backbone for all the baselines 5 . Specifically, we use ViT-S/16 (patch size of 16) on CHAMMI and JUMP-CP, and ViT-S/8 (patch size of 8) on So2Sat. The AdamW optimizer [55] is used to train the models, minimizing cross-entropy loss on JUMP-CP and So2Sat, and proxy loss on CHAMMI.CHAMMI dataset [14] . The goal of CHAMMI is to train a model to learn the feature representation for the input image. Thus, we use the [CLS] token at the final layer as the feature representation and train the model to minimize the proxy loss [52] . We then evaluate the model on various tasks following the evaluation code provided by the authors, in which a 1-Nearest Neighbour classifier is used to predict the macro-average F1-score for each task separately 6 . The channel-adaptive interfaces are adapted from the author's implementation code 7 . Besides the model, we incorporate the same data augmentation as introduced by the authors, such as thin-plate-spline (TPS) transformations [56] . We train each model for 60 epochs with a learning rate of 0.00004, and a batch size of 64.JUMP-CP [12] and So2Sat [17] datasets. Following Bao et al. [18] , the learning rate is warmed up for the initial 10 epochs, peaking at 0.0005 after which it will gradually decay to 10 -6 following a cosine scheduler. We also apply a weight decay of 0.04 to the weight parameters, excluding the bias and normalization terms to mitigate overfitting. Additionally, we use the same data augmentation as used in the code provided by the authors. To get the final prediction, we pass the Transformer encoder's representation for the [CLS] token into a classifier head to predict the probability of each class. We train each model for 100 epochs, with a batch size of 64 on JUMP-CP, and 128 on So2Sat. We adapt the code provided by the authors [18] for the baselines in our work 8 .Compute resources. In this study, experiments were conducted on So2Sat and CHAMMI using a single NVIDIA RTX (48GB RAM) and three Intel(R) Xeon(R) Gold 6226R CPUs @ 2.90GHz. For experiments on JUMP-CP, two NVIDIA RTX A6000 GPUs and six Intel(R) Xeon(R) Gold 6226R CPUs @ 2.90GHz were utilized.Extended main results. Tab. 7 shows an extension of the main resulting table in the main paper (Tab. 1), where we include CNN-based (ConvNeXt backbone [8] ) models from [14] . To ensure a fair comparison, we adjust the number of layers in these CNN-based models so that all models in Tab. 7 have approximately 21M parameters. We can observe that in general, DiChaViT outperforms CNN-based and ViT-based models on the three datasets.Extensive ablation results on DiChaViT. Tab. 8 extends Tab. 3 in the main paper to have a better understanding of the individual effects and contributions of each of the losses. We observe that adding DCS helps improve the performance (e.g., by 4% on CHAMMI), and robustness of the model, especially when tested on partial channels (a boost of 35% on So2Sat Partial). Similarly, TDL and CDL also show improvement across the three datasets. For example, TDL improves the performance by 2.5% on CHAMMI and 1.7% on So2Sat on full channels.Effect of CDL on channel token distributions. Fig. 6 illustrates the distributions of channel tokens with (blue) and without (red) CDL. Each subplot presents the distribution of a trained channel token on the CHAMMI dataset. We observe that CDL results in more flattened distributions with more non-zero values in the channel tokens.Attention scores of the [CLS] token to the patch tokens at different layers. Fig. 7 shows an extended version of Fig. 1(b ) in the main paper, where we calculate the attention scores of the [CLS] token to the patch tokens at layers 4, 8, and 12 (the penultimate layer), and then aggregate them by channel. This indicates that ChannelViT (top) relies more heavily on specific channels (e.g., microtubules and nucleus) for making predictions, while other channels (e.g., protein and er) are less considered. In contrast, DiChaViT (bottom) displays more evenly distributed attention scores across channels, indicating that each channel contributes more significantly to the model's predictions. Table 7 : Test accuracy of channel-adaptive models across multi-channel datasets. DiChaViT performs better than other CNN-and ViT-based baselines. It shows overall better performance on CHAMMI, especially on Allen and CP, and a 1.5 -2.5% improvement on JUMP-CP and So2Sat. "Full" refers to testing on all channels, while "Partial" means testing on a subset of channels. We use Sentinel-1 channels for So2Sat, and fluorescence channels for JUMP-CP.Architecture CHAMMI [14] JUMP-CP [12] to make predictions and less on other channels (e.g., protein and er). In contrast, DiChaViT demonstrates more evenly distributed attention scores across channels, suggesting that each channel contributes more to the model's predictions. Leave-one-channel-out at test time. In Tab. 9, we provide individual channel combination results when using seven channels (of eight) of the JUMP-CP dataset for inference. This corresponds to the details in column "7" from Tab. 2 in the main paper, representing C 7 8 = 8 different channel combinations. For each combination, we report the mean and std of the models computed over three runs. Our results demonstrate that DiChaViT gets 1 -2% better performance for each combination while also providing more stable results (i.e., smaller model variance) than baseline ChannelViT.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] We release the code and instructions on downloading the datasets at https://github. com/chaudatascience/diverse_channel_vit. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We provide the implementation details with computer resources in Section A in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).The research conducted conforms with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://github.com/chaudatascience/channel_adaptive_modelshttps://github.com/insitro/ChannelViThttps://github.com/facebookresearch/dinov2https://github.com/broadinstitute/MorphEmhttps://github.com/facebookresearch/dinov2https://github.com/broadinstitute/MorphEmhttps://github.com/chaudatascience/channel_adaptive_modelshttps://github.com/insitro/ChannelViT 


Decision: The paper introduces DiChaViT an enhanced modeling of hyper-spectral images by increasing the diversity of multiple channels using orthogonal channel embedding, token diversification, and diverse channel sampling strategy. The proposed pipeline boosts the classification performance on multi-channel image settings such as satellite and microscopy imaging by encoding MCI into a more robust embedded feature representation. The paper has received all accept reviews. There is consensus among reviewers that the utility of DiChaViT is novel and it is well compared with several baseline approaches and can generalize to different data domains. Overall the paper is well reasoned, motivated and presented which opens a new direction on MCI. Further, the details of the proposed DiChaViT are provided in comprehensive manner and thorough experiments with ablation studies are demonstrated. Some concerns were raised by reviewers on (a) limitation of the proposed method where all channels are needed for training which raised the question about channels that are not seen during training, (b) sensitivity of composite loss hyper-parameters and their transferability to other dataset distribution, (c)  Initialization of channel anchors, (d) sensitivity on temperature parameter, (e) the rational and interpretability of DiChaViT’ design and how it compares with baseline approaches, (f) ablation on each losses individually, (g) intuition behind patch token orthogonality, and (h) notation typos and naming of variables can be miss leading. Authors have addressed most of reviewers concerns during rebuttal. The AC finds strong support from reviewers to merit the paper for publication. It is highly encouraged for the authors to take the advantage on the discussions raised by reviewers as highlighted above for their final revision using from both pre-/post-rebuttal phase comments.

Reviews: review: paper_summary: paper_summary: Multiple channel imaging (MCI) is widely used in different application domains, ranging from medical image analysis to satellite imagery. Each channel can contain information that is orthogonal compared to other channels, which can be useful in downstream tasks. Obtaining such orthogonal signals from individual channels without extracting repetitive information is challenging. In this work, three contributions are made to improve the channel diversity in vision transformer (ViT) class of models. Firstly, to enhance separation between channel specific tokens a contrastive loss, known as channel diversification loss (CDL) is used. Next, patch-level token diversification loss (TDL) ensures extraction of patch-level features that are not dependent on the channels. Finally, masked-training of sorts across channels is performed using diverse channel sampling (DCS). These components result in a composite loss function that is used to demonstrate competitive performance on three MCI datasets. The proposed diverse channel ViT (DiChaViT) method outperforms relevant baseline methods, yielding considerable performance improvements.
strength_weakness: strength_weakness: * **Unseen channels during training**: One of the key limitations of this work, which is also briefly acknowledged by the authors, is that the method is only focused on dealing with missing channels only at inference. Meaning, all channels have to be seen by the model during training. This might not always be the case. I would like the authors to elaborate more on why this is not considered in this work. I would be happy with an explanation in terms of which of the contributions would breakdown for unseen channels. Just to be clear, I am not seeking additional experiments. 

* **Balancing loss function**:The composite loss function has multiple hyperparameters: $\lambda_{CDL}, \lambda_s,\lambda_d$, and temperature, $t$. The authors describe the tuning of these parameters individually with discussions on the ranges without elaborating on how to obtain these for other datasets. For instance, $\lambda_{CDL}$ is set to 0.001 for So2Sat and 0.1 for CHAMMI. How were these obtained? Fig. 4 (a,b) show the sensitivity of test accuracy wrt these parameters. And do these magnitudes mean anything? Why the log-scale? 

* **Channel anchors**: The discussion on using channel anchors is missing some key information. This could be because I was not aware of the ProxyNCA++ loss referenced in [52]. How are the channel anchors initialised and then selected for each channel? What would happen if instead of learning these vectors, one used a one-hot encoding to specify the channels? Perhaps some clarification here can be useful.  

* **Temperature, t**: In L. 148 authors say that results are not sensitive to the choice of $t$ in Eq. 1. However, they set it to a specific (and a peculiar) value of $t=0.07$. And, there is another temperature parameter (?) in Sec. 3.3 when describing DCS which is later reported in Table 5. They are different softmax temperatures, as far as I can see. This is very confusing. And which of the temperatures is more important?
questions: questions: See points under weaknesses.
limitations: limitations: The authors have addressed the main limitations in the paper.
paper_summary: paper_summary: In this paper, the authors present an improved methodology for modeling hyper-spectral multi channel images that can support a variety of channel configurations at test time. The authors reason that the current baselines treat all channels equally and do not consider the diverse qualities of each channel type. Hence they introduce 3 modifications to the training procedure that improves the performance of multi-channel image evaluations over state-of-the-art baseline methods.

The key contributions of the paper include:
1. Orthogonal initialization of Channel Embeddings and Channel Diversification Loss $L_{CDL}$
2. Token Diversification Loss $L_{TDL}$ with different weighting for tokens from the same channel ($\lambda_s$) and different channels ($\lambda_d$).
3. Diverse Channel Sampling strategy that samples diverse set of channels during training

Using the above modification to loss function and training procedure, the authors demonstrate an improvement of 1.5-5% in classification accuracy over the best performing baseline methodology (ChannelViT) on microscopy benchmarks CHAMMI and JUMP-CP and a satellite imaging benchmark So2Sat. The authors have performed further ablation studies to compare the usefulness and contribution of the loss terms.
strength_weakness: strength_weakness: While the overall methodology is well presented and has reasonable intuitive motivation, there are some weaknesses to the proposed methodology. 

1. One of the key problems addressed by the channel adaptive image models are generalization to settings where only a subset of channels available. This requires the models to learn both redundant and non-redundant information across channels for generalization to settings where some of the channels are missing or unavailable. While, the authors show channel diversification improving performance of the model, it is important to investigate and interpret where the new setup is improving the performance. Additional experiments on interpretations and comparisons of predictions between baseline methods and proposed methodology might help address this weakness.

2. In Fig 5, the authors compare the distributions of channels sampled during training between HCS (hierarchical channel sampling) and DCS (diverse channel sampling) methodologies. As that DCS samples some channels more compared to other during training, it is important to compare the performance between the baselines and proposed methodologies on those individual channel combination settings. Given the observation above and performance improvements in Table 2 being quite small with large standard deviation, the data is insufficient to the readers to demonstrate strict performance gains.

3. In Table 3, the authors have performed ablation studies by leaving out each of the losses or sampling methods. However, it is also important to show performance improvements with inclusion of each of the losses independently to understand the independent effects and contributions of each of the modifications
questions: questions: * Can the authors provide a more intuitive explanation for why patch token orthogonality (especially within channel patch token orthogonality) is necessary or might be useful? As the patch tokens are generated using the same set of weights for all patches of images, an identical patch of pixels would produce an identical set of tokens. In such a case, patch token diversification would be unreasonable. The results from Fig 4c further indicates that imposing weaker constraints on tokens from same channels compared with tokens from different channels obtains best performance. So, how is the patch token diversification loss on same channels helpful?
limitations: limitations: * The authors have shown results on limited datasets with smaller dataset sizes. The general applicability of this methodology to large data settings is unknown and would require further experimentation.
paper_summary: paper_summary: The authors provided the plug-and-play framework for MCI analysis, considering the classification as a downstream task. To do so, they introduced two-channel and token diversification strategies, and in addition, they proposed a new channel sampling strategy for converging the training faster.
strength_weakness: strength_weakness: - The authors repeatedly stress that each modification in DCS and TDL helps ensure the model's robustness. What ablation study did you apply to endorses it? Did you do any statistical testing on your experiments?

- The method, as stated, is plug-and-play. Did the authors apply their strategy to other efficient transformer structures with token sampling strategies [1], window-based [2], or linear attentions [3,4]? Because such methods don't have such redundancy exist in the standard Transformer.

The paper needs proofreading. As instances: 
- In line 37, the authors used “to” two times repeatedly. Omit one.
- In line 120, it is better to say “as noted in Section 1” rather than “as noted in the Introduction”.
- When citing a work, cite it after the author's name. For example, in line 98, reformat it as “ Recently, Bao et al. [18] …”.

[1] Fayyaz, Mohsen, et al. "Adaptive token sampling for efficient vision transformers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[2] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." Proceedings of the IEEE/CVF international conference on computer vision. 2021.

[3] Ali, Alaaeldin, et al. "Xcit: Cross-covariance image transformers." Advances in neural information processing systems 34 (2021): 20014-20027.

[4] Shen, Zhuoran, et al. "Efficient attention: Attention with linear complexities." Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.
questions: questions: Please see the Weaknesses.
limitations: limitations: Please see the Weaknesses.
paper_summary: paper_summary: This manuscript addresses the challenge of learning robust feature representations from datasets with heterogeneous channels, where the number and type of channels vary during training and testing. The authors propose a new DiChaViT, a model for Multi-Channel Imaging (MCI) based on the Vision Transformer (ViT) backbone. 

DiChaViT is proposed to improve existing MCI-ViT models like ChannelViT (*Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words, Bao et al., 2023*). Indeed, it re-elaborates ideas such as channel-wise patches and channel sampling, which contribute to the model's robustness in handling partial channel configurations at test time. Specifically, DiChaViT addresses previous models' tendency to learn redundant features across channels and patches, limiting their performance and generalizability.

DiChaViT introduces three novelties to promote feature diversity and improve generalization across different channel configurations:

1. **Channel Diversification Loss (CDL):** An additional loss term that encourages learning of distinct channel tokens by maximizing their Euclidean distance.
2. **Token Diversification Loss (TDL):** An additional loss term that aims to diversify the features extracted from each patch token by promoting orthogonality between them (especially between patches belonging to different channels).
3. **Diverse Channel Sampling (DCS):** A novel channel sampling strategy that selects channels based on their dissimilarity.

Overall, they enhance the diversity of learned features, both between different channels and within individual tokens, situating the work in the broader context of learning disentangled representations.

These novel methods, combined with an MCI-ViT model, improve the classification accuracy on multi-channel fluorescence microscopy and satellite image data by 2-5 percentage points, making DiChaViT SOTA on this specific task.
strength_weakness: strength_weakness: - Despite improving classification accuracy over previous SOTA methods, the paper lacks consistent experiments (or even hypotheses) demonstrating how and why the introduced novelties lead to such performance (e.g., in-depth analysis on the effect of losses on the distribution and content of channel and patch tokens).  In this context, the analyses reported in Appendix B are a good starting point, but the current analyses are too superficial and do not provide enough insights. 

- Comparing the accuracy of ChannelViT and DiChaViT for different subsets of channels (as shown in Table 2) is an interesting way to assess the robustness of the model to different channel configurations. However, the conclusion that “DiChaViT consistently demonstrates improved robustness when some input channels are missing” is rather weak and probably not statistically significant due to the large standard deviations. Similarly, the ablation study is extensive and clear, but the assertions about the positive impact of CDL and TDL (see Fig. 4) are not backed up by sufficient statistical evidence. Indeed, 
    1. the number of runs is not sufficient to produce reliable uncertainty estimates; 
    2. the differences in performance for varying hyperparameters are not remarkable. 
Finally, it could be useful to introduce more metrics than simple accuracy.
For these reasons, I would suggest readjusting the emphasis of the claims to what can comortably be claimed.

- There are some points in which the notation and naming is quite misleading:
    1. f() is used both at line 141 to identify a map from a channel to its anchor, and at line 166 to identify a map from a patch to its corresponding channel. 
    2. In the CDL paragraph, the authors used c_i to identify the channel token, while in the DCS paragraph c_i is an alias for the channel, and f_i is instead the channel token. 
    3. In Fig.2 patch embeddings are referred to as p_i, which is the same notation used for the input patches in the paragraph about TDL. 
    4. In algorithm 1 mirror, I would not call the first sampled channel as c_n, since it seems the last channel sampled over a total of n. Something like c_k or $\bar c_i$ would be more coherent.  
    5. The name “channel tokens” for the vectors c_i that are concatenated to the patch tokens is a bit unfortunate. Indeed, as far as I am concerned, their role is closer to the one of “patch embeddings” p_i. As a result, I believe that referring to them as “channel embedding” would be a better choice.
questions: questions: 1. Regarding CDL:
    - The orthogonal initialization of channel tokens introduces a strong prior/bias, especially considering that, in MCI, neighboring channels may contain very similar information. Have you tried to see what is the impact of imposing this initialization on tokens belonging to very similar channels? How do you expect the resulting tokens to be once the model is trained?
    - Why did you use the Euclidean distance in the loss? Have you also tried other distances that are more appropriate for multi-dimensional objects?
    - How are the anchors initialized? I believe it would be nice to briefly mention it since it is a key aspect of understanding CDL’s behavior.

2. Why is cosine similarity between channel tokens used in DCS, whereas in CDL euclidean distance is employed instead? I could see potential reasons for that, but I would appreciate it if the authors included a brief note about that in the manuscript.

3. In the first paragraph of section 4.4. the role of channel tokens is analyzed. In this context, why is the evaluation performed on ChannelViT instead of DiChaViT? Indeed, although CDL and the DCS are not applicable without channel tokens, TDL could still be used.

4. The methods seem very flexible to different kinds of data and channel configurations. Have you tried to measure their effectiveness on hyperspectral images with several tens of channels?
limitations: limitations: - I expect the proposed losses CDL and TDL to struggle in the specific case of MCI data where groups of channels share similar semantic information. An example is spectral imaging, where neighboring (wavelength) bands often contain similar information. 
In this case, I believe the following:
    1. Regarding the Token Diversity Loss, the choice of introducing a larger penalty for the similarity between patches of different channels is counterintuitive. Indeed, this choice implicitly assumes that the channel information is more important than the spatial one. However, in the context of such images, I would argue that the opposite is instead true in most situations. Likely, two patches referring to the same spatial location but belonging to different channels usually exhibit more similar semantic features than two patches in different locations for the same channel.
    2. Moreover, as already mentioned above, the approach proposed with CDL channel tokens introduces quite a strong prior on the channel token structure, which can be limiting for the datasets mentioned here.
Overall, I see these methods as effective only on MC datasets in which channels contain sufficiently diverse information (but I’m happy to read more results and be proven wrong).

- It would be interesting and meaningful to compare the learned features at different levels of the model (e.g., by visualizing the attention maps) to explicitly assess the disentangling effect brought by these methods.

Comments: title: Thank you for your response!
Thank you for your response!
Official Comment by Reviewer 4kDP
Thank you for your response!
Response to author rebuttal
comment: Thank you for your feedback and for considering the additional results and clarifications! We appreciate your acknowledgment of the technical soundness and potential impact of our work.

-Authors of paper 13979
The authors have provided sufficient clarification and additional results requested in their rebuttal. Thank you for performing the additional experiments and providing the clarifications. The paper is technically good with no concerns and a moderate to high-impact paper. Hence I have updated my score to reflect that.
Thank you for your positive feedback and for acknowledging the improvements we've made to our work! We appreciate your support and will make sure these discussed experiments are included in the final version.

-Authors of paper 13979
Thank you for providing clarifications. The authors have addressed most of the requested clarifications and included relevant experiments to address other reviewers' comments. Given these improvements, I will maintain my good score. Please add the discussed experiments to the final version of the paper.
Thank you for your thorough review and positive feedback on our revisions! We will make sure to highlight the justification for dealing with unseen channels in the limitations section of the final version.
The authors have addressed most of the clarifications sought by me, and also added relevant experiments in addressing other reviewer comments. In particular, the justification for dealing with unseen channels is convincing. I would suggest the authors include this in limitations in the final version of the paper. 

I will raise my score to Accept.
Hello Reviewer K1xF,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979
Hello Reviewer 4kDP,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979
Hello Reviewer gjzo,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979
Hello Reviewer q1Xa,

As we are halfway through the discussion period, we are hoping you would read through our rebuttal and let us know if you have any additional questions. We would be happy to answer them, thank you!

-Authors of paper 13979