Channel Vision Transformers: An Image Is Worth 1 x 16 x 16 Words
Authors: Yujia Bao
Srinivasan Sivanandan
Theofanis Karaletsos
Venue: ICLR 2024 poster (2024)
Keywords: vision transformer
representation learning
hyper spectral imaging
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: ChannelViT facilitates robust representation learning across different input channels.

Abstract: Vision Transformer (ViT) has emerged as a powerful architecture in the realm of modern computer vision. However, its application in certain imaging fields, such as microscopy and satellite imaging, presents unique challenges. In these domains, images often contain multiple channels, each carrying semantically distinct and independent information. Furthermore, the model must demonstrate robustness to sparsity in input channels, as they may not be densely available during training or testing. In this paper, we propose a modification to the ViT architecture that enhances reasoning across the input channels and introduce Hierarchical Channel Sampling (HCS) as an additional regularization technique to ensure robustness when only partial channels are presented during test time. Our proposed model, ChannelViT, constructs patch tokens independently from each input channel and utilizes a learnable channel embedding that is added to the patch tokens, similar to positional embeddings. We evaluate the performance of ChannelViT on ImageNet, JUMP-CP (microscopy cell imaging), and So2Sat (satellite imaging). Our results show that ChannelViT outperforms ViT on classification tasks and generalizes well, even when a subset of input channels is used during testing. Across our experiments, HCS proves to be a powerful regularizer, independent of the architecture employed, suggesting itself as a straightforward technique for robust ViT training. Lastly, we find that ChannelViT generalizes effectively even when there is limited access to all channels during training, highlighting its potential for multi-channel imaging under real-world conditions with sparse sensors. Our code is available at https://github.com/insitro/ChannelViT.

Published as a conference paper at ICLR 2024
CHANNEL VISION TRANSFORMERS: AN IMAGE IS WORTH 1 X 16 x 16 WORDS
Yujia Bao*i
Srinivasan Sivanandan"
Accenture yujie.bao@@ccennuuee.co
Insitro
s wrwpeaannninnittt.oooomoom insitro.com
Theofanis Karaletsos
Chan Zuckerberg Initiative
cleofanis@karrletssos. com
ABSTRACT
Vision Transformer (ViT) has emerged as powerful architecture in the realm of modern computer vision. However, its application in certain imaging fields, such as microscopy and satellite imaging, presents unique challenges. these domains, images often contain multiple channels, each carrying semantically dis- tinct and independent information. Furthermore, the model must demonstrate ro- bustness to sparsity in input channels, they may not be densely available during training or testing. In this paper, we propose a modification to the ViT architec- ture that enhances reasoning across the input channels and introduce Hierarchical Channel Sampling (HCS) as an additional regularization technique to ensure ro- bustness when only partial channels are presented during test time. Our proposed model, Vhan constructs patch tokens independently from each input chan- nel and utilizes learnable channel embedding that is added the patch tokens. similar to positional embeddings. We evaluate the performance of Channel ViT on ImageNet, JUMP-CP (microscopy cell imaging), and So2Sat (satellite imag- ing). Our results show that Channel outperforms VIT on classification tasks and generalizes well, even when a subset of input channels used during testing. Across our experiments, HCS proves to be powerful regularizer, independent the architecture employed, suggesting itself as a straightforward technique for robust ViT training. Lastly, we find that ChannelViT generalizes effectively even when there is limited access to all channels during training. highlighting its po- tential for multi-channel imaging under worldworld conditions with sparse sensors. Our code is available at https: httpp::///itthbb com/ insitiiiinn com//nssttoo//hhhneeee
INTRODUCTION
Vision Transformers (ViT) have emerged as a crucial architecture in contemporary computer vision, significantly enhancing image analysis. However, application to specific imaging domains, such as microscopy and satellite imaging, poses unique challenges. Images in these fields often comprise multiple channels, each carrying semantically distinct and independent information. The complexity is further compounded by the fact that these input channels may not always be densely available during training or testing, necessitating model capable of handling such sparsity.
In response to these challenges, we propose modification to the ViT architecture that bolsters reasoning across the input channels. Our proposed model, ChannelViT, constructs patch tokens in- dependently from each input channel and incorporates a learnable channel embedding that is added to the patch tokens in addition to the location-specifie positional embedding. This simple modifica- tion enables the model to reason across both locations and channels. Furthermore, by treating the
"Fqual contribution.
"Research supporting this publication conducted while authors were employed at Insitro.

---

Published as a conference paper at ICLR 2024
Classier
Transformer Enooder
polle
pollo
pilts
poin
poos
pods
pDN
CLS
cfting
ciity
RIEAT
FigA
Input mage (B-channsl)
Channef/T I preaos pason takens
Figure 1: Illustration of Channel Vision Transformer (ChannelViT). The input for ChannelViT is a cell image from JUMP-CP, which comprises five fluorescence channels (colored differently) and three brightfield channels (colored in B). Channel generates patch tokens for each individual channel, utilizing a learnable channel embedding chn to preserve -ssecfi--ppecffi information. The positional embeddings pos and the linear projection W are shared across all channels.
channel dimension as the patch sequence dimension, ChannelViT can seamlessly handle inputs with varying sets of channels.
Despite these advancements, two main challenges persist. While Channel can leverage existing efficient implementations VIT with minimal modifications, the increase in sequence length intro- duces additional computational requirements. Moreover, if ViT is consistently trained on the same set of channels, its ability to generalize to unseen channel combinations at test time may be compromised. To address these challenges, we introduce Hierarchical Channel Sampling (HCS), new regularization technique designed to improve robustness. Unlike channel dropout, which drops out each input channel independently, HCS uses a two-step sampling procedure. It first samples the number of channels and then, based on this, it samples the specific channel configurations. While channel dropout tends to allocate more distribution to combinations with specific number of chan- nels, HCS assigns uniform weight to the selection of any number of channels. HCS consistently improves robustness when different channels utilized during testing in both ViT and Channel Notably, our evaluation on ImageNet shows that using only the red channel, HCS can increase the validation accuracy from 29.39 68.86.
We further evaluate ChannelViT on two real world multi colti-channel imaging applications: microscopy cell imaging (JUMP-CP) and satellite imaging (So2Sat). these applications, different channels often correspond to independent information sources. ViT significantly outperforms its ViT counterpart in these datasets, underscoring the importance of reasoning across different channels. Moreover, by treating different channels as distinct input tokens, demonstrate that Channel ViT can effectively generalize even when there is limited access all channels in the dataset during train- ing. Lastly, show that ChannelViT enables additional insights. The learned channel embeddings correspond to meaningful interpretations, and the attention visualization highlights relevant features across spatial and spectral resolution, enhancing interpretability. This highlights the potential of ChannelViT for wide-ranging applications in the field of multi-channel imaging.
RELATED WORK
Vision transformer and its applications multi-channel imaging Vision Transformer (ViT) has demonstrated state-of-the-an performance in various computer vision tasks Dosovitskiy et al. Touvron al.) (202T); Carion et al. (0000): Zhu et al. (202bb). Recently, researchers have started adopting VIT for multi-spectral imaging. For example, in satellite imaging. Kaselimi et al. (2022) showed that a ViT-based classifier outperforms CNN models, especially on imbalanced classes. Ad- temporal positional encodings to model satellite images over time, while Cong et al. (2022) demonstrated the benefits of using dis- tinct spectral positional encodings with VIT. Recently, Nguyen et al. (2023) proposed modification to the ViT architecture, introducing variable tokenization and variable token aggregation methods to handle heterogeneous input data sources in climate and weather modeling. Moreover, [Scheibenreif
ditionally, Tarasiou et al. (2023) proposed ioquisitionntime-spetee
2

---

Published as a conference paper at ICLR 2024
al. (2022) found that ViT, when combined with self-supervised pre-training, performs on-par with
state-of of-the-art benchmarks.
In the field of cell biology, Sivanandan al. (2023) utilized ViT with self-supervised pre-training
to learn representations of cells across multiple fluorescence channels. Furthermore, Hatamizadeh al. (2022a b) leveraged VIT for segmenting 3D MRI images. (Hussein al. (0022) proposed to
train multiple VITs, one for each input channel, for epileptic seizure predictions.
In contrast to previous work, address practical challenge in multi-channel imaging, where different datasets often have different available channels[] To tackle this challenge, we propose ChannelViT, which creates image patches from each individual input channel. This simple modifi- cation unifies the modeling across data with different input channels and offers robust performance test time, even when only a subset of the channels available.
Robustness for Vision Transformer Robustness can be defined in different ways. One aspect is the vulnerability to adversarial attacks. MMahmood et al. (202T) found that ViTs are as susceptible to white-box adversarial attacks CNNs. To improve robustness, Robust ViT incorporates more robust components like global pooling (Mao et al. 2022). Additionally, Chefer et al. (2022) propose regularization of the relevancy map of VIT to enhance robustness, Zhou et al. (022 Zhang et al. (202T): Song et al. 2022) augments transformers with feature-wise attention to improve robustness and performance. Another approach focuses on generalization over distribution shifts Sagawa al. (2019; Liu et al. (2021). [aao & Karaletsos (2023) introduces a context token inferred from ViT's
hidden layers encode group-specific information.
In our work, we specifically focus on improving the generalization performance across different channel combinations, which is common scenario in multi-channel imaging. We argue that the original ViT is sensitive to changes in input channels, as it computes single patch token across all channels. In contrast, Cha creates separate patch tokens for each channel, making it inher- ently more robust to variations in channel availabilities. To further enhance channel robustness, we introduce hierarchical channel sampling (HCS) during training. This methodology draws inspira- tion from prior studies on channel dropout (Srivastava et al. (2014): TTomsso et al. (2015); Hou & Wang (2019). However, instead dropping out intermediate channels, our approach introduces two-stage sampling algorithm designed to selectively mask out the input channels.
3 METHOD
ChannelViT is modification of the original Vision Transformer (ViT) architecture proposed by Dosovitskiy et al.] Unlike the original architecture, which condenses each multi-channel image patch into a single 'word" token, ChannelViT segregates channel-specine information into multiple tokens. This simple yet effective modification yields three key advantages:
1. ChannelViT facilitates reasoning across both positions and channels with Transformer;
2. By transforming the channel dimension into the sequence length dimension, ChannelViT can seamlessly manage inputs with varying sets of channels;
3. Cha can utilize existing efficient implementations of VIT.
In the following paragraphs, we explore the architecture and implementation of ChannelViT in de- tail. Figure providees visual overview of the model.
3.1 CHANNEL VISION TRANSFORMER (CHANNELVIT)
Patch embeddings Consider an input image x with dimensions x W x C. Given a patch size of P P. this image can be reshaped into sequence of non-overlapping patches
[æ|c]. pı], .,, x(c1, PN], x z[c2,
x ([c2,pPN] pN].
.x(cc, x[co, a[cc.py]],
where z/c;, P.]] corresponds to the n-th P x P' image patch at channel c; and N = HW/P2. As the Transformer encoder requires sequence of dieddmmnsional vectors, each patch is flattened into
evample pnttps:/ /ttp///tttttott con/ wwwp//hciieeeeaaeeoeeeeeeeeeeeee Ltt.----eggry--etaerree imaging often imolves multiple signals
Sontinel-1 (SAR), Senting-2, CAY

---

Published as a conference paper at ICLR 2024
1D vector. Unlike ViT, which generates single token for multi-channel image patch, ChannelViT produces one token from every single-channel image patch.
Tied image filters We apply learnable linear projection W PR the flattened patches. It important to note that in a regular ViT, each channel has its own weights in the linear projection layer. In ChannelViT, our preliminary experiments suggest that tying the image filters across chan- nels offer superior performance compared to untied image filters (Appendix C.3) Therefore, we tic the learnable projection W across channels. The intuition behind this that the low-level image filters can be shared across channels (Ghiasi et al.] (022), and tying the parameters can improve the model's s robustness across channels.
Channel-aware and position-aware patch embeddings Despite tying the linear filter across channels, it remains essential to preserve channel-specific information, given the distinct char- acteristics of different channels (Appendix C.4). We introduce learnable channel embeddings [chn1,.... chng]. where chn. E IgP In line with the original ViT, we also incorporate learnable positional embeddings to maintain positional information of each patch. We denote the positional embeddings [poss..... pos. vl. where pos, E R.. It's worth noting that these position em- beddings are also shared across channels, enabling ChannelViT to recognize the same image patch across different channels. Finally, we prepend learnable classifier token CLS R to the sequence to encode global image features. The resulting input sequence can be written as
[CLS, posı + chni + Wo[ei.p),
POSN + chhn + Wx(ei, POaN + chnc + Wxleo pN]]
pos + chnc + Wxlee, Pil,
Transformer encoder Following the original VIT, we feed the above input sequence into a Trans- former encoder, which captures dependencies between image patches by embedding each patch based on its similarity to others Vaswani et al. (2017). Specifically, the Transformer encoder com- prises alternating layers of multiheaded self-attention blocks and MLP blocks. Layer normalization, proposed by Ba et al. (2016), is performed before each block, and residual connections He et al. (206)) are established after each block. We use the final layer representation of the CLS token to represent the input image. For classification tasks, linear classifier is employed, followed by Softmax function, to predict the corresponding label. We utilize the standard cross entropy loss as our training objective.
3.2 HIERARCHICAL CHANNEL SAMPLING (HCS)
Training ChannelViT directly presents two challenges: 1) The sequence length becomes propor- tional to the number of channels, leading to a quadratic surge in the number of attentions required for computation; 2) Training exclusively on all channels may result in the model not being pre- pared for partial channels at test time, thereby affecting its generalization capability. To mitigate these issues, we propose applying hicrarchical channel sampling (HCS) during the training process. Specifically, for an image z with C' channels, proceed as follows:
1. First, we sample random variable m uniformly from the set f1.2....,C) C). This repre- sents the number of channels that we will utilize during this training step;
2. Next, we sample channel combination C. uniformly from all channel combinations that consist of m channels;
3. Finally, we return the image with only the sampled channels ar(Cm].
HCS shares similarity to channel dropout Tompson et al. (2015), but it differs in terms of the prior distribution imposed on the sampled channels. channel dropout, each channel dropped based on given probability independently. The probability of having channels varies drastically for different ms, which can negatively impact the final performance (Figure 3}. In contrast, since m is sampled uniformly over the total number of channels, HCS ensures that the sampling procedure equally covers each m. Finally, we note that HCS is only employed during training. At test time, ChannelViT has access all input channels.
4

---

Published as a conference paper at ICLR 2024
ImageNec
JUMP-CP
So25al
ImsggeNet
JUMPSCP
Se25al
differani
Comclatioss
charmel unboldinge
Correlation patterns among image channels (left) and the learned channel embeddings (right) for ImageNet, JUMPCP, and So2Sat. ImageNet displays a strong correlation among the three RGB input channels while JUMPCP and So2Sat show minimal correlation between different signal sources (Fluorescence vs. Brightfield, Sentinel I vs Sentinel 2).
Figure 2:
HCS can also be interpreted as simulating test-time distributions during training. Compared to group distributionally robust optimization (Sagawa al. 2019), HCS minimizes the loss rather than the worst-case loss. This approach is logical when considering channel robustness, as having more channels will naturally enhance performance. We don't want the model to over-focus on the worst- case loss, which typically corresponds to situations when we sample very few channels.
4 EXPERIMENTS
We evaluate Cha across three image classification benchmarks: ImageNet Deng et al. (2009), JUMP-CP Chandrasekaran al. (2022), and So2Sat Zhu al. (2019). In Figure (top), we illustrate the correlation among different input channels for each dataset. As observed, ImageNet exhibits a strong correlation among the three RGB channels. For JUMP-CP, while there is a strong correlation within the fluorescence channels and within the brightfield channels, there is minimal to no correlation between the brightfield and the fluorescence channels. similar group structure among the channels is observed for So2Sat. Due to space constraints, primary focus in the main paper is on the comparison between ViT and ChannelViT. For additional comparisons with Mul- tiVIT (Hussein et al. 2022), please refer to Appendix [B.]] Comparisons with FANs (Zhou al. 2022 can be found in Appendix B.2
JUMP-CP
The JUMP-CP benchmark, established by the JUMP-Cell Painting Consortium, serves microscopy imaging standard. In alignment with the work of Chandrasekaaaa al. (022),
utilize the task of perturbation detection from cell images as a means to evaluate and compare the efficacy of various representation models. It is important to recognize that while perturbation detec- tion a valuable task, not the ultimate objective of cell imaging modeling; rather, provides an interpretable metric for model assessment. The dataset includes a total of 160 perturbations. We focused on a compound perturbation plate 'BR001 16991', which contains 127k training images, 45k validation images, and 45k testing images. Each cell image contains 8 channels, comprising both fluorescence information (first five channels) and brightfield information (last three channels).
So2Sat This satellite imaging benchmark encompasses half a million image patches from Sentinel-1 and Sentinel-2 satellites, distributed across 42 global urban agglomerations. Each im- age patch incorporates 18 channels, with originating from Sentinel-1 and the remaining 10 from Sentinel-2. The primary objective of this dataset is to facilitate the prediction of the climate zone for each respective image patch, with total of 17 distinct climate zones being represented.
Implementation details We utilize the Vision Transformer (ViT) implementation provided by Facebook Researct During training, we minimize the cross entropy loss. To ensure fair com- parison, both VIT and ChannelViT are subjected to identical optimization settings. These settings encompass the use of the Adam optimizer, a learning rate scheduler featuring linear warmup and co- sine decay, and a cosine scheduler for the weight decay parameter. For a more detailed description settings, direct readers to the Appendix.
of the
5

---

Published as a conference paper at ICLR 2024
Table 1: Validation accuracy on ImageNet under different testing conditions (using all three channels or only one channel). We observe that 1) hierarchical channel sampling significantly boosts single- channel performance at test time; 2) ChannelViT consistently outperforms the ViT baseline. The expert models, trained using only one channel, represent the upper bound of potential performance.
Use hicrarchical channel sampling?
Val Acc. on RGB
Val Acc. on R-only
Val Acc.
Val Acc.
Backbone
on G-only
on B-only
Models trained on three channels (RGB)
ViT-S/16
71.49
29.39
33.79
21.18
VIT-S/16
73.01
68.86
69.78
67.59
ChannelViT-S/16
74.64
69.90
70.30
68.48
Expert models trained on only one channel
ViT-S/16 (R-only)
N/A
70.04
VIT-S/16 (G-only)
N/A
70.61
ViT-S/16 (B-only)
N/A
69.47
VIT-S/16
VIT-S/6
Distributions of the imput channels
8600
0544
40005
A-M0L
Accunzy
45.0%
Prbaublley
234'
Not
Jaans
83
600
fccaanels sampled for evaluation
#chamnels sampled for evalustion
*channels sampled for eaining
Taput channel bqpurcharaill.eppeerr..
Illeorhiical Chornel Sampling
Figure 3: HCS input channel dropout JUMP CP (trained all 8 channels). On the left, we present the accuracy of ViT-S/16 and ViTnne/ViT--S11 under varying input channel dropout rates and HCS. The accuracy is evaluated across all channel combinations, with the mean accuracy reported for combinations with an equal number of channels (represented on the horizontal axis). On the right, we illustrate the probability distribution of the sampled channel combinations during the training process. We observe I) ViTs trained with input channel dropout tend to favor channel combinations that are sampled the most; 2) ChannelViT with input channel dropout outperforms ViT with input channel dropout; 3) HCS surpasses input channel dropout in terms of channel robustness.
4.1 IMAGENET
Table showcases our results on ImageNet, using ViT small as the representation backbone and patch size of 16 by 16. We observe that without applying hierarchical channel sampling, ViT- S/16 achieves a validation accuracy of 71.49 using all three channels but fails to generalize when only channel is provided at test time. Simulating this test-time channel drop during training via hierarchical channel sampling (HCS) significantly improves performance. For instance, the validation accuracy for using only the red channel improves from 29.39 68.86, demonstrating the effectiveness of HCS as regularizer for enforcing channel robustness. Lastly, while there is limited room for improvement due the strong correlations among the input RGB channels, Channel still consistently outperforms the corresponding ViT baseline (by 1.2 on average), narrowing the gap (1.30 + 0.48) to the expert models that are trained using only one channel.
4.2 JUMP-CP: MICROSCOPY CELL IMAGING
We present results on the microscopy cell imaging benchmark, JUMP-CP, in Table 2 This benchmark involves a I 160-way classification task. Due to computational constraints, we utilize ViT- S as our representation backbone. We consider both the standard resolution with a patch size of 16x16 and high-resolution model with patch size of 8x8.
https://git tttp://ggtthbooooooo /tm/ Faceborkk resee reweerch//eee /ol no/ /ll obs/mei ifvisian www.aattttarrem traneformer-po

---

Published as a conference paper at ICLR 2024
Table 2: Test accuracy of 160-way perturbed gene prediction JUMP-CP. Two training settings are considered: one using only 5 fluorescence channels and the other incorporating all 8 channels, which includes additional brightfield channels. During testing, all possible channel combinations are evaluated and we report the mean accuracies for combinations with the same number of channels (See Appendix [ for detailed error analyses). We observe that cross channel reasoning crucial when the inputs have independent information (fluorescence vs, brightfield).
ViT-S/16 Channel VIT-S/16 ChannelViT-S/16 ViT-S/8 Channel VIT-S/8
Use hierarchical
channel sampling?
x
/
Training on 5 fluorescence channels
5 channels
48.41
53.41
55.51
56.78
60.29
60.03
Mchannels For testing
4 channels
0.85
15.13
43.59
45.94
48.80
49.34
channels
1.89
5.12
33.14
35.45
37.13
38.15
channels
1.46
1.22
25.24
26.57
27.40
27.99
channel
0.54
1.25
20.49
21.43
21.30
21.58
Training on channels (5 fluorescence channels & brightfield channels)
channels
52.06
66.22
56.87
68.09
66.44
74.77
7 channels
5.91
41.03
49.35
61.02
59.01
68.42
#channels for testing
6 channels
1.81
24.57
42.38
53.45
51.29
61.26
channels
2.46
14.20
35.78
45.50
43.39
53.05
channels
2.38
8.56
29.84
37.37
35.60
43.87
channels
2.70
5.65
24.94
29.68
28.59
34.19
2 channels
2.63
3.24
21.54
23.77
23.32
25.73
channel
3.00
2.08
19.92
20.84
20.41
21.20
In the first part of our analysis, train all models using only the five fluorescence channels and evaluate their performance on the test set under various input channel combinations. Our obser- vations are follows: 1) HCS significantly enhances the channel robustness for both VIT and ChannelViT; 2) High-resolution models consistently outperform their low-resolution counterparts; 3) With the exception of the 5-channel evaluation with patch size of 8x8, ChannelViT consistently outperforms VIT.
In the latter part of our analysis, we utilize all available channels for training, which includes three additional brightfield channels for each image. For VIT, the high-resolution ViT-S/8 model improves from 60.29 to 66.44 44, demonstrating the importance of the additional brightfield information, while the improvement for ViT-S/16 is marginal (from 55.51 to 56.87). When focusing ChannelViT, we observe significant performance boost over its ViT counterpart. Channel outperforms VIT-S/16 by 11.22 (68.09 56.87) and ChannelViT--S 66.44). These improvements are consistent across different channel combinations. As we have in Figure [ fluorescence and brightfield channels provide distinet information. ChannelViT effectively reasons across channels, avoiding the need to collapse all information into a single token the first layer, thereby enhancing performance.
outperforms ViT-S/8 by 8.33 (74.77
Lastly, we delve into a comparative analysis between input channel dropout and hierarchical chan- nel sampling, as depicted in Figure 3 It is evident from our observations that the VIT model, when trained with HCS, consistently surpasses the performance of those trained with input channel dropout across all channel combinations. Furthermore, we discern a pronounced correlation be- tween the performance of models trained with input channel dropout and the probability distribution of the number of channels sampled during training.
Data Efficiency In the realm of microscopy imaging, we often encounter situations where not all channels are available for every cell due to varying experiment guidelines and procedures. Despite this, the goal remains to develop universal model capable of operating inputs with differing channels. ChannelViT addresses this issue by treating different channels as distinct input tokens, making it particularly useful in scenarios where not all channels presents a scenario where varying proportions (0%, 25%, 50%, 75%, 100%) the training data have access all eight channels, with the remaining data only having access to the five fluorescence
available for all data. Table

---

Published as a conference paper at ICLR 2024
Table VIT vs. ViT when we have varying channel availability during training. Both mod- els are trained using HCS. The accuracy is evaluated using five fluorescence channels (top) and all eight channels (bottom). For the middle three columns where we have mixed training data, we re- port the mean (and std) accuracy over three randomly generated partitions. ChannelViT consistently outperforms ViT across all settings, and performance gap notably widens as access to more 8- channel data is provided.
Combine fluorescence-only data and 8-channel data for training
% fluorescence-only data
100%
75%
50%
25%
0%
% 8-channel data
0%
25%
50%
75%
100%
Evaluating 5 fluorescence channels
VIT-S/16
55.51
52.55
51.65:2214
49.53
45.75
ChannelViT-S/16
56.78
58.01
58.19:1.49
58.42:1.37
57.60
Evaluating all 8 channels
ViT-S/16
50.29+1.83
52.47:1.82
54.64:1.01
56.87 68.09
ChannelViT-S/16
57.97
61.88
64.80+t... +0.39
channels. The performance of VIT and Vha is evaluated at test time using both the five fluorescence channels (top section) and eight channels (bottom section).
Our observations are as follows: 1) When only a limited amount of 8-channel data (25%) is available, both ViT and VIT show decrease in performance when utilizing eight channels test time compared five channels; 2) As the availability of 8-channel data increases, the performance of the ViT baseline on the fluorescence evaluation steadily declines (from 55.51 to 45.75), while the performance of ChannelViT sees slight improvement (from 56.73 57. 57.60); 3) When evaluated on all eight channels, ChannelViT significantly outperforms ViT, with an average gap of 9.62.
Channcl-specific attention visualization Attention heatmaps, generated by Vision Transformers (ViTs), have emerged valuable tool for interpreting model decisions. For instance, Chefer et al. 22021 introduced a relevance computation method, which assigns local relevance based on the Deep Taylor Decomposition principle and subsequently propagates these relevance scores through the layers. However, a limitation of ViT' is their tendency to amalgamate information across different channels. In the realm of microscopy imaging, discerning the contribution of each fluorescence channel to the predictions is vital due to their distinct biological implications.
Figure 4 (right) presents the class-specific relevance visualizations for ViT-S/8 and ChannelViT-S/3 For the top cell labeled KRAS, Cha appears to utilize information from the Mito channel. For the bottom cell labeled KCNH76, ChannelViT seems utilize information from the ER and RNA channels for its prediction. Compared to ViT, ChannelViT facilitates the examination of con- tributions made by individual channels.
In Figure (left), we further compute the maximum attention score (averaged over 100 cells) for each cell label (perturbed gene) and each input channel. Our observations indicate that ChannelViT focuses different channels for different labels (corresponding to perturbed genes), with the Mito channel emerging as the most significant information source. This heatmap, which describes the discriminability of different labels over different channels, can also aid in better understanding the relationships between different gene perturbations.
Time efficiency One limitation of Channel is the additional computational cost incurred when expanding the channel dimension into the sequence length dimension. Implementing Channel without HCS increases the training time from approximately hours to 12 hours. With HCS, the training duration for ChannelViT is reduced to about 10 hours. During inference, Cha requires approximately 1.6 times more time than its ViT counterpart. An interesting future di- rection would be to combine ChannelViT with more efficient attention mechanisms, such Lin- former Wang et al. (2020) and LongNet Ding et al.) (2023), which scale linearly with sequence length. We direct the reader Appendix C.1 for comprehensive analysis of the running times.
8

---

Published as a conference paper at ICLR 2024
May ENA
B-laandl view
Sugledhamt view
DSA
RNA
BF-1
kiei KRAS
Auretiio
Perting
Ringhefll
Figure 4: Left: Class-specific relevance attribution of Channel for each cell label (perturbed gene) JUMP-CP. For each perturbed gene (y-axis) and each channel (x-axis), we calculate the maximum attention score, averaged over 100 cells from that specific cell label. This reveals that ChannelViT focuses on different input channels depending on the perturbed gene. Right: A visu- alization of the relevance heatmaps for both ViT-S/8 (8-channel view) and Channel (single- channel view). Both models are trained on JUMP-CP using HCS across all 8 channels. Channel ViT offers interpretability by highlighting the contributions made by each individual channel.
Table 4: Test accuracy of 17-way local climate zone classification on So2Sat. We consider two official splits: random split and city split. Both
(Mein and Sidd over there nan))
63.01
ViT and ChannelViT are trained on all channels with hicrarchical channel sampling. We evaluate their performance on 18 channels (Sentinel 1 2) as well as partial channels (Sentinel 1).
'Ieat Accuracy
51.52
6076
61:18
$1.27
VIT-S with HCS
Sentinel (Channel 0-7)
Sentinel & 2 (Channel 0-17)
Viannd with HCS
10%
50%
100%
Random split ZZuu
2021
of data that have Sentinel Sentinel 2 signals (The remaining data only has Sentinel signals.)
ViT-S/8
50.62
97.82
Channel VIT-S/8
59.75
99.10
Test accuracy So2Sat city split with varying channel availabilities during training. Both ViT and ChannelViT are trained with hi- erarchical channel sampling. Performances are evaluated using all channels (Sentinel & 2).
Figure 5:
City split Zhuu et al. 2019)
ViT-S/8
41.07
62.48
ThannelViT-SK
47.39
63.01
4.3
So2SAT: SATELLITE IMAGING
Our results on the So2Sat satellite imaging benchmark are presented in Table 4 We evaluate two official splits: random split and city split, training both ViT-S/8 and ChannelViT-SA models using hierarchical channel sampling across all channels (Sentinel 1 & 2).
Upon evaluation, Cha demonstrates superior performance over its VIT counterpart, with an improvement of 1.28 for the random split and 0.53 for the more challenging city split. In the realm of satellite imaging, Sentinel channels are derived from Synthetic Aperture Radar operating on the C-band, while Sentinel-2 is multispectral high-resolution imaging mission. It's worth noting that Sentinel-2 data can be -ffudtffeetedd underscoring the importance of models that can robustly operate under partial signals using only Sentinel 1. In both random and city splits, ChannelViT significantly outperforms VIT (59.75 vs. 50.62 random split and 47.39 vs. 41.07 in city split).
Lastly, we explore the efficiency of Channel in combining satellite training data with different signals. As depicted in F we consider varying proportions (10% 25% 50%, 75%, 100%) of the training data with access both Sentinel 1 & signals, while the remaining data only has access to Sentinel signals. The models are evaluated using all Sentinel & signals. Our observations consistently show ChannelViT outperforming ViT.
Interpreting the channel embeddings learned by ChannelViT Figure 2 presents the correla- tions between the input channels. It's noteworthy that the first four channels of Sentinel-1 corre-

---

Published as a conference paper at ICLR 2024
spond to: 1) the real part of the VH channel; 2) the imaginary part of the VH channel; 3) the real part the VV channel; and 4) the imaginary part of the VV channel. These four input channels are uncorrelated, evidenced by the bottom left corner of the So2Sat visualization heatmap. However, upon examining the correlations between the learned channel embeddings, we observe a high cor- relation between the real and imaginary parts of both VV and VH channels. This intuitively aligns with the fact that the real and imaginary parts are equivalent in terms of the information they pro- vide. This demonstrates that ChannelViT learns meaningful channel embeddings, which can provide additional insights into the relationships between different input signals.
CONCLUSION
In conclusion, our proposed model, ChannelViT. effectively addresses the unique challenges of multi-channel imaging domains. By enhancing reasoning across input channels and seamlessly handling inputs with varying sets of channels, ChannelViT has consistently outperformed its ViT counterpart in our evaluations on ImageNet and diverse applications such as medical, microscopy cell, and satellite imaging. The introduction of Hierarchical Channel Sampling (HCS) further bol- sters the model's robustness when testing with different channel combinations. Moreover, Chan- VeT not only improves data efficiency but also provides additional interpretability, underscoring its potential for broad applications in the field of multi-channe' imaging.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 2016.
arXiv preprint
Yujia Bao and Theofanis Karaletsos. Contextual vision transformers for robust representation learn- ing. arXiv preprint arXiv:2305.19402. 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650-9660, 2021.
Srinivas Niranj Chandrasekaran, Beth Cimini, Amy Goodale, Lisa Miller, Maria Kost-Alimova Nasim Jamali, John Doench, Briana Fritchman, Adam Skepner, Michelle Melanson, al. Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations. bioRxiv, pp. 2022-01, 2022.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization, 2021.
Hila Chefer, Idan Schwartz, and Lior Wolf. Optimizing relevance maps of vision transformers improves robustness. Advances Neural Information Processing Systems, 35:33618-333 -33633,, 2022.
Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for temporal and multi- spectral satellite imagery. Advances in Neural Information Processing Systems, 35:197-211, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, and Furu Wei. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.0244 02486, 2023.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations.
Irena Gao, Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Out-of- distribution robustness via targeted augmentations. In NewrIPS 2022 Workshop on Distribution Shifis: Connecting Methods and Applications, 2022.
Amin Ghiasi, Hamid Kazemi, Eitan Borgnia, Steven Reich, Manli Shu, Micah Goldblum, An- drew Gordon Wilson, and Tom Goldstein. What do vision transformers learn? visual explo- ration. arXiv preprint arXiv:2212.06722 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 706.02677, 2017.
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Land- man, Holger Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 574- 584, 2022a.
Ali Hatamizadeh, Ziyue Xu, Dong Yang. Wenqi Li, Holger Roth, and Daguang Xu. Unetformer: A unified vision transformer model and pre-traininn pre-training framework for 3d medical image segmentation. arXiv preprint orXiv:2204.000 10,31, 2022b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Saihui Hou and Zilei Wang. Weighted channel dropout for regularization of deep convolutional neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp-
2019.
Ramy Hussein, Soojin Lee, and Rabab Ward. Multi-channel vision transformer for epileptic seizure prediction. Biomedicines, 10(7):1551, 2022.
Maria Kaselimi, Athanasios Voulodimos, Ioannis Daskalopoulos, Nikolaos Doulamis, and Anasta- sios Doulamis. A vision transformer model for convolution-fres multilabel classification of satel- lite imagery in deforestation monitoring. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning, pp. 6781-6792. PMLR, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL http::/:///.peeeeee.....tttttt wwwp:/rmm/oo/mmmmmmmmm
Bkg6Ri Cqy7
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7838-7847, 2021.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVF conference on Com- puler Vision and Patlern Recognition, pp. 12042-12051, 2022.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K. Gupta, and Aditya Grover. Climax: A foundation model for weather and climate, 2023.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference Learning Representations, 2019.
11

---

Published as a conference paper at ICLR 2024
Linus Scheibenreif, Joëlle Hanna, Michael Mommert, and Damian Borth. Self-supervised vision transformers for land-cover segmentation and classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1422-1431, 2022.
Srinivasan Sivanandan, Bobby Leitmann, Eric Lubeck, Mohammad Muneeb Sultan, Panagiotis
Stanitsas, Navpreet Ranu, Alexis Ewer, Jordan E Mancuso, Zachary F Phillips, Albert Kim, et al. A pooled cell painting crispr screening platform enables de novo inference of gene function by self-supervised deep learning. bioRxiv, pp. 2023-08, 2023.
Qi Song, Jie Li, Chenghong Li, Hao Guo, and Rui Huang. Fully attentional network for semantic segmentation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2280-2288. 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15():1192--1958 2014.
Michail Tarasiou, Erik Chavez, and Stefanos Zafeiriou. Vits for sits: Vision transformers for satellite image time series. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10418-10428, 2023.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp.
2015.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Sinong Wang, Belinda 7. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.0477 2020.
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing. pp. 1-10, 2018.
Xin Zhang, Liangxiu Han, Tam Sobeih, Lewis Lappin, Mark Lee, Andew Howard, and Aron Kisdi. The thannel-spatia. iicentionbbased vision transformer network for automated, accurate predic- tion of crop nitrogen status from uav imagery. arXiv e-prints, pp. arXiv-21 2021.
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning. pp. 27374.27394. PMLR, 2022.
Xiao Xiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi, Jian Kang, Lichao Mou, Hossein Bagheri, Matthias Haberle, Yuansheng Hua, Rong Huang, Lloyd Hughes, Hao Li, Yao Sun, Guichen Zhang, Shiyao Han, Michael Schmitt, and Yuanyuan Wang- So2sat lcz42: A benchmark data set for the classification of global local climate zones (software and data sets). IEEE Geoscience and Remote Sensing Magazine, 8(3):76-89) 2020a. doi: 10.1109/M DAW.GRS..200..9647000000
Xiaoxiang Zhu. So2sat lcz42 3 splits, 2021
Xiaoxiang Zhu, Jingliang Hu, Chunping Qiu, Yilei Shi. Hossein Bagheri, Jian Kang, Hao Li, Lichao Mou, Guicheng Zhang, Matthias Häberle, Shiyao Han, Yuansheng Hua, Rong Huang. Lloyd Hughes, Yao Sun, Michael Schmitt, and Yuanyuan Wang. So2sat lcz42, 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: De- formable transformers for end-to -o--to-end object detection. arXiv preprint 25X1::2010.04159, 2020b.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
Implementation Details
14
A.1
Hierarchical Channel Sampling
14
A.2
Training with ViT and ChannelVil
14
A.3
Training on datasets with varying channel availability
15
Evaluation across all channel combinations
15
A.5
Dataset details
16
Additional baselines
16
B.
Baseline: Concatenating Features from Multiple Single-Channel ViTS
16
B.2
Baseline: Fully Attentional Networks (FANs)
17
B33
Baseline: Convolutional neural networks (CNNs)
18
Additional analysis
20
C.1
Running time analysis
20
Attention visualization for ImageNet
20
Ablation: tied untied image filters for ChannelViT
Ablation: shared vs. unshared channel embeddings
21
Investigation: do need separate classifier for each channel combination
22
C.6
Breaking down the performance gain on JUMP-CP for each gene target
23
Backbone: Small Base Large
23
8
Performance variations across different channel combinations
24
CamelyonI7-WILDS: Medical Imaging for Histopathology
24
D.I Dataset
24
D.2 Results
25
E
Self-supervised pre-training with Channel ViT
26
DINQ
27
E.2
Linear Probing
27
Results
27
13

---

Published as a conference paper at ICLR 2024
A
IMPLEMENTATION DETAILS
This section clucidates the specifics of our implementation and the settings of our papeer-prameters..
A.1
HIERARCHICAL CHANNEL SAMPLING
In Section 3.2 outlined the channel sampling procedure of HCS. In this subsection, offer comprehensive example of HCS in conjunction with ChannelViT and VIT.
Hierarchical Channel Sampling for ChannelViT Given three-channel input x, per Sec- tion 3.1. the input sequence for the Transformer encoder can be expressed as
[CLS, pos1 + chn1 + Wx Wx(e1,Pil,
poa + chn1 + WI(eI,PN]:
posı + chn2 + Wr[e,,Pi], .... POa + chn2 + Wx[c2, PN]:
pos1 + chn3 + Wxles,Pil, .... pOaN + chny + Wx(cs, PN]]
Let's assume that sampled channel combination from the HCS algorithm is (1,3). The corre- sponding input sequence for the Transformer encoder would then be modified accordingly.
CCLS, posı + chn1 + Wx),,,P1),
POSN + chn1 Wx(es,PN).
pos + chn3 + Wx Wx|c3, .... POan chn3 W Wx(e3, PN]]
It's important to note that reducing the number of channels only modifies the sequence length. Fur- thermore, since we sample the channel combinations for each training step, the channels utilized for each image can vary
different epochs.
Hierarchical Channel Sampling for ViT Given the identical three-channel input æ, the input sequence for the Transformer encoder be articulated as
[CLS, pos1 + Wir(c1,P1] W2r[e2, + W3a(c3,, Pi] b.
pos, Wir[ci, Pm] + W22/c2, Pm] + W3z(ca, pn. 6).
Here Wi, 22, W3 represent the weights associated with each input channel, and is the bias term. Let's continue with the assumption that our sampled channel combination from the HCS algorithm remains 31. We then adjust the above input sequence as follows:
[CLS, pos + Wis/cı, p1 Wyrles, p1 13/2 b.
pos, P/ + W3a Pr, 3/2
It's noteworthy that, in addition masking the input from the second channel. we also rescale the remaining channels by a factor of 3/2. This is akin to the approach of (Srivastava et al. (2014), and is done ensure that the output of the linear patch layer maintains the same scale, despite the reduction in input channels.
A.2 TRAINING WITH VIT AND CHANNELVIT
Backbone the vision transformer backbone, we employ the PyTorch implementation provided by Facebook Researctf Due to computational constraints, primarily utilize the 'vit-small" chitecture, which has an embedding dimension 386, a depth 12, 6 heads, an MLP hidden dimension of 4 x 386 1544 and pre layer normalization. We also briefly experiment 'vit-base" which increases the embedding dimension to 768, the number of heads to 12, and the MLP hidden dimension to 4 x 768 3072. For Via,
retain the parameter settings as its ViT counterparts for the Transformer encoder. Note that ChannelViT has marginally smaller number of parameters, as the first linear projection layer is now shared across channels.
Objective We employ the standard cross-entropy loss for both VIT and Channel ViT across the four image classification benchmarks. Specifically, utilize the Transformer encoder's representation for the CLS token at the final layer, and append linear layer, followed by Softmax function, to predict the probability of each class.
https://gitt hwu.. www.boob.o.oo......o.. Fwceborkk reree roh//di obs/mei www.iiaaannn..oooo trann Former
14

---

Published as a conference paper at ICLR 2024
Optimization For optimization, employ the AdamW optimizer (Loshchilov & Hutter 2019).
The learning rate is warmed up for the initial 10 epochs, peaking at 0.0005 (Goyal et al. 2017). after which it gradually decays to 10 following cosine scheduler. To mitigate overfitting, apply weight decay to the weight parameters, excluding the bias and normalization terms. 'The weight decay starts 0.04 and incrementally increases during training, following cosine scheduler, up a maximum of 0.4. Each model is trained for 100 epochs with batch size of 256. The training is conducted on an AWS p4d. 4x1 argg instance equipped with 8 A100 GPUs.
A.3 TRAINING ON DATASETS WITH VARYING CHANNEL AVAILABILITY
In Table 3 and Figure we investigated scenarios where our training datasets exhibited varying channel availability. This section provides detailed description of the training settings we employed and presents additional results for an alternative setting.
ChannelViT and VIT Despite the different channel combinations in the training datasets, we utilize consistent approach (as detailed in Appendix to encode the images for both ChannelViT and VIT. For Channel this entails having varying sequence lengths for images with different numbers of channels. For ViT, this involves masking out the unavailable channels and rescaling the remaining ones.
Objective We continue to use the cross-entropy loss. However, in this instance, there are two potential methods for data sampling.
Sampling a random batch from each dataset and minimizing their average loss. This ap- proach will assign more weight to datasets with fewer examples. Mathematically, it opti- mizes
D + 122 |
DD1 + 1D2]
Lupsample
Cn,
Lr
D1
D/
where assume Dj and D2 are the training datasets with different channels.
2. Concatenate the two datasets and draw batch from the combined datasets. This approach simply minimizes the average loss
Laverage Lr, L
Our preliminary experiments indicate that the second method consistently outperformed the first. For instance, in JUMP-CP when training with 25% 8-channel data, ChannelViT-S/16 achieves 57.97% when training with Laverage but only reachs 45.52% when training with Lupsumple- Simi- larly, ViT-S/16 achieves 50.29% when training with Laverage but only scores 42.58% when training with Lupsample. We hypothesize that models exhibit overfitting when trained using the upsampling loss. Therefore, we report the numbers for the normal average loss Laverage in Table and Figure 5
A.4 EVALUATION ACROSS ALL CHANNEL COMBINATIONS
To assess the channel robustness of the trained models, enumerate all possible channel combina- tions and report the corresponding accuracy for each.
have considered two training scenarios: the top section pertains to training on 5 fluorescence channels, while the bottom section pertains to training on all channels. For the top section, can evaluate the models for all subsets of the fluorescence channels. This includes
For instance, in Table 2
Combinations with 5 channels: there is only one CO = combination;
Combinations with 4 channels: there are Cş = 5 combinations;
Combinations with 3 channels: there are C = 10 combinations;
Combinations with 2 channels: there are C3 10 combinations;
Combinations with 1 channels: there are C 5 combinations.
15

---

Published as a conference paper at ICLR 2024
Consequently, evaluate total of + 31 channel combinations. Given specific channel combination, we mask out the testing images accordingly (as described in Appendix (A.I) and compute the corresponding testing accuracy. We then report the average accuracy over com- binations that have the same number of channels. As one might intuitively expect, models tend to perform better when provided with more channels.
A.5 DATASET DETAILS
In this section, we provide detailed description of our datasets and their corresponding input chan- nels.
JUMP-CP, 160-way classification We use the processed version of JUMP-CP released by [aoo & Karaletsos 2023 Each image consists of single masked cell and includes five fluorescence channels: AGP, DNA, ER, Mito, RNA, as well as three brightfield channels: HighZBF (Brightficld- 1), LowZBF (Brightfield-2), and Brightfield (Brightfield-3). Each cell has been perturbed by chemical compound, and the goal is to identify the gene target of the chemical perturbation.
So2Sat, 17-way classification
We use the processed version So2Sat released by the original au- thors Thu et al. 2020a Each image patch consists of 8 channels from Sentinel-1:
1. the real part of the unfiltered VH channel;
2. the imaginary part of the unfiltered VH channel;
3. the real part the unfiltered VV channel;
4. the imaginary part of the unfiltered VV channel;
5. the intensity of the refined LEE filtered VH channel;
6. the intensity of the refined LEE filtered VV channel;
7. the real part of the refined LEE filtered covariance matrix off-diagonal element;
8. the imaginary part of the refined LEE filtered covariance matrix dia-oiagn element.
and 10 channels from Sentinel-2: Band B2, Band B3, Band B4, Band B5, Band B6, Band B7, Band B8, Band B8a, Band B 1 and Band B12. The task is to predict the climate zone for each respective image patch, with a total of 17 distinct climate zones being represented.
B ADDITIONAL BASELINES
B.1 BASELINE: CONCATENATING FEATURES FROM MULTIPLE SINGLE CHLNNEL VITS
Hussein et al. (022)) utilized ViT' for epileptic seizure predictions, proposing method to train multiple ViTs, one for each input channel. The final image representation is derived by aggregating the output CLS tokens across all single-channel ViTs. An MLP is then attached to these aggregated features to predict the image label. In this section, we implement this baseline based on the paper, termed ViT, and evaluate its performance both with and without HCS.
our results on JUMP-CP when training using all eight channels. Without HCS, Mul- tiVIT underperforms compared to ViT when evaluated on all channels, despite having eight times more parameters. This underscores the importance of parameter sharing across different channels to combat overfitting. However, when testing on a subset of channels, MultiViT outperforms ViT, as each VIT operates nels. Interestingly, MiltiViT does not perform well with HCS. While accuracy improves when testing subset of channels, the accuracy significantly decreases (from 49.06 to 30.25) when us- ing all eight channels. We hypothesize that this is due to the channel-wise wise feature aggregation being performed after the single-channel ViTs, preventing the model from conditioning the representation based on the input channel availability.
single channel, thereby improving robustness to changes in the input chan-
pa /// /tg con/
hetpe://gg //tgt http///tttt...//////o//oo.oo wtwob...om sho- Sett -C442
16

---

Published as a conference paper at ICLR 2024
Table 5: 160-way accuracy MultiViT (Hussein et al. 2022) on JUMP-CP. All models are based on the ViT-S/16 backbone and are trained on all 8 channels. During testing, all possible channel combinations are evaluated and we report the mean accuracies for combinations with the same number of channels. VultiViT learns separate VIT per channel and aggregates their output CLS tokens together to form the overall image representation. Since the ViT encoder separate for each channel, it offers better channel robustness than the vanilla VIT. However, if we focus on the performance using all channels, it actually leads to worse performance than the vanilla ViT-S/16, highlighting the importance of parameter tying on the application of cell imaging.
VIT
MultiVIT S/16
ChannelViT S/16
ViT
MultiViT S/16
Channel VIT S/16
S/16
S/16
Use hicrarchical channel sampling?
X
X
8 channels
52.06
49.06
66.22
56.87
30.25
68.09
7 channels
5.91
34.10
41.03
49.35
29.04
61.02
#channels for testing
channels
1.81
23.77
24.57
42.38
27.44
53,45
channels
17.09
14.20
35.78
25.69
45.50
4 channels
2.38
12.98
8.56
29.84
23.96
37.37
3 channels
2.70
10.58
5.65
24.94
22.34
29.68
channels
2.63
9.61
3.24
21.54
20.89
23.77
channel
7.97
2.08
19.92
19.85
20.84
Table 6: 160-way test accuracy of FAN (Zhou et al. 2022) on JUMP-CP. All models are trained on all channels. During testing, we evaluated all possible channel combinations and reported the mean accuracies for combinations with the same number of channels. FAN incorporates channel- wise self-attention mechanism following the standard location-wiss wise self-attention in the transformer encoder. This enhances the model's ability to reason across both input and hidden channels, outper- forming the ViT baseline, However, remains sensitive to availability of input channels.
Without HCS
With HCS
FAN
FAN
FAN
FAN
#channels for testing
VIT
S/16
S/16
ChannelViT S/16
VIT S/16
S/16
S/16
Cha S/16
S/16
(conv
(linear patch)
(conv
(linear patch)
patch)
patch)
8
52.06
65.13
65.42
66.22
56.87
3.49
20.31
68.09
7
5.91
1.24
3,63
41.03
49.35
3,88
20.52
61.02
6
1.81
0.64
4.82
24.57
42.38
3.96
17.46
53.45
5
2.46
2.11
6.62
14.20
35.78
3.15
15.17
45.50
4
6.68
8.56
29.84
3.92
11.74
37.37
3
2.70
5.03
6.03
5.65
24.94
4.54
9,42
29.68
2
2.63
4.36
5,97
3,24
21.54
2.21
6.65
23.77
1
3.00
2.68
2.92
2.08
19.92
2.90
2.52
20.84
We find that Cha significantly outperforms MultiViT. There are three key differences be- tween the two models:
1. ChannelViT learns single VIT across all channels, rather than one VIT for each channel;
2. ChannelViT is aware of the input channel availability at the input patch sequence, while the single-channel ViT's in ViT operate independently;
3. ChannelViT allows cross-channn cross-location attention, while MultiViT only permits cross-location attention.
B.2 BASELINE: FULLY ATTENTIONAL NETWORKS (FANs)
Zhou et al. 2022) introduced family Fully Attentional Networks (FANs) that combine channel- wise attention with the ML.P in transformer encoder layer. Notably, the channels in this context
17

---

Published as a conference paper at ICLR 2024
Table 7: 160-way test accuracy ResNet50 and ResNet152 (Zhou et al. 2022 on JUMP-CP. All models are trained on all 8 channels. ResNets are trained without HCS. VIT and ChannelViT are trained with HCS. During testing, we evaluated all possible channel combinations and reported the mean accuracies for combinations with the same number of channels.
Model
ResNe150
ResNet152
ViT-S/8
ViT-S//ViT-S/S
#parameters
25M
60M
22M
22M
Use hierarchical
channel sampling?
8 Channels
65,96
66.54
66.44
74.77
7 Channels
2.39
3.05
59.01
68.42
#channels for testing
6 Channels
2.22
4.29
51.29
61.26
5 Channels
1.57
5.35
43.39
53.05
4 Channels
1.19
5.91
35.60
43.87
3 Channels
0.78
5.69
28.59
34.19
2 Channels
0.56
4.29
23.32
25.73
1 Channels
0.51
2.66
20.41
21.20
extend beyond the input channels. FANS aggregate feature channels with high correlation values across the transformer encoder layers and isolate outlier features with low correlation values.
We adopted the implementation provided master /wwde 1 s fan. py and evaluated the FAN small with a patch size of 16 by 16. It's worth noting that FAN, by default, employs four stacks of by 3 convolution layers (each followed by GELU activations) to construct the input patch tokens, whereas VIT and Channel ViT use single linear layer over the 16 by 16 input patches. We refer this FAN baseline as FAN S/16 (conv patch). We also experimented with replacing these convolution layers with the same linear projection used the regular ViT, terming this modified version of FAN as FAN S/16 (linear patch).
stttps://githi
com Coom/NVlabe FAA//bob//
presenss our results on FANs. Without HCS, the default FAN-S/16 (conv patch) significantly outperforms ViT (65.13 vs 52.06), demonstrating the effectiveness of cross-channel attention. How- ever, it still falls short of ChannelViT (65.13 13 vs. 66.22). Furthermore, when evaluated using subset of channels test time, its performance significantly declines (1.24 vs. 41.03 on channels). terestingly, observed that the FAN with linear patch embedding layer performs slightly better than the default FAN with convolution patch embeddings.
We also investigated training FANs with HCS. We discovered that FAN with convolution patch embeddings struggled to learn a meaningful classifier. Replacing the convolution layers with a simple linear transformation improved the performance, and we observed that when trained with HCS, FAN-S/16 (linear patch) outperforms its counterpart without HCS when evaluated on a subset of channels. However, the performance is still significantly lower than the regular ViT-S/16. We hypothesize that since FANs explicitly leverage the correlation between different hidden channels to build its representations, it becomes more sensitive to channel perturbations at test time.
In conclusion, we highlight the key differences between ChannelViT and FANs:
1. VhT performs cross coossechaneel and cross-location attention jointly, meaning that each patch token can attend to a different channel at a different location.
2. ViT maintains the distinction of different input channels throughout the trans- former encoder and tie the transformer encoder across channels, which we argue enhances robustness to channel changes.
B.3 BASELINE: CONVOLUTIONAL NEURAL NETWORKS (CNNS)
In this section, we further compare the Vision Transformer (ViT) and ChannelViT with conventional Convolutional Neural Networks (CNNs). We use the widely-adopted ResNet-50 and ResNet- 152 as our baseline models, as described by He et al. (2016). We present the number of parameters for each model and their corresponding performance the JUMPCP dataset in Table
18

---

Published as a conference paper at ICLR 2024
Initially, trained the ResNet baselines using Hierarchical Channel Sampling (HCS), but this approach led to training instability, with the top-1 accuracies of both models converging to ap- proximately 5% by the end of training. Without HCS, ResNet-50 and ResNet-152 exhibit perfor- mance comparable to the ViT-S/8 baseline. Despite having three times more parameters, ResNet-152 achieves only slight improvement over ResNet-50. When compared with Channe/ViT-S/8, there still remains a significant performance gap.
We hypothesize that the parameter sharing within ChannelViT enables the efficient and robust con- struction of channel-invarian filters. Conversely, the explicit cross-channel attention in ChannelViT effectively facilitates the model's ability to infer relationships across related channels.
Single-channel view
8-channel view
AGP
DNA
ER
Mito
RNA
BF-1
BF-2
BF-3
Input
UGT1A9
Attention
Input
GJB4
Attention
Input
TUBB
Attention
Input
11SP90AAI
Attention
VIT
ChannelViT
Figure 6: Extra visualizations of the relevance heatmaps for both ViT-S/8 (8-channel view) and ChannelViT-S/8 (single-channel view). Both models are trained on JUMP-CP using HCS across all channels. ChannelViT offers interpretability by highlighting the contributions made by each individual channel.
19

---

Published as a conference paper at ICLR 2024
Table 8: Time efficiency of different models on the JUMP-CP dataset. All models are trained and evaluated using all eight channels. We use GPU cluster equipped with eight A100 GPUs for the evaluation, Section B.3 provides a more detailed analysis for ResNet models.
Model
#parameters
Training time
Inference time
8-channel accuracy
ResNet50
25M
3.9 hours
65.8 see
65.96
ResNet152
60M
4.4 hours
81.8
66.54
ViT-S/16
22M
2.8 hours
54.5 sec
56.87
ChannelViT-S/Ie w/o HCS
22M
12.1 hours
91.0 sec
66.22
ChannclViT-S/16 w/ HCS
22M
10.2 hours
90.7 sec
68.09
Table 9: ChannelViT-S/16: tied image filter vs. untied image filter. Both models are trained on the five fluorescence channels in JUMP-CP, with HCS applied during training. Tying the image filter weights across channels enhances both the performance and robustness.
Tied linear
x
projection weights?
5 channels
54.78
56.78
#channels for testing
4 channels
43.88
45.94
3 channels
33.67
35.45
2 channels
25.57
26.57
channel
21.07
21.43
C ADDITIONAL ANALYSIS
C.I RUNNING TIME ANALYSTS
Our proposed ChannelViT model, which expands the channel dimension into the sequence length dimension, introduces an inherent increase computational cost. As shown Table the training duration for the ViT-nelViT-S/166 model the JUMP-CP dataset, utilizing all eight channels, is significantly longer without the application of Hierarchical Channel Sampling (HCS). However, the integration of HCS results in 15% reduction in training time, decreasing from 12 hours and 6 minutes to 10 hours and 17 minutes. This demonstrates that HCS not only bolsters the model's robustness but also markedly enhances training efficiency.
In terms of inference cost, ChannelViT exhibits 1.6 times increase in processing time compared its ViT counterpart, yet it achieves an 11.22% higher accuracy (on absolute scale). When measured against the better performing ResNet- 152 baseline, Channel ViT's inference time is only 1.1 times longer.
In this paper, we have explored the Via utilizing the standard quadratic attention mecha- nism. Looking ahead, it would be intriguing to investigate the integration of Channel with more efficient algorithms, such as Linformer Wang et al. (2020) and LongNet Ding et al. (2023). which scale linearly with sequence length. Such combinations could potentially yield further improve- ments in both performance and computational efficiency.
C.2 ATTENTION VISUALIZATION FOR IMAGENET
Figure illustrates the attention heatmaps for VIT and ChannelViT models trained on the Im- ageNet dataset. For each image, we generate the rolled out attention scores for two dis- tinct
and wine for the top image, and elephant and zebra for the bottom im- age foce-follwwingeee Chefer et al. (02T). We observe that ChannelViT precisely focuses attention on the relevant channels, such the red channel when predicting red wine. In scenarios where the contrast pattern, such as black and white for a zebra, is distributed across all channels, Channel effectively utilizes all channels inform its prediction.
20

---

Published as a conference paper at ICLR 2024
Iepal Imreg (BGB)
161:5/16 Altaniion Map
Tipur Iaage (RGH Charnels)
Chamd Altoriion Map
sifcae not urn
dhe 386
dbs 30
2ahea
Figure 7: Relevance visualizations for ViT-S/16 and ChannelViT-S/16 trained on ImageNet. For each image, we generate the relevance heatmap for two distinct classes (espresso and wine for the top image, elephant and zebra for the bottom image) using the methodology described in Chefer jct al. (202I). It's observed that ChannelViT precisely allocates its attention to the relevant channel (red channel for predicting red wine). In the case of predicting zebra, where the black and white contrast pattern is present across all channels, Channel utilizes all channels for its prediction.
Clasetter
Transtoreer Encoder
pole
piodo
poow
pale
poss
CLS
Oftic
citi
oftin
cifitiy
chiu
ofiie
Input Image (R-channe)
Crannevil cach individul chardd.
Figure& 8: Illustration of ChannelViT with untied image filters. Each channel is assigned its unqiue linear projection weight, denoted as We. Contrarily, in Figure all channel share a common image filter, represented by W.
C.3 ABLATION: TIED VS. UNTIED IMAGE FILTERS FOR CHANNELVIT
In the main paper, we introduced ChannelViT with a linear projection layer tied across various channels. This section delves into the exploration of flexible weights for each channel (Figure B). The input sequence to the Transformer encoder can be represented as follows:
[CLS, pos + chn, + Wi
POSN + chn, + Wir[eI,PN].
pos + chnc + Wer|cc,pıl, .... POSN + chnc + Wealco.
where Wi...., We denote the linear transformations associated with the input channels. Table y showeases our findings on JUMP-CP.
observed that VhannelViT, when trained with tied image filter weights, consistently outperforms its untied counterpart. We hypothesize that the first layer filters are generally shareable across channels, and tying the parameters can prevent overfitting, thereby enhancing the model's robustness.
C.4 ABLATION: SHARED VS. UNSHARED CHANNEL EMBEDDINGS
In this section, we conduct an ablation study to investigate the impact of channel embeddings on the performance of ChannelViT models. Specifically, we consider the following simplification of
21

---

Published as a conference paper at ICLR 2024
Shard chonnnel smbbeidingg:
chn
Mean
ctry
Closeiter
Transtomer Encoder
pilk
pois
plou
polw
poor
poss
pod-
pon
CLS
chn
oftin
citin
cltin
citin
ctn
cltn
chn
Input image (l-channel)
Grannelvil creates patch tokens for each individual charns.
Figure 9: Illustration of Channel ViT with shared channel embeddings. We investigate the impace of channel embeddings on the performance of Channel ViT. Specifically, we set replaced each channel embedding by the mean channel embeddings across all channels. The resulting performance is presnted Table TO
Ablation study on the channel embeddings of channelViT. The ChannelViT models, trained on JUMP-CP and So2Sat with hierarchical channel sampling across all channels, are uti- lized for this ablation study. For the top row (V), the mean channel embeddings across all channels computed and used to replace the channel embedding for each individual channel (see Fig- ure 9). Observations indicate that the preservation of original channel embeddings is critical for both datasets. Notably, ChannelViT exhibits higher sensitivity to modifications in channel embed- ding on JUMP-CP as compared to So2Sat.
Table 10:
ChannelViT- on JUMP-CP
ChannelViT on So2Sat
Shared channel
fluorescence fluorescence & brightfield channels)
Sentinel-1 (8 channels)
Sentinel-1 I & -2 (18 channels)
embedding?
channels)
1.26
2.49
10.44
52.13 63.01
57.60
68.09
47.39
ChannelViT where we have shared channel embedding across all channels:
[CL.S, pos1 + chn + Wx(ci,P1),
pOS + chn + Wx[es,Px],
... pos + chn + W Wxlec,Pil Pil,
poa + chn + Wx|ee, Px]].
We consider VhannelViT's trained on both JUMP-CP and So2Sat. A natural way to define chn is to set it as the mean embeddings of the learned channel embeddings:
Σ chn
chn C
We present our ablation study in Table 10 performance, underscoring the importance of maintaining the original channel embeddings. Inter- estingly. the ChannelViT model demonstrates higher degree of sensitivity to alterations in channel embedding on the JUMP-CP dataset as compared to the So2Sat dataset. This suggests that the specific characteristics of the dataset influence the model's reliance channel embeddings.
We observe that this modification significantly harms the
C.5 INVESTIGATION: DO WE NEED A SEPARATE CLASSIFIER FOR EACH CHANNEL COMBINATION?
The application of hierarchical channel sampling results in the model receiving variety of input channel combinations, leading to significant changes in the input distribution. This prompts an investigation into whether it's necessary further condition the final classifier based on the sampled channel combinations. Table II presents our ablation analysis, where we consider three methods for incorporating the information of the input channels into the final classifier:
1.
The first baseline involves learning a separate linear classifier on top of the ViT embeddings for each channel combination.
22

---

Published as a conference paper at ICLR 2024
Table 11: Assessing the necessity of conditioning the classifier on input channel combinations dur- ing hierarchical channel sampling. The backbone models, ChannelViT-S/16, are trained and tested on channels. Our findings suggest that the simplest shared linear classifier (bottom) delivers su- perior results, eliminating the need for extra conditioning. We hypothesize that the utilization of a shared linear classifier contributes to regularization of the model's internal representation, thereby enhancing its robustness across channels.
Additional features
Classifier
Classifier shared across channel combinations?
Accuracy
besides last-layer '[CLS)
on top of the features
Informing the final classifier of the sampled channel combination
None
Linear
26.56
Embeddings for each channel comb.
MLP
61.98
One-hot encoding each channel comb.
MLP
66.86
ChannelViT
None
Linear
68.09
Figure 10: Accuracy gain of ChannelViT-S/8 over ViT-S/8 over all cell labels (gene targets) on JUMP-CP. Both models are trained using HCS over all 8 channels.
2. The second baseline learns an embedding vector for each channel and constructs the repre- sentation for the sampled channel combination by summing up all the embeddings for the selected channels. This representation then concatenated with the ViT representation and fed to a shared MLP with one hidden layer.
3. The third method is similar to the second baseline, but uses one-hot encoding as the repre- sentation for the sampled channel combination.
Our observations indicate that all three methods underperform when compared to the basic Chan- ViT, which uses shared linear classifier across all channel combinations. We hypothesize that the shared linear classifier regularizes the ViT to embed inputs with different channel combinations into the same space. This bias appears enhance robustness and performance.
C.6 BREAKING DOWN THE PERFORMANCE GAIN ON JUMP FOR EACH GENE TARGET
In Figure 1O we delve into comparative analysis of the performance between Channel and VIT-S/8 across each cell label (gene target). Our figure reveals that ChannelViT surpasses ViT in 90% of the gene targets, while underperforming in the remaining targets. It's important to note that the gain is computed from 160-way classification task, where the models are trained to optimize the average loss across all gene targets. If we reframe the problem using multi-task learning objective, the distribution of gains per gene could potentially differ, and we expect the improvements of ChannelViT to be more consistent.
C.7 BACKBONE: SMALL VS. BASE VS. LARGE
In the main body of the paper, we explored the ViT and ChannelViT across different resolutions (16x16 and 8x8). To provide comprehensive analysis, extend our investigation include
23

---

Published as a conference paper at ICLR 2024
Table 12: Configurations of ViT and ChannelViT with different backbone sizes.
size
embed dim
depth
num heads
MLP hidden dim
Small
384
12
1536
Base
768
12
12
3072
Large
1024
24
4096
Table 13: Test accuracy of 160-way perturbed gene prediction on JUMP-CP. All models are trained on 8 channels, which includes additional brightfield channels. During testing, all possible channel combinations are evaluated and report the mean accuracies for combinations with the same number of channels. Please refer to Table 12 for detailed model configurations.
VTT (trained with HCS)
ChannelViT (trained with HCS)
Small/16
Basc/16
Large/16
Small/16
Base/16
Large/16
channels
56.87
57.85
57.96
68.09
68.53
68.87
channels
49.35
50.35
50.93
61.02
61.56
62.01
#ichatnels for lesting
channels
42.38
43.98
43,98
53.45
53.53
53.59
channels
35.78
37.26
37.26
45.50
45.96
46.54
4 channels
29.84
30,82
30.82
37.37
37.90
38.09
3 channels
24.94
25.37
25.37
29.68
29.96
30,06
channels
21.54
21.73
21.73
23.77
23.78
23.62
channel
19.92
20.04
20.04
20.84
21.61
21.06
various backbone sizes. Adhering to the conventions established by Dosovitskiy et al., we evaluated the performance of the ViT and Channel models with small, base, and large backbones. The specific configurations of these models are detailed in Table
Performance metrics for the different model sizes are presented in Table 13 incremental performance improvements both ViT and ChannelViT as the number of parameters increases. Concurrently, the performance disparity between VIT and ChannelViT remains consistent and significant.
We note trend of
C.8 PERFORMANCE VARIATIONS ACROSS DIFFERENT CHANNEL COMBINATIONS
Table 14 presents analysis of the standard deviation for both VIT and ChannelViT on the JUMP- CP dataset, considering all channel combinations. We report the mean accuracies for groups cat- egorized by the same amount of channels. It is important note that, despite maintaining a con- stant number of channels, the informational content of different channel combinations can differ markedly, which reflected in the substantial standard deviations observed in the table.
To further dissect and comprehend this variance, we examined the performance gains of Chan- VelViT over VIT for each individual channel combination. The mean improvements, along with their standard deviations, are presented in Table 15] Our analysis substantiates that the performance enhancements attributed ChannelViT are not only consistent across various combinations but also notably significant.
D CAMELYON
MEDICAL IMAGING FOR HISTOPATHOLOGY
In this section, we introduce another dataset, Camelyon 17-0looO which was not included in the main paper due to space limitations.
D.1 DATASET
The Camelyon 7- WILDS dataset encompasses 455k labeled images from five hospitals. The task involves predicting the presence of tumor tissue in the central region of an image. Although the dataset employs standard RGB channels, these derived from the hematoxylin and eosin stain-
24

---

Published as a conference paper at ICLR 2024
Analyzing the standard deviation of 160-way perturbed gene prediction on JUMP-CP. We evaluated all possible channel combinations during testing. We report the mean accuracies for groups of combinations that include the same number of channels. However, even when the number of channels is held constant, the amount of information conveyed by different channel combinations can vary significantly, leading to a large standard deviation in our results. To isolate and understand this variance, we present the mean and standard deviation of the performance differences between ChannelViT and VIT for each channel combination in Table I5
Table 14:
ViT-S/16
VhannelViT--/16
VIT-S/8
Channel
Use hicrarchical
channel sampling?
Training on all 8 channels (5 fluorescence channels 3 brightfield channels)
channels (C2 = 1)
56.87
68.09
66.44
74.77
7 channels (C² 8)
49.35119.38
61.02:19.78
59.011100007
68.42.19.11
#channels for testing
6 channels (Cğ 28)
42.38:110.6
12.45=12.40
51.29112.44
61.26:111.91
5 channels (Cğ = 56)
35.78+10.18
43.50+13.23
43.89++1289
53.05++13441
4 channels (C) 70)
29.84++.322
37.37+12.25
35.50
43.87+13.36
3 channels (C3 56)
24.94455.43
29.68+9.22
28.59+8.38
34.19+11.10
2 channels (C2 28)
21.54:2..7
23.77=4.89
23.32:+4.27
25.73:16.57
channel (Cf 8)
19.92:40.51
20.84+1.64
20.41#1.26
21.20::2.17
Improvements of ChannelViT over its VIT counterpart for the JUMP-CP microscopy cell imaging benchmark. We evaluated all possible channel combinations during testing. For each specific combination, we calculated the performance gains of ChannelViT over ViT. These improve- ments are summarized mean values with corresponding standard deviations (std) for groups cat- egorized by the same number of channels. Despite the considerable variability inherent in different channel combinations, our findings reveal that the performance enhancements of ChannelViT both consistent and substantial.
Table 15:
ChannelViT-S/It over ViT-S/16
ChannelViT-S/EE over ViT-S/16
ChannelViT-S/8 over ViT-S/8
Use hierarchical channel sampling?
x
Training on all 8 channels (5 fluorescence channels & 3 brightfield channels)
8 channels (C$ 1)
14.16
11.22
8.32
channels (C 8)
35.13 1 18.37
11.67 1 1.17
9.41 1.80
#channels for testing
6 channels (C) 28)
22.76 + 18.64
11.07 2.22
9.96 + 1.90
5 channels (C. 56)
11.75 11.33
9.72 3.46
9.66 +
4 channels (Cî 70)
6.18 6.86
1.52+4.11
8.27 + 3.08
3 channels (C) 56)
2.95 5.27
1.96
5.60 3.58
2 channels (C 28)
0.61
2,24 1 2.62
2.41 2.82
1 channel (Cî 8)
-0.92
0.93 + 1.15
0.79 + 0.95
ing procedure, which can vary across hospitals. We adopt the processed version from the WILDS benchmark
D.2 RESULTS
Table 16 presents our results for Camelyon17. 17, medical imaging benchmark for histopathology. Given the smaller image size (96 by 96), employ patch size of by 8 for the ViT backbone.
Starting with the standard ViT-S/8 (first column), we note that it achieves an accuracy of 99.14 for the in-distribution hospitals. With HCS, it also attains an accuracy of over 97 when using only two or one channels for predictions. However, when evaluated on out-of-distribution hospitals, its 3-channel accuracy drops to 83.02. This is not only lower than its in-distribution performance,
h/tt Ids. strdddddeedde
25

---

Published as a conference paper at ICLR 2024
Table 16: Test accuracy of binary cancer classification on CamelyonI7-WILDD We consider all channel combinations and report the mean accuracy over combinations with the same number of channels. All models are trained with HCS. We observe 1) tying the linear patch projection layer across channels improves out-of-distribution generalization; 2) Cha outperforms ViT on out-of-distribution hospitals.
ViT-S/8
ViT-S/8
ViT-B/8
Cha-nelViT-S/8
Channe/ViT-S/8 ChannelViT-BBi
Tied weights
x
x
channels?
Evaluation on in-distribation hospitals
3 channels 99.14
98.46
98.28
98.98
98.99
99.13
channels 98.65
98.42
98.22
98.51
98.66
98.73
channel
97.59
98.24
97.98
97.71
98.14
98.11
Evaluation on out-of-distributiin hospitals
channels
83 .2.02
89.14 19.14
88.57
89,96
92.67
91.39
2 channels
85.12
88.78
88.32
88.11
88.25
87.17
channel 87.97
87.19
86.93
87.04
88.30
87.60
Table 17: Top-1 Accuracy on ImageNet Using DINO Pre-training with ViT and ChannelViT. We apply DINO pre-training with both VIT and ChannelViT the ImageNet training data. Upon completion of the pre-training phase, we conduct the standard linear probing evaluation and the resultant validation accuracy is reported. Hierarchical channel sampling is not used we found that it introduces extra instability during the DINO pre-training phase. The findings indicate that 1) In comparison to supervised training, DINO inherently enhances the channel robustness for ViT; 2) Cha consistently outperforms VIT in significant manner across all evaluations.
Val Acc. on RGB
Val Acc. on R-only
Val Ace. on G-only
Val Acc.
Backbone
on B-only
Models trained on three channels (RGB)
Supervised ViT-S/16
71.49
29.39
33.79
21.18
DINO + ViT-S/16 LinearProb
72.62
64.34
65.46
61.12
DINO VhannelViT-S/16- + LincarProb
74.38
67.44
67.85
65.97
Expert DINO models pre-trained only channel
DINO ViT-S/16 (R-only) LinearProh
67.76
DINO + ViT-S/16 (G-only) LinearProb
68.09
DINO + ViT-S/16 (B-only) + LinearProb
66.65
but also lower than the accuracy achieved when using only one channel for evaluation in the out- of-distribution hospitals (87.97). We hypothesize that this discrepancy is due to the staining shift across hospitals Gao et al. (2022). The mismatch in color distributions results out-of-distributioo inputs for the first linear patch embedding layer. To test this hypothesis, we experiment with tying the parameters across different channels for the first linear patch embedding layer. As seen in the second column, ViT-S/8 with tied weights, while performing slightly worse in the in-distribution hospitals, performs significantly better in the out-of-distribution setting. We also explore ViT-B/8 but found exhibited overfitting.
By default, we share the first linear patch embedding layer across different channels for ChannelViT. On the -uitof-distribution hospital, ChannelViT-S/s significantly outperforms ViT-S/8 (92.67 vs. 89.14). We also observe that if we untie the weights for different channels in Channel the generalization performance degrades.
E SELF-SUPERVISED PRE-TRAINING WITH CHANNELVIT
This section delves into the integration of self-supervise learning with Vha
26

---

Published as a conference paper at ICLR 2024
E. DINO
We use the DINO algorithm (Caron et al., 2021) for self-supervised learning. It involves a self- distillation process where the student model, provided with local views of the input image, has to learn from the teacher model which has the global views of the same input image.
We follow most of the the configuration suggested by DINO repository m Specifically, we pre-train DINO with ViT-S/16 and Channel for total of 100 epochs on ImageNet with a batch size of 256. The AdamW optimizer (Loshchilov & Hutter 2019) is employed, and the learning rate warm-up phase is set for the first 10 epochs. Given our batch size, the maximum learning rate is set 0.0005, in line with recommendations from You al. (2018). The learning rate subsequently decayed using cosine learning rate scheduler, with target learning rate of 10-6, Weight decay applied to all parameters, excluding the biases. The initial weight decay is set to 0.04 and gradually increased to 0.4 using a cosine learning rate scheduler towards the end of training. The DINO projection head utilized has 65536 dimensions, and batch normalization is not employed in the projection head. The output temperature of the teacher network is initially set to 0.04 and is linearly increased to 0.07 within the first 30 epochs. The temperature is maintained at 0.07 for the remainder of the training. To enhance training stability, the parameters of the output layer frozen during the first epoch.
E.2 LINEAR PROBING
Upon the completion of the pre-training phase, the parameters of both ViT and ChannelViT are frozen. In alignment with the methodology proposed by Caron et al. (2021), the final four layers of the CLS representation are concatenated to represent the image. Subsequently, a linear classi- fier is trained this image representation. The training of the linear classifier is conducted using SGD, with a learning rate of 0.005 and a momentum value of 0.9. The learning rate is decayed in accordance with cosine annealing scheduler. We train the linear classifier for 100 epochs using the ImageNet training split. Once training is done, we report its Top-1 accuracy on the validation split.
E.3 RESULTS
Table 17 showcases our results. It is noteworthy that hierarchical channel sampling is not used during DINO pre-training due to its potential to introduce additional instability to the self-distillation objective. However, we observe that DINO-pretrained ViT inherently provides superior channel robustness. Compared to the supervised ViT-S/16. it achieves 64.34 the red-only evaluation, which is 34.95 better than its supervised version. Furthermore, the integration of prrotpregr with ChannelViT consistently enhances performance across all evaluations, bridging the gap towards the expert DINO model that is pre-trained on each individual channel.
Pettss//gittnne wtw.g//ttooooo...oo. Facebookreeeerchhhok
27


Reviews:
[{"review_id": "xIHJuGWRoS", "reviewer": "Reviewer_PjTm", "summary": "While the Vision Transformer has demonstrated robust performance with real-world images, its capacity to process multi-channel images, such as those from satellites, is somewhat constrained. To address this limitation, the authors have enhanced the conventional vision transformers by introducing a Hierarchical Channel Sampling (HCS) approach, which tackles the issue of sparse input channels. The proposed ChannelViT model has exhibited impressive results across three datasets, outperforming the traditional vision transformer.", "questions": "-    Is HCS also employed during the testing phase? It was mentioned that HCS simulates a test-time distribution during training.\n-    Regarding Table 2, when selecting channels for testing, are these chosen at random, or were all possible combinations tested? If it's the latter, could you provide the mean and standard deviation of the results?\n-    Has the model's performance been evaluated on datasets with varying channel availability, where some data might have only partial channels while other portions are fully channeled?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+  The Hierarchical Channel Sampling (HCS) module enhances robustness by performing channel-wise sampling, which proves beneficial in scenarios involving incomplete image channels.\n+  ChannelViT surpasses the conventional Vision Transformer (ViT) by demonstrating insensitivity to the number of input image channels, where ViT shows vulnerability.\n+    A novel two-stage sampling algorithm is introduced within ChannelViT to selectively obscure input channels, optimizing the model's performance.", "weaknesses": "-    There is a potential risk of information loss, as highlighted in Section 3 of the methodology. The model's approach to segmenting the input image into various channel sequences and processing them individually could disrupt the alignment of channels, particularly in a 3-channel image prediction task.\n-    The patch embedding technique described in Section 3.1 overlooks the issue of channel alignment when deconstructing images into separate channels.\n-    The innovation of the proposed method warrants further scrutiny. Elements such as patch embedding, positional embedding, and Transformer Encoders, as discussed in Section 3.1, do not significantly diverge from the traditional Vision Transformer framework. These aspects should be acknowledged in the Related Work section, with a stronger emphasis on the unique contributions of this research.\n-    The assertion in Section 3.2 that 'HCS guarantees equitable sampling across each m' lacks intuitive justification. While empirical results support this claim, a theoretical rationale would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "skCLRENkJx", "reviewer": "Reviewer_ptju", "summary": "This work presents ChannelViT, a new ViT variant that: 1) learns a channel embedding for each channel, and 2) proposes a technique called Hierarchical Channel Sampling (HCS) for robustness to partial channels. Together, ChannelViT is reported to demonstrate better robustness to missing channels. Multiple experiments were performed, such as: 1) different model architectures (Channel variants of ViT-S/16 and ViT-S/8), 2) comparison with and without HCS, 3) comparisons across multiple benchmarks including microscopy imaging, 4) comparisons across varying number of channels available, 5) exploration into SSL compability in DINO, and others.", "questions": "Summarizing my above concerns, though this work would benefit from having additional experimentation related to its targeted application in multiplex imaging, such as:\n- Additional evaluation on relevant datasets in multiplex imaging, such as RXRX1 and TissueNet\n- Discussion and benchmark evaluation against label-free approaches in multiplex imaging\n- Ablation experiments with ViT-B and ViT-L, with expanded discussion on the intended usage of ChannelViT w.r.t. efficiency", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- ChannelViT with HCS proposes a simple but relatively intuitive extension of ViTs, which would have unique applications in multiplexed imaging in which not all \"channels\" (e.g. - fluorescent probes) are made available. Overall, this work presents a method that targets an important application and may enable significant biological / clinical findings in multiplexed imaging.\n- The ablation experiments regarding assessment of partial channels, comparisons and baselines with training on single channels, and experimentation with DINO are thoughtful and well-organized. In particular, the assessment of partial channels and pretraining with DINO highlights its adaptability as a ViT in potentially replacing vanilla ViTs. Table 2 and Figure 4, which demonstrates the performances of Channel ViT (in comparison to vanilla ViT) on JUMP-CP with partial channels, demonstrates minor but consistent improvement.", "weaknesses": "- While the main application of ChannelViT is in targeting problems such as those in multiplexed imaging due to the challenge of generalizing encoders across datasets that would have the same set of probes, only one dataset explored in this work is related to multiplexed imaging. Experimentation on ImageNet, C17-WILDS, and others are informative and appreciated, are not as relevant to the ultimate application that ChannelViT serves. Though C17-WILDS is microscopy, hematoxylin and eosin (H&E) pathology is not a domain where only the hematoxylin or eosin stains are performed. Rather, it would be more interesting to explore this problem on other relevant multiplexed imaging benchmarks such as RXRX1 [1], RXRX1-WILDS [2], TissueNet [3], and others [4-6]. To this end, it would also be useful to discuss and compare ChannelViT in context with label-free approaches, which suggest channel synthesis as a paradigm for addressing missing channels.\n- As a drop-in replacement for ViT in conventional natural image classification or medical imaging tasks, one concern I have (raised in the work) is lack of experimentation on model scale. The title of this work, \"Channel Vision Transformers: An Image Is Worth C x 16 x 16 Words\", is similar to a title of a related seminal work in ViTs [7], but does not evaluate and demonstrates the scale of Channel ViTs in the same manner in demonstrating its universality of overtaking  current state-of-the-art architectures. As ViT-Small architectures are not the most commonly-used model size for ViT, it would be informative to explore Channel ViTs at greater scale (such as ViT-B and ViT-L). An additional limitation connected to this is the efficiency of learning channel embedings, which may have been the reason for not developing larger ViT models. As shown in the DINO experimentation, even with HCS, training is approximately 3.64x more expensive than its baseline comparison, which may limit its usability. \n\nReferences:\n1. Haque, I., 2023. Rxrx1: A dataset for evaluating experimental batch correction methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4284-4293).\n2. Sypetkowski, M., Rezanejad, M., Saberian, S., Kraus, O., Urbanik, J., Taylor, J., Mabey, B., Victors, M., Yosinski, J., Sereshkeh, A.R. and Haque, I., 2023. Rxrx1: A dataset for evaluating experimental batch correction methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4284-4293). (https://wilds.stanford.edu/datasets/)\n3. Greenwald, N.F., Miller, G., Moen, E., Kong, A., Kagel, A., Dougherty, T., Fullaway, C.C., McIntosh, B.J., Leow, K.X., Schwartz, M.S. and Pavelchek, C., 2022. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nature biotechnology, 40(4), pp.555-565.\n4. Bray, M.A., Gustafsdottir, S.M., Rohban, M.H., Singh, S., Ljosa, V., Sokolnicki, K.L., Bittker, J.A., Bodycombe, N.E., Dančík, V., Hasaka, T.P. and Hon, C.S., 2017. A dataset of images and morphological profiles of 30 000 small-molecule treatments using the Cell Painting assay. Gigascience, 6(12), p.giw014.\n5. Cross-Zamirski, J.O., Mouchet, E., Williams, G., Schönlieb, C.B., Turkki, R. and Wang, Y., 2022. Label-free prediction of cell painting from brightfield images. Scientific reports, 12(1), p.10001.\n6. Moshkov, N., Bornholdt, M., Benoit, S., Smith, M., McQuin, C., Goodman, A., Senft, R.A., Han, Y., Babadi, M., Horvath, P. and Cimini, B.A., 2022. Learning representations for image-based profiling of perturbations. Biorxiv, pp.2022-08.\n7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. and Uszkoreit, J., 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R717cWAjBR", "reviewer": "Reviewer_EcAm", "summary": "The paper presents an extension of the Vision Transformer (ViT) architecture for image analysis domains with multi-channel information. This includes microscopy (fluorescence), and satellite (spectral) imaging, among others. The method consists of decoupling channels from the input and passing them as different tokens. To this end, the authors designed a learnable channel embedding and a hierarchical sampling method. As a result, a ChannelViT model learns to associate tokens in the space and channel dimensions, and is robust to channel drops.", "questions": "* Can the authors report additional baselines and compare to other state-of-the-art methods in the same datasets that they evaluate? This could clarify the achievements of the method in context to prior art in these domains.\n* Can the authors add computational analysis and comparisons to other methods for training / testing time of the methods? Even if the cost is a limitation, reporting and evaluating this aspect transparently can enhance the understanding of how useful this method can be in practice.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Well motivated paper. The problem of processing multi-channel images is important and the paper presents arguments to design methods that deal with this information effectively.\n* The method is simple and effective. Splitting images in a sequence of tokens is an intuitive approach and the results show its effectiveness.\n* Beyond tokenizing, an important aspect of the method is sampling channels strategically to learn their associations.\n* The analysis in various datasets, including ImageNet, JUMP-CP and So2Sat, adds to the evidence that the method works in practice.\n* Various ablations and extensive details presented in the manuscript and the appendix.\n* The evaluation includes quantitative and qualitative results that show the benefits of the proposed approach.", "weaknesses": "Major comments:\n* Limited baselines reported. The paper only considers ViTs as a baseline, which is a natural comparison given that the proposed method is an extension of this architecture. However, comparison to other architectures, especially CNNs to solve the multi-channel problems should be reported. This is specially important to appreciate the relative performance compared to other solutions tested before in multi-channel images.\n* Computational cost increases quadratically with the number of channels. This problem is not addressed or commented in the main manuscript. According to the reported times in the appendix, the training cost increases linearly with the number of channels. The computational cost should be analyzed and discussed in more detail. While the solution seems effective, it can be prohibitive in practice.\n* In a similar note as the first point, other adaptations of ViTs have been reported for video analysis and 3D imaging, which also increase the image dimensions in a new axis (different than channels). Do any of these existing adaptations apply for channels?\n* The prediction of treatment in the JUMP-CP dataset is not necessarily a biologically relevant task. Prior literature makes the distinction that this is a pretext task for learning representations (weakly supervised learning), and not the main goal of analyzing these images. This clarification should be mentioned and prior work should be cited. Ideally, a biologically relevant task should be reported to ensure that classification performance in this pretext task is not dominated by confounders or spurious correlations. If this is not possible, an alternative test or explanation should be provided. \n\nOther comments:\n* The results in Table 3 are based on a random selection of images with different numbers of channels (25%, etc). The reviewer assumes that the random partition is the same across experiments. The results should be repeated a few times with different partitions and standard deviations should be reported.\n\nMinor comments:\n* Figure 5 reports KRAS in the top row and KCNH76 in the bottom row, but the main text refers to them in the opposite order.\n* Does figure 6 include error bars? Can you clarify how these were obtained?\n* Minor typos: demonstrats, utiliz, involvs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c7LtUWZpRF", "reviewer": "Reviewer_rJx2", "summary": "The authors propose a new tokenization scheme for Vision Transformer (ViT), where an input image is split into patches not only in spatial dimensions but also in channel dimensions. This may be beneficial for data where channels carry different semantical information, thus having little correlation, and might not always be present altogether in the data. Additionally to the tokenization scheme, the authors propose Hierarchical Channel Sampling (HCS) to regularize training, adapting the model to imperfect inputs.", "questions": "1. I suggest revisiting the title. The inspiration for the title used in this paper has clearly come from the ViT’s “An image is worth 16x16 words”. But in the case of ViT, it was about patches of size 16x16 across all channels, i.e., \"Cx16x16\", while the proposed approach uses \"1x16x16\" \"words\", as the image is split into channels as well. \n2. Please clarify why the proposed channel sampling is “hierarchical.”\n3. It would be beneficial to review the notation used in the formulae, e.g. use AB instead of A\\cdotB for matrix multiplication.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The proposed method outperforms the original ViT model in almost all considered experiments.\n* The proposed regularization training scheme proves its efficiency against input channel dropout even for the original ViT.\n* It is claimed that it is possible to increase explainability through attention map analysis with the new scheme. \n* The experimental base behind the work is diverse and extensive. Most authors’ statements are supported by an experiment or an ablation in the main section or the Appendix.\n* The paper is easy to read and understand (illustrations, tables, etc.), and well-organized.", "weaknesses": "* The novelty of the work is limited, as it seems that the channel splitting proposed in this paper is similar to a multi-modal approach to multi-channel data that has been explored earlier (albeit not with images). Also, HCS can be viewed as a modification of input channel dropout that has also been known before.\n* The gains from the proposed approach are limited to the data with many channels (e.g., some microscopy images) and seem less reasonable for other types. At the same time, the processing with this new approach takes significantly more time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)