LUM-ViT: Learnable Under-sampling Mask Vision Transformer for Bandwidth Limited Optical Signal Acquisition
Authors: Lingfeng Liu
Dong Ni
Hangjie Yuan
Venue: ICLR 2024 poster (2024)
Keywords: hyperspectral imaging
optical modulation
real-time detection
vision transformer
pre-acquisition modulation
learnable mask
weight binarization
Primary area: general machine learning (i.e., none of the above)

Abstract: Bandwidth constraints during signal acquisition frequently impede real-time detection applications. Hyperspectral data is a notable example, whose vast volume compromises real-time hyperspectral detection. To tackle this hurdle, we introduce a novel approach leveraging pre-acquisition modulation to reduce the acquisition volume. This modulation process is governed by a deep learning model, utilizing prior information. Central to our approach is LUM-ViT, a Vision Transformer variant. Uniquely, LUM-ViT incorporates a learnable under-sampling mask tailored for pre-acquisition modulation. To further optimize for optical calculations, we propose a kernel-level weight binarization technique and a three-stage fine-tuning strategy. Our evaluations reveal that, by sampling a mere 10\% of the original image pixels, LUM-ViT maintains the accuracy loss within 1.8\% on the ImageNet classification task. The method sustains near-original accuracy when implemented on real-world optical hardware, demonstrating its practicality. Code will be available at [https://github.com/MaxLLF/LUM-ViT](https://github.com/MaxLLF/LUM-ViT).

Published as a conference paper at ICLR 2024
LUM-VIT: LEARNABLE UNDER-SAMPLING MASK VISION TRANSFORMER FOR BANDWIDTH LIMITED OPTICAL SIGNAL ACQUISITION
Lingfeng Liu', Dong Ni Hangjie Yuan
!College of Information Science & Electronic Engineering, Zhejiang University "College of Control Science and Engineering, Zhejiang University (llfeng, dni, hj. yuan 3 ju. edu. cn
ABSTRACT
Bandwidth constraints during signal acquisition frequently impede real-time de- tection applications. Hyperspectral data is a notable example, whose vast volume compromises real-time hyperspectral detection. To tackle this hurdle, we intro- duce a novel approach leveraging pre-acquisition modulation to reduce the acqui- sition volume. This modulation process is governed by deep learning model, utilizing prior information. Central to our approach is LUM-ViT, Vision Trans- former variant. Uniquely, LUM- ViT incorporates learnable under-sampling mask tailored for pre-acquisition modulation. To further optimize for optical cal- culations, we propose a kernel-level weight binarization technique and three- stage fine-tuning strategy. Our evaluations reveal that, by sampling mere 10% of the original image pixels, LUM-ViT maintains the accuracy loss within 1.8% the ImageNet classification task. The method sustains near-original accuracy when implemented on worldworld optical hardware, demonstrating its practicality. Code will be available at https: /gi thub. com MMaLLL LLMM-VI.
INTRODUCTION
In today's data-driven world, efficient data acquisition and processing are pivotal across numerous domains. Despite breakthroughs in sensing techniques, bandwidth constraints remain a persistent hurdle in applications such as hyperspectral detection (Wang & Xiao, 2020), remote sensing (Al- tamimi & Ben Youssef, 2022), biological optical imaging (Park et al., 2021), hiihe-iigh-eefinttion imaging (Zhou et al., 2020), and high-demand detection (Mehmood & Anees, 2020), to name few.
Hyperspectral imaging (HSI) (Ronan et al., 2022) pivotal example of capturing 2D images across vast electromagnetic spectrum range with acute spectral resolution. Within the canoni- cal sqguissionntoongeettannniiiioooo prooosssiiiiin sequence, the prodigious volume of HSI data intro- duces substantial challenges, notably concerning bandwidth capacities (Fowler, 2014). Upon transi- tioning to the processing phase, traditional methodologies, typically tailored for low-intensity data, grapple with the complexities inherent in handling thousand-channel hyperspectral information. As a remedy, these techniques frequently turn to feature extraction to reduce dimensionality (Rasti et al., 2020). Notably, unlike structured high-level data such as language, image data data-being more alementary-harborss greater degree of redundancy. This notion is echoed by He et al. (2021), suggesting that mere subset of input tokens adequately reconstruct the corrupted view.
Despite its earlier introduction, Compressive Sensing (CS) theory (Donoho, 2006) remains a valu- able reference for pre-acquisition modulation. The central tenet of CS is its ability to reconstruct original data from under-samp representations, assuming the data is inherently sparse. Applica- tions like single-pixel imaging (Suo et al., 2016), rooted the CS principle, exhibit potentials across various imaging domains using digital micromirror device (DMD) (Zhang et al., 2021). However, this method is effective only with relatively high ander-sampling rates. Indeed, the reconstructed images become unrecognizable to humans when the under-sampling rate falls below 30%. A piv- otal shortfall of the CS approach as a signal processing technique is its incapacity to leverage prior
"oorreponndngg Author

---

Published as a conference paper at ICLR 2024
DMD data acquisition system
Npectnmanto
Lens 2
Lens
DMD
Kernel- -evel Bimary Conv
Learnable
Gumbel- Softmax
Parameters
Channel Mask
Layer Norm
Positional
encoding
Learnable
Patch Embedding
Under-sampling
VIT Blocks
Mask
Imange
Dataflow specifie to the Training Phase
Dataflow specifie
Light
the Real-World Application Phase
Figure 1: The comprehensive structure of LUM-ViT. The method unfolds in two stages: the electronic-only training and infernce phase, referred to as the Training Phase, with dataflow depicted in green, and the DMD- involved inference phase, referred the Real-World Application Phase, with dataflow depicted blue. The illustration shows how the patch modulated by DMD using the first kernel, a process replicable all kernels and patches, though not depicted. The learnable undor-sampling mask determines their selection.
knowledge on the detection subject. anticipated that methods capable of harnessing this prior information could surpass this traditional compressive sensing method.
Deep learning (DL), a burgeoning branch of machine learning, emerges as a compelling alternative prior knowledge when given abundant training data. Deep learning methods like Convolutional Neural Networks (CNN) (Simonyan & Zisserman, 2014), Vision Transformers (ViT) (Yuan al., 2021), and Recurrent Neural Networks (Cho et al., 2014) stand out in feature extraction and processing for multispectral and hyperspectral data (Pang et al., 2021; Zhang et al., 2023; Lee al., 2023; Islam et al., 2023). Hyperspectral-speciff Transformer variants such as MST (Cai et al., 2022a) and MST++ (Cai et al., 2022b) have been designed for efficient hyperspec- tral data processing and achieved exceptional results.
for its capability
Many models largely assume negligible data acquisition costs, focusing on data processing through efficient network architectures (Zhou et al., 2021; Zamir al., 2022) and robust pre-training schemes (Xue et al., 2022). However, in hyperspectral imaging processing, data acquisition is time-consuming, with single image acquisition taking minutes or more versus a few seconds for processing 2080Ti. This disparity highlights the need for deep learning approach integrated with optical hardware. Rather than accelerating data processing, we aim to devise a method for pre-acquisition modulation, reducing data volume overhead from the beginning.
Specifically, we aim to employ DMD to perform calculations the patch-embeddiig layer within the VIT model. Over recent years, DMD has been utilized neural network research, primarily executing the role of Linear layers (Wang et al., 2021; Bueno et al., 2018; Liu al., 2023a). To knowledge, LUM-ViT is at the forefront of adopting DMD for -atch-embedding. A central aspect of our work involves tailoring the manec--mbedding layer align with DMD operations. The inherent function of single DMD operation entails execut-
spatial light modulator in optical
2

---

Published as a conference paper at ICLR 2024
ing spatial binary modulation across all spectral channels, using consistent 2-D display pattern. To accommodate this feature, we have devised a patch-embedding layer comprising a one-channel kernel-level weight binarized convolutional layer.
We obtain patch-embedding outputs through pre-acquisition optical modulation and minor post- acquisition computations, with each output point corresponding to single DMD operation. To achieve under-sampling (reducing the required optical modulation and sampling instances), we em- ploy learnable mask with trainable parameters refined during training to selectively retain essential points for downstream tasks from the patch-embedding outputs. Then, only the retained points are acquired in real-world signal acquisition, while others are bypassed. Note that single inference (i.e., detecting a single target object), the masking strategy must be determined before the target object detected and cannot be adjusted based on yet-to-know information. In neural network pruning tasks (Wimmer et al., 2023) like Dynamic (Rao et al., 2023) and Ada (Meng et al., 2021), learnable masks simplify networks while retaining performance. Unlike these works, LUM- VIT requires mask scheme determination pre-acquisition and generates mask fixed after training. The mask is suited for the validation set and downstream taskuhthrough training on a compatibly distributed training set. Furthermore, lightweighting the backbone is not objective.
Building upon the described design, we developed LUM- ViT, as illustrated in Fig 1. LUM-Vit VIT incorporates a patch- bettc--embedding layer consisting of kernel-eeeel binarized single-channel convolu- tional layer tailored for DMD operations, along with a learnable mask for under-sampling purposes. We also devised three-stage training strategy train LUM-VI ViT effectively.
We assessed LUM-ViT on the ImageNet- Ik classification task. With an under- sampl-g rate of less than 10%, LUM-ViT exhibited a minor accuracy degradation of 1.8%, and with rates less than 2%, the accuracy drop remained under 5.5%. We further subjected LUM-ViT to real-world tests. A 4% performance drop from software-based results due to hardware- induced error, underscored LUM- ViT's real-world feasibility. Finally, experiments with actual hyperspectral data confirm LUM-ViT capability in processing real-world hyperspectral information.
2 RELATED WORK
DMD Signal Acquisition System. The Digital Micromirror Device (DMD) encompasses matrix of microscopic, electronically actuated mirrors. Each micromirror can be individually manipulated, toggling between two states to steer incoming light towards designated directions. As shown in 1 light from the object is directed onto the DMD by Lens 1. The DMD modulates the light spatially, which Lens 2 then concentrates onto a spectrometer. This sensor excels in discerning light intensity across diverse spectra. Utilizing its pre-acquisition light modulation facility, the DMD signal acquisition system has been employed in single-pixel imaging(Suo et al., 2016) under the purview of CS theory. For details of the DMD, please refer to Section D in the Appendix.
Vision Transformers (ViT). Originally developed for natural language processing tasks (Vaswani et al., 2017), transformer architectures have quickly gained traction within the computer version community. Challenging the longstanding dominance of CNNs, these models have shown sig- nificant efficacy across various visual tasks, including image classification, object detection, and semantic segmentation. VIT (Dosovitskiy et al., 2021) is noteworthy as it initiates using a purely transformer-based architecture for image classification. Subsequent developments like DeiT (Tou- vron et al., 2021), MAE (He et al., 2021), and Swin (Liu et al., 2021) have enhanced performance and broadened applicability across visual tasks. A distinct feature of ViT and its derivatives is the patch-embedding layer, which segments an image into fixed-size, non-overlapping patches, typically using a convolutional layer with its stride equal its kernel size.
Transformer Pruning. Pruning primarily aims to streamline models by discarding less crucial ele- ments for performance. In the context of Transformers, pruning typically focuses on patches, heads, or blocks with methods like Adaptive Sparse ViT (Liu et al., 2023b), Heat- (Dong et al., 2023), and Evo-ViT (Xu et al., 2021) exemplifying this approach. Identifying which network components to prune is critical. Models such as Din (Rao et al., 2023) and Ada ViT (Meng et al., 2021) address this by employing learnable masks. They utilize the Gumbel-Softmal technique to morph the intrinsically don-differentiable mask selection issues into differentiable probability scenarios, ensuring the masks remain trainable. However, these methods are unsuitable for under-sampling as under-sampling requires masks on pre-acquisition optical calculations for yet-to-capture objects.
3

---

Published as a conference paper at ICLR 2024
To the senvar
State
DMD display
Incoming light
Spatial light modulation
Sum
Cross section of DMD
(a)
Figure 2: The function of the DMD acquisition system. (a) illustrates the spatial modulation principle of incoming light by the DMD. (b) outlines the modulation process. The multichannel input matrix undergoes binary masking via the DMD, following its display pattern. Lens 2 then optically sums residual pixels.
METHOD
3.1 PRELIMILARIES
LUM-ViT leverages prior information to modulate signals, maintaining performance under low under-sampling con-
ditions. Utilizing prior knowledge involves training the model on compatibly distributed training set. Therefore, the method unfolds in two stages: Initially, we undertake training on electronic computing hardware using the train- ing set, followed by electronic inference for evaluation. We mark this as Training Phase. Subsequently, we oper- ates DMD-involved reference to assess the real-worla per- formance. Before data acquisition, the system oblivious to the target object. During acquisition, the DMD optically modulates the target information prior to capture and then relays it the electronic system for subsequent processing. We mark this as the Real-World Application Phase.
X
W
Kernel-tevel
Binarization
X2
x n
W
S
DMD(x, O)v"
v2
JN
Linear
Y
Softmax
Fig. 2 illustrates the function of the DMD acquisition sys- tem. It's pertinent to note that within the entire system, a single DMD operation serves execute spatial binary mod- ulation across all spectral channels employing a consistent display pattern. We represent single signal acquisition un- der one DMD operation as DMD(.. -). Assuming DMD
π
Cumbei-Softmax
Y Y
D
display pattern as 0 E {0, 1]*x* the process for C₁-- channel 2-D input matrix X E *x* Cn with identical spatial dimensions can be articulated as:
Figure 3: The dataflow in LUM-VIT.
DMD(X, 0) [sum(O 0 3 X.,.)). sum X..2),.... sum 0 O X E.......)R......
(I)
single DMD(..) spatially modulates with the same pattern, producing an intensity value for each channel.
where represents the Hadamard product. Across every spectral channel
The DM operation entirely optical. Given the inherent parallelism of optical computation, the efficiency of (-.....) remains constant and unaffected by the sizes of matrices engaged in the computation, provided they fit within the DMD's physical pixel dimensions. Therefore, efficiency is solely contingent the number of DMD(:,)) operations. In scenarios with fixed-size 2-D input image, enhancing under-sampling performance can achieved by ensuring that each DMD(.,)) operation covers many pixels possible, striving to trade the computational load in single operation for the quantity of operations.
overall
A straightforwwad approach might involve executing the DMD operation across the whole image, essentially utilizing a Linear layer. However, for standard images sized 224x224, this layer becomes overparameterized and inefficient. A more efficient strategy entails partitioning the input image into siual, non-overs non-overlapping patches, then applying the DMD(-. operation to each patch. The patch-embedding technique aligns well with the requirements of this strategy.

---

Published as a conference paper at ICLR 2024
Since the DMD only perform 2-D spatial optical modulation, it necessitates single-channel con- volution for the patch-embedding layer. Specifically, for a given image E " xCh, , the patch- embedding layer segments it into N non-overlapping sub sub-images, denoted as X = 22..... 2 N where each a: E R xxxx, Here, K is the patch size, and = (H//K) K) x (W// K') denotes the expected to be of identical spatial dimensions, i.e., 0 E {0, 1] To assimilate data across all spectral channels and en- sure adaptive interaction of each convolutional kernel with various spectral channels, introduce a learnable vector v E RC for every convolutional kernel.
total number of patches. The DMD patterns used for computation
X
This convolutional approach, entailing single-chamne convolutions followed by aggregating multi- channel outputs, bears resemblance to depthwise separable convolution, albeit with distinctions, In pursuit of enhanced efficiency, we opt to leverage the DMD to execute convolution across multiple spectral channels employing singular-chane convolutional kernel in a single DMD(.,) operation.
Given the patch-embedding has C' convolutional kernels, we denote the sequence of these kernels as + 10,, 02. ..., Ocl. Similarly, the spectral weight vectors corresponding to these kernels are represented by V [1i, v2...., Uc]. Hence, for an input image X, the output Y from the patch- embedding layer can be expressed as:
y: [DMD( 01 )of DMD(x;; 0,)02 DMD(a, Oc) E ERC iE (1,2...N),
Y [31, V/2 yN NNC.
In this computation, Y is obtained by N x C' instances of the DMD(.:)) operation. Given the con- stant efficiency inherent to the DMD(..)) operation, apparently, the number of DMD(..-) operations to compute Y solely influence the under-sampling rate. Thus, it is the optimization target.
3.2 LEARNABLE UNDER-SAMPLING MASK
To achieve under-sampling, we employ a binary decision mask with trainable parameters. In the Training Phase, these parameters directly determine the values of the mask, thereby affecting the loss function values of two metrics: the accuracy of downstream tasks and the under-sampling rate. Consequently, these parameters updated during backpropagation, ultimately yielding mask that meets the undersampling rate criteria and is optimally adapted to the downstream tasks.
Specifically, The binary decision mask D {0, 1) 1J determines whether retain or bypass each patch for specific convolutional kernel. If the element at the i-th row and j-th column of the matrix D then the DMD will bypass 2; optical modulation when using convolutional kernel 0,. D sparsifies Y, effectuating mnder-sampling in optical computations:
D.
(3)
The count of points passing through the mask corresponds to the number of DMD( (-.-) operations required get Y, equating to the summation of D. We define the passing-three throough ratio dops. where daps = sum (N x C)..
The trainable parameters z E NxCx2 determines the value of D. undergoes Linear layer followed by Softmax operation to to assign probabilities the elements of D:
Softmax(Linear(z)) NXCx2
(4)
where Ti.j. denotes the probability of bypassing the i-th patch for the j-th convolution kernel, and Ti.j.1 represents the probability of retaining it. Thus, D can be sampled the probabilities provided by π. We deem the Linear layer here crucial since removing it led to a performance decline.
However, sampling from is non-differentiable, which hinders the end-to-end training. To address this problem, we employed the Gumbel-Softmax technique (Jang et al., 2017) to sample from m: E 10
D
(5)
where the Gumbel-Softmal operation yields two-element one-hot vector for each position in D: one-hot vector [0. 1] indicates value 1 and [1.0] value 0. The probabilistic expectation of [o, 1] equals π. Through the Gumbel-Softtax technique, sampling from π transforms into mathematical operation involving æ and values sampled from a uniform distribution. This alteration makes the sampling process differentiable to æ, thus enabling end-to-end training.
5

---

Published as a conference paper at ICLR 2024
During the network's finetuning process, is trained concurrently, and the under-sampling rate is integrated into the training objective using the Mean Squared Error (MSE) loss. Denoting the target mask ratio as dear. we have:
B
ratio
Σ (dear - dope)"
(6)
B
6
where B denotes the batch size. The total training loss merges the mask loss and the classification loss as follows:
L Lels + AratioCration
where Lete denotes the classification loss. We set Aratio 5 in experiments.
3.3 KERNEL-LEVEL WEIGHT BINARIZATION
The DMD exclusively performs spatial binary modulation, necessitating a single-channel binary pattern 0 in the patch-embedding layer. To ensure DMD compatibility, the patch-embedding layer's weights warrant binarization. Note that the binarization here is intended to adapt to DMD optical computation, not for network lightweighting.
Binarization in deep learning, such as in Binary Neural Networks (BNN)Qin et al., 2020), sim- plifies models for reduced inference requirements. Typically, this involves binarizing weights and sometimes dataflow. Notable efforts in Transformer binarization include BiT(Liu et al., 2022), Bi- ViT (Xiao al., 2023) and BiSCI (Cai et al., 2023).
Due to the substantial performance degradation caused by full-layer binarization, a more refined strategy is required. As our binarization objective straints, it's adequate to ensure the pattern used in a single DMD(.,-) operation is binarized, i.e., E {0, s (s Specifically, the parameter matrix that determines the values of the convolutional kernels in the patch-embedding layer represented as W E RxxxC, with its binarized outcome being W E {0, 1 The binarization process is realized through the step(-) function:
conform to the DMD operational con-
elaborated upon in the following). instead of binarizing entirely.
wh = stop (w) - -
1, w 2 0,
(8)
0, W <
where w E W and wh E W Let w; E R be the parameter matrix determining each convolutional kernel in W, for i E 2 C), and let wo (0,1) be its corresponding binary matrix, we compute a weight coefficient 8 E R for each convolutional kernel:
K
1
8
K
max(O, (w:)))), E 21,
j=1
Thus, a single binarized convolutional kernel 0 can be computed by:
(10) Despite optimizations, binarization introduces certain training challenges. Specifically, LUM-ViT, being binarized-full precision hybrid model, necessitates distinct training configurations for each precision type. Furthermore, as the binary layer forms the network's foundation, alterations herein significantly affect subsequent layers, especially the mask training. Attempts to concurrently train the binary layer and the mask have thus been unsuccessful. To address this, we employed a three- stage fine-tuning strategy. Specifically, we utilize the MAE (He al., 2021) base model as the pre-trained model and train LUM-ViT for the following three stages:
0, s {1, 2.....C) C}.
Stage 1: Initially, trained the mask without binarization, using w in place of 0, em- ploying standard AdamW (Loshchilov & Hutter, 2019) optimizer settings and regular data augmentations. The mask weights were then locked to ensure stability later stages.
Stage 2: Binarization was introduced and trained, deriving O from w. We utilized a more sensitive AdamW setting, minimizing data augmentations for this phase.
Stage 3: With the mask and binarization layer frozen, we fine-tuned subsequent layers, reverting to conventional Adam W settings and standard data augmentation strategies.
This three-stage training scheme yields favorable performance. For more detailed training settings, please refer to Section E of the Appendix.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
In this section, we demonstrate LUM-ViT's performance. section 4.1, we compare LUM-ViT against baseline methods and perform ablation studies on its design choices the ImageNet- Ik dataset. section 4.2, real-world experiments on the DMD signal acquisition system are conducted demonstrate LUM-ViT's real-world feasibility. In section 4.3, we apply LUM-ViT three widely used hyperspectral image classification datasets, validating its utility for hyperspectral data.
4.1
THE TRAINING PHASE EXPERIMENTS
(a) Top-1 Acc before and after bianrization.
due (%)
LUM-VIT Top-1 1 Acc (%)
55.0
MAE
before bi
after bi
50.9
22.3
$0.5
20
82.7
82.5
10
82.3
80.0
79.8
81.9
15.9-
16.2
LUM
DU-VIT
78.9
78.3
ImageNNet Top-1 Ave
Mag-mosk-Vi VIT
70.9
Rardfam-mssk-W
65.4
65.0
7.5
16
50-
14
25-
method
0.7
(b) Mean keep probabilities of each patch position in -
Under-sumpling Rate (%)
(c) Comparison with basic methods.
Figure 4: The main results in the Training Phase of LUM-ViT, providing the Top-1 Acc results on the ImageNet-IK Ik classification task. Comparisons with the basic methods reveals the reliability of the learnable snder-sampling mask strategy, especially extremely low under-sampling rates (2%-5%). The dark red line marks the baseline upper bound.
Dataset. Our experiments are conducted on ImageNet-Ik (Deng et al., 2009), reporting the Top-1 classification accuracy. We refrained from utilizing hyperspectral datasets for their lack of compa- rability in comprehensiveness and richness to ImageNet. Additionally, the scarcity of display devices for rendering hyperspectral images presents a major challenge for conducting prac- tical experiments with such datasets.
Pre-trained Model. Our three-stage training begins with MAF-base the pre-trained model. Its performance, with an accuracy 83.7%, serves as our baseline upper bound. Following the MAE- base configuration, we set the patch size K = 16 and the number of kernels C 768 in LUM-VIT.
Performance and Training Details. We trained LUM- ViT models separately for under-sampling rates of 20%, 10%. 5%, and 2%. Fig. 4 (a) shows their performance after Stage I and Stage 3, showcasing the learnable mask's 's impact and influence of binarization. Our training, detailed in Section 3.3, consists of three stages with 50, 20, and 80 epochs, respectively, including warm-up for 1/10 of the total epochs per stage. In stages I and 3, we used Adam optimizer with momenta 0.9 and 0.999 while adjusting 0.6 and 0.9999 in stage 2. Common augmentations were used in stages and 3, omitting cutmix (Zhong et al., 2020) and mixup (Zhang et al., 2018) in stage 2.
Methods for Comparison. Finding comparable analogous DL methods for benchmarking LUM- ViT is challenging as it's a pioneer in its field. CS methods, unsuitable for benchmarks, struggle with very low under-sampling rates. Given these cireumstances, called DU-ViT that attempts directly undersampling by decreasing the number of convolutional kernels, serving as a comparative baseline. To further validate the effectiveness of the learnable mask strategy, we also experimented with random masking (Random- Baroddommmms--iii VIT) and magnitude- based masking (Mag-mask- ViT), both of which showed a significant gap compared to the learnable mask strategy. Detailed descriptions of the comparative methods can be found in Section B of the Appendix.
compromise, we introduce method
7

---

Published as a conference paper at ICLR 2024
Performance Analysis. Fig. 4 (c) depicts LUM-ViT surpassing DU-VIT at all under- sampling rates, notably maintaining accuracy loss below 5.5% at extremely low rates (2%-
Impelee Top- Are (%)
5%). This reflects that the original image holds more information than needed for classifica- tion, with the learnable mask adept isolat- ing vital information for the task. Fig. 4 (b) illustrates the average pass-through probabil- ity across all patch positions for LUM-VVi-2% LiT with 2% mask). No global center- edge pass-through rate disparity was observed, indicating that a fixed mask solely focusing on the image center may be inadequate for this dataset. For further analysis of the mask, please refer Section C of the Appendix.
1.30
2000-777
2 8/40
Dreank-iDDaes
Patch Mask Rato (%)
Figure 5: Comparison of mask patch ratios. This is not performance comparison for network lightweight- ing, but to prove the effectiveness of the learnable mask of LUM-ViT.
Mask Analysis. By pruning patch information to 2% through mask before the backbone, LUM-VIT accomplishes classification tasks, aligning with the notion that the inputs excess information for classification. Fig. 5 contrasts LUM-ViT with other mask patch methods on VIT. indirectly substantiating the effectiveness of the learnable mask. Notably, this is not performance comparison. Indeed, LUM-ViT and these methods are not directly comparable performance, given that LUM-ViT does not aim lightweight the backbone.
Ablation Study. Four scenarios were explored in ab- lation studies on LUM- ViM-Vi--0% (LUM-Vit ViT with 10% mask) to validate our methodology. Each setup under- went thorough training, reporting Top-1 accuracy pre and post-binarization (post Stage-1 and Stage-3 train- ing). Here, full-layer indicates binarizing at layer- level rather than kernel-level. W/o learnable token denotes zero assignment to masked points instead of learnalle token substitution. L1-norm involves using L1-norm for loss calculation during mask training, re- placing MSE. No Linear signifies excluding the Linear layer following the masking parameters.
Table 1: Results of the ablation study.
Top-1 Acc (%)
Config
before bi
after bi
LUM-ViT
82.3
81.9
full-layer bi
82.3
81.3(-0.6)
w/o learnable token
82.1(-0.2)
81.7(-0.2)
L1-norm no Linear
80.6(-1.7)
80.0(-1.9)
79.7(-2.6)
7..)
4.2
THE REAL- WORLD APPLICATION PHASE EXPERIMENTS
while
4000
yellow
Spectrometer Less
bise groen urange
bateelty
3000
3000
0000
Light Source
Wineelongth (mm)
Display
(a)Spectrum of cach color in the e-ink display.
(b)Rcal-worff DMD signal acquisition system.
Figure 6: Real-world DMD signal acquisition system and spectrum of each color in the e-ink display. We established sctup in line with the standard DMD signal acquisition system. To ensure spectral integrity, images are displayed on c-ink screen with stable LED light source employed for illumination.
Hardware Setup. Our configuration comprises DMD from Texas Instruments (resolution: 1024*768 pixels), Ocean Optic Spectrometer (spectral range: 177.1-886.6 nm across 2024
8

---

Published as a conference paper at ICLR 2024
channels), and stable LED for illumination. For object display, given the need for natural spectra display and adequate light projection onto the DMD, we forwent a conventional monitor in favor of 7-color e-ink display, which reflects natural spectra under robust light source illumination.
Model Adjustment. The LUM-ViT ViT model trained in the Training Phase cannot directly deploy on real-world hardware. Images from ImageNet comprise merely three color channels, i.e., CA. 3, whereas the high-spectral data encompasses 2048 color channels, i.e., Ch = 2048. To address this discrepancy, we reconfigured the spectral channel weighting parameter V (based on the spectrum information illustrated in Fig. 6 (a)) to establish a new color correspondence and conducted fine- tuning on a small sample set.
Real-world Experiment Results. We conducted a real-world application experiment on LUM-ViT- 2%. Our basic hardware setup, including 7-color e-ink display, led significant accuracy loss. We argue that this loss, being evaluation. Thus, we color-redueed ImageNet images to 7 colors in software simulation, reporting the accuracy loss across different models. Certain samples, initially classified by MAE, failed post- color-reduction and were marked as device-induced anomalous samples.
should be isolated from our method's performance
In the real-world application experiment, we processed 371 random images from the ImageNet- 1 test set using LUM-VIT, with DMD implementation in computation. A total of 237 images were correctly classified, yield- ing a 64.4% accuracy. Among the misclassified images, 51 were device-induced anomalous samples. Exclud- ing these, the accuracy improves to 74.7%. Therefore, the LUM-ViT model with 2% masking demonstrates practical feasibility in a real-world scenario with DMD involvement, keeping the accuracy loss within 4% for RGB images and within 5% for 7-color images. This showcases its real-world feasibility.
Table 2: Results of the real-world experi- ments on LUM-ViT-2% %.. Accuracy degrada- tion induced by 7-color conversion is reported.
Top-1 1 Acc (%)
Model
RGB
7-color
MAE-base
83.7
75.7
LUM-ViT-220
78.3
69.1
LUM-VIT-255reel-wword
74.7
64.4
4.3 THE HYPERSPECTRAL IMAGE CLASSIFICATION EXPERIMENTS
To ascertain whether LUM-VIT can handle real-world hyperspectral data with richer spectral
information, we evaluated its performance on three re- mote sensing datasets widely used in hyperspectral im- age classification tasks: Indian Pines (Baumgardner al., 2015), Pavia University, and Salinas. As indi- cated in Table A, LUM-ViT achieved commendable re- sults (Performance is measured by the proportion of correctly classified samples (Overall Accuracy)). No- tably, this experiment aimed to verify LUM-ViT's effi- cacy in handling datasets with abundant spectral chan- nel information, complementing the ImageNet- Ik dataset classification task. mental design, please refer to Section A of the Appendix.
Table 3: Results HSI classification.
Overall Accuracy (%)
Dataset
before mask
after mask
Indian Pines
87.4
86.2
Pavia University Salinas
89.5
88.6
99.4
99.1
For detailed experi-
CONCLUSION
In this work, we innovatively applied deep learning to under- sample hyperspectral data acquisi- tion, achieving significant data reduction from signal collection to processing, Utilizing ViT as the backbone network and a DMD signal acquisition system for patch-embedding, we enabled pre- acquisition optical modulation. With learnable mask, we identified and retained embedded points crucial for downstream tasks while bypassing less significant embedded points, achieving under- sampling. On the ImageNet-Ik classification task, we maintained accuracy loss within 1.8% at 10% under-sampling and within 5.5% at an extreme 2% under-sampling. Real-world experiments showed that the accuracy loss of LUM-ViT did not exceed 4% compared to the software environ- ment, demonstrating its practical feasibility. Future research could explore dynamic mask strategies and extend this framework to other tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported by National Natural Science Foundation of China under Grant 62173298. (Corresponding author: Dong Ni.)
REFERENCES
Amal Altamimi and Belgacem Ben Youssef. A systematic review of hardware-accelleatte com- pression of remotely sensed hyperspectral images. Sensors, 22(1), 2022. ISSN 1424-8220. doi: 10.3390//22010263.
M. F. Baumgardner, L. Biehl, and D. A. Landgrebe. 220 band AVIRIS hyperspectral image data set: June 12, 1992 Indian Pines test site 3, 2015.
Bueno, S. Maktoobi, L. Frochly, I. Fischer, M. Jacquot, L. Larger, and D. Brunner. Reinforcement learning in large-scale photonic recurrent neural network. Optica, 5(6):756, jun 2018. doi: 10.1364/optica.50000 5.00755..
Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang. Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool. Mask-guided spectral-wise transformer for efficient hyperspectral image recon- struction. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17481-17490, 2022a. doi: 00.1009/CVPS222882222..0000
Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, Radu Timofte, and Lue Van Gool. Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction, 2022b.
Yuanhao Cai, Yuxin Zheng, Jing Lin, Xin Yuan, Yulun Zhang, and Haoqian Wang. spectral compressive imaging, 2023.
Binarized
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using mm pecoder-decoder for statistical machine translation. arXiv preprint arXiv: 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-244222 248-255, 2009.
doi: 10. 109/CVPR.2009 5200948.
Peiyan Dong, Mengshu Sun, Alec Lu, Yanyue Xie, Kenneth Liu, Zhenglun Kong, Xin Meng, Zhen- gang Li, Xue Lin, Zhenman Fang, and Yanzhi Wang. Heatvit: Hardware-efficien adaptive token pruning for vision transformers. In 2023 IEEE International Symposium on High-Performunce Computer Architecture (HPCA), pp. 442-455, 2023. doi: 10. 11110//PAAA5555622210000000 10071044
D.L. Donoho. Compressed sensing. IEEE Transactions Information Theory, 1284):128- 2006. doi: 10. 10./100/77.200..88777 871502...
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale, 2021. 3
James E. Fowler. Compressive pushbroom and whiskbroom sensing for hyperspectral remote- sensing imaging. In 2014 IEEE International Conference on Image Processing (ICIP), pp. 684 688, 2014. doi: 10.1 10.1109//CIP2200 109/ICIP2201..72255533
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021. 3,
Md Touhid Islam, Md Rashedul Islam, Md Palash Uddin, and Anwaar Ulhaq. A deep learning-bases based hyperspectral object classification approach via imbalanced training samples handling. Remote Sensing, 15(14), 2023. ISSN 2072-4292. doi: 10.3390/ss 2
10

---

Published as a conference paper at ICLR 2024
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax, 2017.
Chin Poo Lee, Kian Ming Lim, Yu Xuan Song, and Ali Alqahtani. Plant-cnn-vit: Plant classification with ensemble of convolutional neural networks and vision transformer. Plants, 12(14), 2023. ISSN 2223-7747. doi: 10.3390/plants 00.33900plamts12214225. 1214p22.. 2
Lingfeng Liu, Dong Ni, and Liankui Dai. Spatial anomaly detection in hyperspectral imaging using optical neural networks. IEEE Intelligent Systems, 38(():64 72, 2023a. doi: 10.1 20993...2023. 3241431.
Xiangcheng Liu, Tianyi Wu, and Guodong Guo. Adaptive sparse vit: Towards learnable adaptive token pruning by fully exploiting self-attention, 2023b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. 3
Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoor- thi, and Yashar Mehdad. Bit: Robustly binarized multi-distilled transformer, 2022. 6
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 6
Erum Mehmood and Tayyaba Anees. Challenges and solutions for processing real-time big data stream: A systematic literature review. IEEE Access, 8: 8:119123-119143, 2020. doi: 109/
SCCCESS.202.30052660 1
Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recognition, 2021.
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM Comput. Surv., 54(2), mar 2021. ISSN 0360-0300. doi:
10.1145/3439950. 2
Park, D.J. Brady, G. Zheng, L. Tian, and L. Gao. Review of bio-optical imaging systems with high bandw-bandwidth product. Adv Photonics, 3(4):044001, 2021. doi: 10.1117/1.aa.3...444000.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nieu Sebe. Binary neural networks: A survey. Pattern Recognition, 105:107281, 2020. ISSN 0031-3203. doi:
https://doi.org/00001166. pttp:///rrr.0.1....aaaooo0000000 200.cp 107281. 6
Yongming Rao, Zuyan Liu, Wenliang Zhao, Jie Zhou, and Jiwen Lu. Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks, 2023.
Behnood Rasti, Danfeng Hong, Renlong Hang. Pedram Ghamisi, Xudong Kang, Jocelyn Chanussot, and Jon Atli Benediktsson. Feature extraction for hyperspectral imagery: The evolution from shallow deep: Overview and toolbox. IEEE Geoseience and Remote Sensing Magazine, 8(4): 60-88, 2020. doi: 10. WW09/MGRS.2020.9999999
Philip Ronan, C. Günther, and Frank Frichl. Hyperspectral Imaging and Applications. MDPI, 2022. doi: (0.3399/bookk9788330399------3
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rX:v 2014. 2
Jinli Suo, Yuwang Wang, Liheng Bian, and Qionghai Dai. Single-pixel hyperspectral imaging. In Qionghai Dai and Tsutomu Shimura (eds.). Optoelectronic Imaging and Multimedia Technology IV. volume 10020, pp. 38 44. International Society for Optics and Photonics, SPIE, 2016. doi: 10.1117/12.2247809. 3.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017. 3

---

Published as a conference paper at ICLR 2024
Tianyu Wang, Shifan Ma, Logan G. Wright, Tatsuhiro Onodera, Brian C. Richard, and P. McMahon. An optical neural network using less than photon per multiplication. Nature Communications, 12(1):1-9, 2021. doi: 10. 10.1038//44466--012-777744
2
Zhongliang Wang and Hua Xiao. Distributed compressed hyperspectral sensing imaging based on spectral unmixing. Sensors, 20(8), 2020. ISSN 1424-8220. doi: 10.3390/s20082055.
P. Wimmer, J. Mehnert, and A.P. Condurache. Dimensionality reduced training by pruning and freezing parts of deep neural network: A survey. Artificial Intelligence Review, 2023. doi: 10. 10.10077/104422-23-1144444 122--999----11
Junrui Xiao, Zhikai Li, Lianwei Yang, and Qingyi Gu. Binaryvit: Towards efficient and accurate binary vision transformers, 2023.
Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision trans- former, 2021. 3
Linting Xue, Aditya Barna, Noah Constant, Rami AI-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards Token-Free Future with Pre-trained Byte-to-Byre Models. Transactions of the Association for Computational Linguistics, 10:291-300,, 03 2022. ISSN 2307-387X. doi: 10.1 1621t6// a 00461.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis E. H. Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021. 2
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Yang. Restormer: Efficient transformer for high-resolution image restoration. In 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5718- 5729, 2022. doi: 10.1009/CVPR$2688.2222 202.000064.....0 2
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez- Pazz mixup: Beyond empiri- cal risk minimization, 2018.
Jiahong Zhang, Honglie Guo, Jin Guo, and Jing Zhang. An information entropy masked vision transformer (iem-vit) model for recognition of tea diseases. Agronomy, 13(4), 2023. ISSN 2073- 4395. doi: 10.3390/agroommy 3304 1344 156.
Yijing Zhang, Phil Surman, and Sailing He. A resolution-enhaacee digital micioooiiooooeviiiiiii (dmd) projection system. IEEE Access, 9:78153-78 164, 2021. doi: 10.1009AACCISSSS22113332222
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. Proceedings of the AAAI Conference on Artificial Intelligence, 34007):13001113000, Apr. 2020. doi: 10.1609/aaii. vaiiiaaiiii34177.. v34i07.7000.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12): 1116-111555, May 2021. doi: 10. naai.v35ij 135.12.17325
Yun Zhou, Xiao Hu, Xiaoqiang Guo, Xiaoyu Li, Xiao Li, and Zhiping Xia. Can based hdr/weg characteristic detection for ultra-high definition video. In 2020 International Wireless Communi- cations and Mobile Computing (IWCMC), pp. 1584-1589, 2020. doi: 10.1 10.11000/CCM4410.. 0020.9148244 1
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A THE HYPERSPECTRAL, IMAGE CLASSIFICATION EXPERIMENTS
A.1 DATASETS
Due to the absence of comprehensive hyperspectral dataset for object classification as extensive as ImageNet in the open-source domain, we opted for the next best alternative: three remote sensing datasets commonly used in hyperspectral image classification tasks Indian Pines, Pavia University, and Salinas. Below is detailed introduction to these three datasets.
Colazcall ES
Conephtty
Civcle
tine
Seaeddedd Etemuntt
(b)
Figure 7: Indian Pines dataset: (a) Grayscale schematic diagram, (b) Ground truth labels (the pixel
count for each sample type is displayed on the legend).
Taeastier Fuieal
H
Eneee 1200
(a)
(b)
Figure 8: Pavia University dataset: (a) Grayscale schematic diagram, (b) Ground truth labels (the pixel count each sample type is displayed on the legend).
Indian Pines (Fig, 7): The AVIRIS sensor captured the Indian Pines scene over a test site in North- western Indiana, which includes a 145x 145 pixel array with 224 spectral reflectance bands spanning the 0.4-2.5 micrometer wavelength range. This dataset a subset of a larger scene, predominantly composed of two-thirds agricultural land and one-third forest other perennial natural vegetation. The area features significant infrastructure, including major highways, railway, low-density hous- ing, other built structures, and minor roads. Captured in June, the image shows early-stage crops like
13

---

Published as a conference paper at ICLR 2024
Fallee much thee
30
100
Boll sirvat dnc 43O0
Auk maco
802
202
400
180
Figure 9: Salinas dataset: (a) Grayscale schematic diagram, (b) Glonnd truth labels (the pixel count for each sample type is displayed on the legend).
and soybeans with less than 5% coverage. The provided ground truth is divided into sixteen classes, which are not entirely distinct. To avoid water absorption features, the number of spectral bands has been reduced to 200 by excluding bands [104-108], [150-163], and 220.
Pavia University (Fig. 8): This scene was captured by the ROSIS sensor during flight campaign over Pavia in Northern Italy. It comprises 103 spectral bands. The Pavia University image 610 by 610 pixels, though some samples in both images are void of information and must be discarded prior to analysis. The geometrie resolution of the image is 1.3 meters. The image ground truths distinguish nine different classes. The figures show the discarded samples wide black strips.
Salinas (Fig. 9): The Salinas scene was collected using the 224-band AVIRIS sensor over Salinas Valley, California, noted for its high spatial resolution with 3.7-meter pixels. The captured area includes 512 lines by 217 samples. Similar to the Indian Pines scene, 20 water absorption bands were omitted, specifically bands: [108-112] 112], [154-167], and 224. This imagery was only available the form at-sensor radiance data. The scene features a variety of elements including vegetables, bare soils, and vineyard fields. The Salinas ground truth encompasses 16 classes.
A.2 TASK DESCRIPTION AND EXPERIMENTAL SETUP
We conducted pixel classification tasks on these three datasets, classifying each pixel based on its own spectral data as well that of its surrounding pixels. Although there no universally accepted standard for dividing training and validation data, we followed the conventional approach of large validation set and small training set (6:4), ensuring no overlap between training and validation data. The dataset partitioned for this task comprises samples, each 9x9 sub-image consisting of the target pixel for classification and surrounding pixels.
To tailor to this task, we adjusted the configuration of LUM-ViT: the number of input channels was matched to the spectral channels of the hyperspectral datasets, the patch size was set to 9, and images input into the network were first enlarged to 27x27, resulting in 9 patches. Additionally, reduced the embedding dimension to 192, while the rest of the settings remained largely consistent with those used for ImageNet.
We conducted training for 100 epochs from scratch on all three datasets, including a 5-epoch warmup. with batch size of 64 and peak learning rate of 0.01, utilizing cosine annealing. We observed some degree overfitting and made fine adjustments accordingly. A large weight decay (0.001) setting could mitigate this issue.
14

---

Published as a conference paper at ICLR 2024
B BASE METHODS FOR COMPARISON
In the experimental section, we compared LUM-VIT with DU-VIT, Random-mask ViT, Mag-mask- VIT and CS-method on the ImageNet- Ik classification task to demonstrate the superior performance and design of LUM-ViT. This section will introduce the experimental setup for these four methods.
B. DU-VIT
DMD data acquisition system
Sensor
Lens
Lens
DMD
Binary Conv
Layer Norm
1x1 Conv
Embeded
patches
Binary Compressed Patch- (mbedding(BCPE))
ViT Blocks
Positional encoding
Figure 10: DU-VIT. We integrated the design of under-sampling and binary convolution within the patch-embedding layer, termed the Binary Compressed Patch- Embedding (BCPE) layer. The dashed lines in the diagram represent data flows exclusive to the training phase.
In early experiments, we considered a straightforward method to reduce the number of convolutional kernels the patch-embedding layer achieve under-sampling. We experimented with this method but observed poor performance. We speculate that the reduction in convolutional kernels led to significant decrease the embedded dimension of the data input to the backbone, and this reduction in data dimensionality severely limited the model's expressive capability. Building this, added x convolutional layer after the reduced-kernn -ernel convolution to restore the embedded dimension to its pre-under-sampe sampling size. We refer to this network as DU-Vit DU-V that is, Directly Under-sampling VIT. This network achieved reasonably good results in our experiments and can serve as a baseline model for under-sampin sampling..
The structure of DU-ViT is shown in Fig. 10. To accommodate DMD operations, this method also employs binary convolutional layers. In experiments, primarily observed that adding
15

---

Published as a conference paper at ICLR 2024
Layer Normalization after the 1 convolution further enhances performance, hence this setup was retained. During the training of DU-ViT, we maintained most of the training settings consistent with LUM-ViT to facilitate comparison, including the three-stage training strategy.
We noted that the performance difference between DU-VIT and LUM-ViT remains relatively stable when the under-sampling rate is above 10%. However, under extreme under-sampling conditions, such as 2%, there is significant gap in performance between DU-VIT and LUM-VIT.
B.2 RANDOM-MASK-V VIT AND MAG-MASK-VIT
We believe that LUM-ViT's ability to maintain performance at low under-sampling rates is not solely due to the adoption masking approach for under-sampling, but also attributed to the success of the learnable mask strategy. To confirm the superiority of the learnable mask approach, conducted experiments with other masking strategies, summarized into the following two schemes:
This approach utilizes random mask strategy. Specifically, we randomly generated a mask and applied it to the output of the patch-embedding layer in the pre-trained model, followed by finetuning with the mask fixed in place.
Random-mask-We VIT:
Mag-mask- VIT: Drawing on common strategies from neural network pruning, we implemented a mask based data magnitude. Specifically, calculated the average magnitude each position of the patch-embedding layer's output from the pre-trained model using the training dataset. We then selected the positions with the highest magnitudes to retain according the required under-sampling ratio, masking the rest. With the mask fixed, we proceeded to finetune the model.
Under the condition that most training settings remained unchanged, we trained and tested the mandom-mask-i ViT and Mag-mask-Vin models. We observed that the performance of these two schemes was inferior DU-ViT and even less so LUM-ViT Similar DU-ViT we also noted more rapid decline in performance at extreme under-sampling rates (2%). Based on the comparison of LUM-VVT with these two models, we conclude that the learnable mask strategy with trainable parameters is superior, adapting well to the target datasets and downstream tasks.
B.3 CS-METHOD
Compressed Sensing (CS) is innovative signal acquisition and reconstruction theory that allows for the capture of signals at a rate significantly below the Nyquist sampling criterion and accurately reconstructs sparse signals or signals that can be sparsely represented from these few samples.
The principle of CS indicates that for a sparse signal, or a signal that can be sparsely represented, possible to capture most of the signal information through diwdemmnsionll measurements and to accurately reconstruct the original signal from these measurements. This process can be described by the following mathematical model:
(11) where x IR" is the original signal, & E R is the measurement matrix, and E R are the measurements with m < n, meaning that the dimension of measurements much less than dimension of the signal.
4x,
The sparsity of signal refers to the number of non-zero elements in a transform domain (like Fourier, wavelet, etc.) being few. If the signal itself is not sparse, it can be transformed using transformation matrix V
VX,
(12)
where s is the sparse representation in the transform domain.
For signal reconstruction, one of the common algorithms is Orthogonal Matching Pursuit (OMP). OMP is an iterative greedy algorithm that seeks to find the set of sparse coefficients that best matches the measurement vector y. The steps of the OMP algorithm are as follows:
Initialize the residual ro = y, support set So = and iteration count 0.
2. Find the column oj that correlates most significantly with the residual r; and update the support set S S (j).
16

---

Published as a conference paper at ICLR 2024
3. Update the signal estimate by solving the least squares problem: X arg minx lyy
4 Sc-1 x||2.
4. Update the residual to rk+1 = S X6+1-
5. If stopping criterion is met (such the residual is small enough predetermined number of iterations is reached), stop the iteration.
6. Otherwise, increment k and go back to step 2.
Based on the aforementioned methods, we conducted tests on the ImageNet-Ik Ik dataset. Specifically, undersampled the original images at different under-sampling rates using Gaussian random mask (this process is adapted for DMD computation), then reconstructed the images using the OMP algorithm, and finally completed the classification task through ViT.
Because the CS- eS-method yields poor reconstruction of the original image at low under- sampligg rates (< 20%), the subsequent ViT model struggles with the classification task, leading very poor final results. The contrast between these results and those of LUM-ViT ViT aligns with our logic: the CS- method cannot effectively utilize the prior information of the object being detected, making un- suitable for low under-sampling rates, whereas deep learning methods can achieve better results by effectively using prior information, maintaining performance even at extremely low under-sampling rates.
C VISULIZATION OF THE LEARNABLE MASK AND ANALYSE
1.3
1.4
23
1.3
1.4
2.
1.8
1.0
8.5
1.4
1.4
1.7
1.2
21
23
2.0
2.1
2.0
0.6
0.5
107
6.7
8.3
10.00
1.4
1.8
8.7
3.2
9.0
8.5
8.5
9.0
85
8.7
2.1
2.0
1.6
1.4
11
2.1
21
1.8
21
2
7.0
2.1
1.8
0.6
0.4
8.3
10
0.6
0.5
0.7
02 30
9.0
9.0
84
50
1.7
1.8
2.1
2.
1.7
2
2
2.0
21
1.4
21
2.1
1.7
3.0
8.5
28.2
8.5
8.5
85
8.
8.5
2.1
1.3
1.4
1.6
1.4
011
2.1
2.1
2.1
2.1
2.1
1.6
8.8
8.5
1.4
2.1
2.1
2.1
2.1
2.1
2.1
8.6
8.8
8.5
9.00
1.4
2.0
1.4
2.1
0.2
9.0
3.5
8.6
8.5
8.6
8.5
8.5
8.5
8.5
8.5
2.1
1.6
21
2.2
1.8
2.1
2.3
1.8
2.0
1.8
1.6
3.7
1.4
8.73
2.3
2.1
0.9
8.3
8.9
10.2
10 10
8.3
84
8.5
05
102
8.3
3.6
1.4
1.4
2
20
1.6
2
2.1
2.1
1.4
2.1
2.1
1.6
8.50
1.2
85
0022
85
85
8.9
85
88
9.0
85
8.25
(a)Heatmap for LUM-ViT-2%% patches.
(b)Hcatmap for LUM-ViT-109- patches.
Figure 11:
The passing-throgh probability heatmap of patches at different positions LUM-VIT. In figure (b), we truncated the top 5% of the data to allow the image to reveal meaningful information and to prevent the distortion caused by extreme outliers.
In LUT, the mask is applied to each data point output by the patch-embedding layer, with each data point corresponding to the output of specific convolutional kernel on specific patch. The mask filters both kernels and patches: if all positions of patches affected by a kernel are masked, then the kernel is eliminated; similarly, if all kernel positions patch are masked, the patch is eliminated.
To explore the selective effect of the learnable mask on different kernel-patch data points, we visu- alized two models: LUM-ViT-1099 and LUM-ViT-2%.
Fig. 11 presents the visualization results for different patch positions, offering an intuitive realiza- tion of the macroscopic emphasis of all kernels on different image locations. Overall, we did not observe a pronounced spatial preference, which might be due to the fact that all patches could con- tain important information across the entire dataset. Comparing the two models, the heatmap of LUM-ViTE 10% appears more random, and we observed four extreme outliers with values exceeding 60%, which, if not truncated, would significantly impact the visualization effect.
17

---

Published as a conference paper at ICLR 2024
30-
50
Numbbr dfhomat
25-
Nou Box hemees
40
-0
15
-20
10
100
Probubility sum
Probabiity Sum
(a)Histogram for LUM-ViT-2% kernels.
(b)Histogram for LUM-ViT-10%E kernels.
Histogram of the number of patches retained by each convolutional kernel in LUM-VIT. The horizontal axis represents the sum of probabilities for all patch positions a single kernel to pass through the mask. Due to the selection and elimination of most convolutional kernels in low under-sampling scenarios, the number of data points at position O in the histogram is overwhelmingly large. To prevent this from impacting the visual effect, we have excluded the data points within the [0.5] range.
Figure 12:
Fig. 12 visualizes the selection probability of different kernels. We calculated the sum of probabili- ties for all patch positions passing through the mask for each kernel and tallied the number of kernels for different total probability sums. Many kernels are directly discarded at low under- sampl-g rates, leading to an overwhelming number of points near To avoid impacting the visualization, we re- moved data within the [0,5] interval. We observed that in LUM-ViT-10% there is tendency to select kernels without selecting patches, meaning the model tends to keep or discard all patches under one kernel. This phenomenon disappears in LUM-ViT-2%; where the model tends to allocate different patch positions to different kernels.
Synthesizing the visual results from Fig. 11 and Fig. 12, we infer that the model tends to select kernels under relatively low under-sampling pressure (e.g., 10%), satisfying under-sampling requirements by retaining only the most useful kernels. When the muder-sampling pressure is relatively high (e.g., 2%), selecting kernels alone does not meet the under-sampling rate requirements, thus forcing the model make selections among different patch positions under different kernels, leading collaborative division of labor across different kernels for different patch positions.
D DETAILS ON THE DMD SIGNAL ACQUISITION SYSTEM
D.1 DETAILED OVERVIEW OF DMD
Digital Micromirror Devices (DMDs) are sophisticated optical semiconductor devices that form the core of Digital Light Processing (DLP) technology. A typical DMD chip hosts an array of up to millions of micromirrors, each with dimensions in the order of few micrometers.
The core functionality of a DMD lies in its ability to modulate light spatially. Each micromirror can be individually tilted by electrostatic forces, typically between +12 to 12 degrees, corresponding to and off states, respectively. This tilt controls whether light is directed towards the projection lens (on) or absorbed by a light trap (off), thereby controlling the brightness of each pixel in the projected image.
The speed at which these mirrors switch affects the image's refresh rate, with current DMDs capable of switching at speeds of thousands of times per second.
DMDs are integral to various applications due to their high spatial resolution and rapid response time. Common uses include high-definition projectors for cinema and business presentations, where their ability to precisely control light ensures high-quality image projection.
In the field of hyperspectral data processing, DMDs enable the selective reflection of different wave- lengths, facilitating the capture of spectral information for each pixel, which critical for applica- tions in remote sensing and material analysis. Furthermore, the potential of DMDs in optical neural networks is being explored, where they can modulate light in complex patterns to mimic neural connections, potentially leading advancements in photonic computing and artificial intelligence.
18

---

Published as a conference paper at ICLR 2024
Light
Lamp
(a)The reflective function of single micromirror on a DMD chip.
(b)DMD evaluation board.
Figure 13: DMD chip and its micromirror function.
D.2 DMD SIGNAL ACQUISITION SYSTEM
In LUM-ViT, the DMD serves key component for optical modulation before signal acquisition, termed the DMD signal acquisition system. Its schematic is shown in Fig. in the main text, and its physical representation in Fig. 6, which are not displayed here again.
The DMD signal acquisition system comprises two sets of focusing lenses, a DMD chip, and single-point light signal detector. During operation, light from the target object imaged onto the DMD's micromirror surface by Lens Group 1. The DMD modulates the image projected onto it by setting micromirrors different reflection states. The modulated image is then reflected towards the optical detector, where Lens Group 2 focuses the light onto the detector, completing signal acquisition. It's worth noting that while the spectrometer here can capture comprehensive spectral information at extremely high spectral resolution, only measures the light intensity single point at a time, rather than capturing a multi-pixel image in one shot like conventional photographic equipment.
During its operation, the DMD can control the state of its micromirror array at each timestep, modulating the optical signal in various ways. Since the DMD performs spatial optical modu- lation- simultnnimuu modulating all spectral channels within its working range with the same mask-it is well-suited for modulating hyperspectral signals.
The DMD is the critical control component of the DMD signal acquisition system, with signal modulation achieved through different masks. In single-pixel cameras based on CS theory, the DMD typically uses random masks following Gaussian distribution. In LUM- VIT. neural network weights obtained through training are used as masks for the DMD, integrating the DMD signal acquisition system into the neural network computation.
E TRAINING SETTINGS OF THE THREE STAGE OF THE TRAINING PHASE
During the Training Phase, we divided the training process into three stages, using distinct training parameters to train the mask, the kernel-level binary layer, and overall finetuning, respectively. In initial experiments, we attempted to combine these stages, but due to the instability introduced by the learnable mask and binarization, training them together led to difficulty in convergence or suboptimal performance. Therefore, we consider this three-stage training approach necessary.
Regarding the first stage of training, conventional training parameters suffice for mask training. The primary consideration is the initial value of the mask's 's trainable parameters. We experimented with an all-ones mask (i.e.. a non-selective mask) as the initial value, but results were inferior to those with randomly initialized mask. During training, the undersampling rate quickly drops the target value within a few epochs and then fluctuates slightly throughout the remaining training process.
The second stage is the most unstable part of the entire training process. Due the significant dif- ference between binary and full precision, and because LUM-ViL ViT requires binarization of the entire first layer (which affects all subsequent layers' computations), many settings from full-precision
19

---

Published as a conference paper at ICLR 2024
network training longer applicable. Specifically, froze the mask weights from the first stage to prevent changes in the mask from affecting the training of the binary layer. We abandoned most data augmentation strategies and adopted more sensitive and stable learning rate momentum parameter setting, commonly used in training binary networks.
The third stage is the final finetuning. Since the second stage's training settings primarily tailored to the binary layer and do not suit the full-precision layers, we need to finetune these layers last with parameters suitable for full-precision training. Specifically, in this stage, froze the weights of the binary layer and the mask, finetuning only the remaining network layers with conventional parameter settings.
Table Training settings.
value
config
Stage Adam 0.9, 0.999
Stage 2
Stage 3
optimizer
AdamW
Adam
optimizer momentum (3,
0.6, 0.9999
0.9, 0.999
base learning rate
5c-4
le-4
lc-4
weight decay batch size
0.03
5e-5
0.03
4096
4096
4096
learning rate schedule
cosine decay
cosine decay
cosine decay
warmup epochs
5
5
frozen layers
none
mask
bi-conv-kereels, mask
label smoothing
0.1
0.1
0.1
random erase
0.25
0.25
mixup
0.8
0.8
cutmix
1.0
1.0
drop path
0.1
0.1
0.1
20


Reviews:
[{"review_id": "88HuKEghJp", "reviewer": "Reviewer_rvtr", "summary": "This paper proposes a learnable under-sampling mask vision Transformer, which incorporates a learnable undersampling mask tailored for pre-acquisition modulation.", "questions": "See the above Weaknesses part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-organized and clearly written.\n\n+ The proposed three-stage training strategy for training LUM-ViT is effective.", "weaknesses": "- Technical details should be clear. How to achieve the learnable under-sampling mask? How is this learnable achieved? Is the learning accurate? Relevant visualization results should be provided.\n\n- The experimental results seem insufficient. The author only conducted validation on the ImageNet-1k classification task, and other tasks should also be further explored.\n\n-----------------------After Rebuttal---------------------------\n\nThank you for your feedback. The rebuttal addressed my concerns well. Considering other reviews, I have decided to increase my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wmebRA4XBY", "reviewer": "Reviewer_vQM3", "summary": "This paper proposes a novel approach using pre-acquisition modulation with a deep learning model called LUM-ViT. Specifically, it utilizes ViT as the backbone network and a DMD signal acquisition system for patch-embedding. Moreover, a kernel-level weight binarization technique and a three-stage fine-tuning strategy is proposed for optimizing the optical calculations. With low sampling rates, LUM-ViT maintains high accuracy on ImageNet dataset.", "questions": "1.\tThe training phase uses images of 3 color channels while the real-world experiment uses 7 color images. What is the meaning of ‘reconfigured’? Or the author just fine tuned the LUM-VIT with 7 color samples? Can the pre trained model be used directly for images in different bands without the need for matching data?\n2.\tThis acquisition system seems to have to work together with vit to obtain intermediate features of images. Can it reconstruct a complete hyperspectral image in a real-world environment?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea is of the paper is interesting. The proposed method performs calculations of the patch-embedding layer instead of directly sampling the whole images. The proposed LUM-ViT is suited for both dataset and downstream tasks.\n2.\tThe accuracy loss is low with extreme under-sampling", "weaknesses": "1.\tThe description of the entire system and methods is not clear and intuitive enough, and the figures are also misleading. The RGB image is used as an example in Figure 1, which does not reflect the characteristics of hyperspectral imaging. \n2.\tLack of experiments on real hyperspectral imaging. The author has emphasized hyperspectral imaging in the introduction section, but in reality, it has not been verified using real hyperspectral images. The performance of this method in real hyperspectral imaging tasks still needs to be discussed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eu8H9nX4ji", "reviewer": "Reviewer_wWh6", "summary": "The proposed method of this paper is LUM-ViT, a learnable under-sampling mask vision transformer for bandwidth-limited optical signal acquisition. It is a novel approach that utilizes deep learning and prior information to reduce acquisition volume and optimize for optical calculations. The methodology unfolds in two primary stages: training from pre-trained models in a solely electronic domain using existing datasets, followed by inference to evaluate model performance, and assessing the real-world performance of LUM-ViT with a DMD signal acquisition system. During acquisition, target information undergoes a single instance of DMD optical modulation before capture, and then is funneled into the electronic system for further processing.", "questions": "The technical route and core idea in this paper is to accelerate the HSI processing, which is similar to Coded Aperture Spectral Snapshot Imaging (CASSI). Could you please analyze the differences, advantages, and disadvantages of these two systems?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(i) The studied problem about under-sampling hyperspectral data acquisition, achieving data reduction from signal collection while preserving model performance. This accelerates the HSI processing in real applications such as remote sensing, object tracking, medical imaging, etc.\n\n(ii) The idea of using a learnable mask refined during training to selectively retain essential points for downstream tasks from the patch embedding outputs, and thereby achieving under-sampling (reducing the required sampling instances) is interesting.\n\n(iii) The performance is good. On the ImageNet-1k classification task, the proposed LUM-ViT maintains accuracy loss within 1.8% at 10% under-sampling and within 5.5% at an extreme 2% under-sampling.\n\n(iv) This work not only conducts experiments on the synthetic data but also sets up hardware (as shown in Figure 6) to evaluate the effectiveness of the proposed method. It is a good and non-trivial exploration. The accuracy loss of LUM-ViT does not exceed 4% compared to the software environment, demonstrating its practical feasibility.", "weaknesses": "(i) The detailed formulations for DMD are missing, which is confusing. More explanations are required.\n\n(ii) Code and pre-trained weights are not submitted. The reproducibility of this work cannot be checked. \n\n(iii) The multi-stage training pipeline is tedious, which makes the whole technical route unreliable. What's worse, the finetuning details about stage 3 are missing.  Other researchers cannot re-implement this complex approach. \n\n(iv) The writing should be further improved, especially the mathematical notations in Section 3.3. The formula for binary compression is not formal.\n\n(v) The experiments are not sufficient and many critical comparisons are missing. For example, the backbone is ViT variants. However, the ViT is very computationally expensive because its computational complexity is quadratic to the input spatial size. This also embeds the real-time applications of HSI processing. In contrast, MST [1] or MST++ [2] are specially designed for HSI processing. They treat spectral feature maps as a token to capture the interdependencies between spectra with different wavelengths. Most importantly, they are very efficient with linear computational complexity regarding spatial resolution. Thus, it is better to add a comparison with the spectral Transformer in Figure 5.\n\n(vi) The binarization mechanism is out of fashion. BiSCI [3] provides a specially designed binarized convolution unit BiSR-conv block to process HSI data cubes. It is also better to add a comparison with this new technique.\n\n[1] Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction. In CVPR 2022.\n\n[2] MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction. In CVPRW 2022, NTIRE 2022 Winner in Spectral Recovery.\n\n[3] Binarized Spectral Compressive Imaging. In NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)