Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging
Authors: Jiamian Wang
Zongliang Wu
Yulun Zhang
Xin Yuan
Tao Lin
ZHIQIANG TAO
Venue: NeurIPS 2024 poster (2024)
Keywords: snapshot compressive imaging
hyperpectral imaging
prompt learning
federated learning
Primary area: machine_vision

Abstract: Existing reconstruction models in snapshot compressive imaging systems (SCI) are trained with a single well-calibrated hardware instance, making their perfor- mance vulnerable to hardware shifts and limited in adapting to multiple hardware configurations. To facilitate cross-hardware learning, previous efforts attempt to directly collect multi-hardware data and perform centralized training, which is impractical due to severe user data privacy concerns and hardware heterogeneity across different platforms/institutions. In this study, we explicitly consider data privacy and heterogeneity in cooperatively optimizing SCI systems by proposing a Federated Hardware-Prompt learning (FedHP) framework. Rather than mitigating the client drift by rectifying the gradients, which only takes effect on the learning manifold but fails to solve the heterogeneity rooted in the input data space, FedHP learns a hardware-conditioned prompter to align inconsistent data distribution across clients, serving as an indicator of the data inconsistency among different hardware (e.g., coded apertures). Extensive experimental results demonstrate that the proposed FedHP coordinates the pre-trained model to multiple hardware con- figurations, outperforming prevalent FL frameworks for 0.35dB under challenging heterogeneous settings. Moreover, a Snapshot Spectral Heterogeneous Dataset has been built upon multiple practical SCI systems. Data and code are aveilable at https://github.com/Jiamian-Wang/FedHP-Snapshot-Compressive-Imaging.git

git Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging
Jiamian Wang Zongliang Wu Yulun Zhang Xin Yuan?, Tao Lin?, Zhiqiang Tao Rochester Institute of Technology. Westlake University,
Zhejiang University, *Shanghai Jiao Tong University
Abstract
Existing reconstruction models in snapshot compressive imaging systems (SCT) are trained with single well-calibrated hardware instance, making their perfor- mance vulnerable to hardware shifts and limited in adapting to multiple hardware configurations. facilitate rross-hardware learning, previous efforts attempt to directly collect multi-hardware data and perform centralized training, which is impractical due to severe user data privacy concerns and hardware heterogeneity across different platforns/insiiutions. In this study, explicitly consider data privacy and heterogeneity in cooperatively optimizing SCI systems by proposing a Federated Hardware- Permmre learning (FedHP) framework. Rather than mitigating the client drift by rectifying the gradients, which only takes effect on the learning manifold but fails solve the heterogeneity rooted in the input data space, FedHP learns a hardware-connee conditioned prompter to align inconsistent data distribution indicator of the data inconsistency among different
across clients, serving hardware (e.g., coded apertures). Extensive experimental results demonstrate that the proposed FedHP coordinates the pre-trained model to multiple hardware figurations, outperforming prevalent FL. frameworks for 0.35dB under challenging heterogeneous settings. Moreover, Snapshot Spectral Heterogeneous Dataset has been built upon multiple practical SCI systems. Data and code are aveilable at https://gituub.co//aaaaaa httpa/////aaaaaaaaa//a/////////-
Introduction
The technology of snapshot compressive imaging (SCI) [Yuan al., 2021] has gained prominence in the realm of computational imaging. Taking an example of hyperspectral image reconstruction. the spectral SCI [Gehm al., 2007] can fast capture and compress 3D hyperspectral signals as 2D measurements through optical hardware, and then restore the original signals with high fidelity by training deep neural networks [Meng et al., 2020, Miao al., 2019]. Despite the remarkable performance [Cai et al., b022a,b, Lin et al., 2022, Huang et al., 2021, Hu et al., 2022], existing deep SCI methods are generally trained with specific hardware configuration, e.g., a well-calibrated coded aperture (physical mask). The resulting model is vulnerable hardware shift/perturbation and limited adapting to multiple hardware configurations. However, directly learning reconstruction model cooperatively from multi-hardware seems to be infeasible due to data proprietary constraint. It also non-trivial to coordinate heterogeneous hardware instances with a unified model.
To elaborate, we first recap previous research efforts of centralized learning solutions. A naive solution to jointly train single reconstruction model with data collected from different hardware configurations, i.e,, coded apertures. As shown in Fig. right, this solution enhances the ability of reconstruction (0.5dB+) by comparison single hardware training scenario. However, the
"Corresponding authors: Jiamian Wang (ju4905erit (du and Zhiqiang Tao (zhiqiang, taoerit. edu)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Trainset
Hordware: CASS!
CASSI contiguration (coded apertune)
Reconstruction learning mothod
APSNR@
10
BUND
M hardware
P4
0.5
series hardware
1. Prevalem
aectment
2. Jaintly train
3. Self-turing
FedAvg
FedHP
0.0
3
Figure 1: Comparison of hyperspectral reconstruction learning strategies. (1) The model trained with the single hardware (Prevalent treatment) hardly handles other hardware. Both (2) Jointly train and (3) Self-tuning [Wang et al., 2022] are centralized training solutions. Both (4) FedAvg and the proposed (5) FedHP adopt the same data split setting. We compare the performance gain of different methods over (1). All results are evaluated by unseen masks -oo-oralapppng) sampled from the practical mask distributions {Pi. P2, Ps). FedHP learns a prompt network (.)) for cooperation.
performance inconsistent coded apertures is still non non-guaranteed since the model only learns to fit coded apertures in a purely data-driven manner. Followed by, self-tuming [Wang et al., 2022] advances the learning by approximating the posterior distribution of coded apertures in variational Bayesian framework. Despite the significant performance hoost, it is only compatible with the coded apertures drawing from homogeneous hardware (same distribution) yet cannot handle heterogeneons hardware. Nevertheless, centralized learning presumes that hardware instances and hyperspectral data are always publicly available, which hardly holds in practice both the optical systems (with different confidential configurations, e.g., coded apertures) and data samples (i.e., measurements captured from overraappp scenes) are generally proprietary assets across institutions, adhering to the strict privacy policy constraints [Vergara-Lauue Laurens et al., 2016, Li et al., 2021]. while considering the multi-hardware cooperative training confining to this concern remains unexplored.
In this work, we leverage federated learning (FL. [Kairouz et al., 2021, Li et al., 2020a, Wang et al., 2021] for cross- pross-platfomm/sllo multi-hardware reconstruction modeling without sharing the hardware configurations and local training data. Firstly, the FL benchmark, FedAvg [McMahan et al., 2017], adopted and brings performance boost (compared by 3 and 4 in Fig. 1 right). However, FedAvg has been proven to be limited in solving heterogeneous data [Hsu et al., 2019, Karimireddy et al., 2020] - the heterogeneity in SCI substantially stems from the hardware, which usually absorbed into the compressed data and governs the network training. Thus, different configurations, e.g., coded apertures, yield different data distributions. Besides, we consider a more practical scenario by extending the sample-wis hardware difference into distribution i.e., not only the different coded apertures yield heterogeneity, but also coded apertures from different clients may follow different distributions (see P1 ~ P3 in Fig. 1).
To adress the heterogeneity issue, this work proposes a Federated Hardware-Promp (FedHP) frame- work to achieve hurli-harrdware cooperative learning with privacy piratically preserved. Prevalent FL methods handle the heterogeneity by regularizing the global/local gradients [Karimireddy et al., 2020, Li et al., 2020b], which only take effect on the learning manifold but fail to solve the heterogeneity rooted in the input data space. Differently, FedHP traces back to the source of the data heterogeneity of this application, f.e., inconsistent hardware configurations, and devises prompt network to solve the client drift issue in input data space. By taking the coded aperture as input, the prompter better accounts for the underlying inconsistency and closes the gap between input data distributions across clients. Besides, the prompter explicitly models the correlation between the software and hardware, empowering the learning by following the spirit of the co-optimization [Goudreault et al., 2023, Zheng et al., 2021, Robidoux et al., 2021] in computational imaging. In addition, FedHP directly operates pre-trained reconstruction backbones with locally well-trained models and keeps them frozen throughout the learning, which improves the training efficiency than directly optimizing the reconstruction backbones in FL from scratch. We summarize the contributions follows.
We introduce and tackle an unexplored problem of hardware cooperative learning in SCI, under the presence of data privacy constraints and heterogeneous configurations. To our best knowledge, the proposed FedHP first integrates federated learning into spectral SCI.

---

We uncover the data heterogeneity of SCI that stems from distinct hardware configurations. A hardware prompt module developed solve the distribution shift across clients and empower the hardware-software co-optimization in computational imaging. The proposed method provides an orthogonal perspective handling the heterogeneity of the existing FL. practices.
We build a new Snapshot Spectral Heterogeneous Dataset (SSHD) from multiple practical spectral snapshot imaging systems. Extensive experiments demonstrate that FedHP outperforms both centralized learning methods and classic federated learning frameworks. The proposed method can inspire future work in this novel research direction of hardware collaboration in SCI.
2 Method
2.1 Preliminary Knowledge
We study the cooperative learning problem by taking the representative setup of coded aperture snap- shot spectral imaging system for hyperspectral imaging as an example, due its recent advances [Cai et al., 2022a,b, Lin et al., 2022]. Given the real-world hyperspectral signal X E IR "x* WNN,, where N₂ denotes the number of spectral channels, the hardware performs the compression with the physi- cal coded apterture M of the size H x W, ie., M [0, 1]. Accordingly, the encoding process
produces 2D measurement YM E TR. Hx where A denotes the shifting
YM-
Σ n==1
X (GAAA)) O M
Yhh, = X(h, w + d(A (AAA nx),
where @ denotes the pixel-wise multiplication and S presents the measurement noise. For each spectral wavelength X, the corresponding signal ((:,,nn, is shifted according to the function d(A X*) by referring to the pre-defined anchor wavelength A", such that A d(Nxx - 1). Following the optical encoder, recent practices train a deep reconstruction network f(-) to retrieve the hyperspectral
data X H xNx by taking the 2D measurement YM input. We define the initial training dataset D and the corresponding dataset for the reconstruction as D.M
D (E. DM = (YM X.))
where X, is the ground truth and Y. is governed by specific coded aperture M The reconstruc- tion model finds the local optimum by minimizing the mean squared loss
i arg min
1 N
Σ llf(e: Y. 1x,
where 8 expresses all learnable parameters in the reconstruction model. = f(:: Y. is the prediction. Pre-trained reconstruction models [Cai et al., 2022a, Huang et al., 2021] demonstrates promising performance when is compatible with a single encoder set-up, where the measurement in training and testing phases are produced by the same hardware using fixed coded aperture of M*
Motivation. Previous work [Wang et al., 2022] uncovered that most existing reconstruction models experience large performance descent (e.g., > 2dB in terms of PSNR) when handling the data encoded by a different coded aperture M from training, i.e., M + M as mask determines the data distribution and also takes effect in learning as (3). Thus, a well-trained reconstruction model can be highly sensitive to a specific hardware configuration of coded aperture and is hardly compatible with the other optical systems in the testing phase. A simple solution of adapting the reconstruction network to a different coded aperture M is to retrain the model with corresponding dataset DM' = (Y.') X, )= and then test upon Mt accordingly. However, this solution does broaden the adaptability of reconstruction models multi-hardware and can introduce drastic computation overhead. In this work, we tackle this challenge by learning a reconstruction model cooperatively from multiple hardware with inconsistent configurations.
2.2 Centralized Learning in SCI
Jointly Train. To solve the above problem, Jointly train (Fig. part 2) serves as naive solution to train model with data jointly collected upon a series of hardware. Assuming there total number
3

---

of K hardware with different coded apertures, i.e, M1. M2....... .... Mx. Each hardware produces training dataset upon DD as DM, M. x. The joint training dataset for reconstruction is DM = DM U Da U...UU U.......M............ UDM,
(4)
where different coded apertures can regarded as hardware-driven data augmentation treatments toward the hyperspectral data, The reconstruction model will be trained with the same mean squared loss provided in (3) upon DM [Wang et al., 2022] demonstrated that jointly learning brings performance boost compared with single mask training (Fig. I right). However, this method adopts single well-trained model to handle coded apertures, failing to adaptively cope with the underlying discrepancies and thus, leading to compromised performances for different hardware.
Self-tuning.
Following Jointly train, recent work of Self-tuning [Wang et al., 2022] recognizes the coded aperture that plays the role of hyperprameter of the reconstruction network, and devel- ops a hyper-net to explicitly model the posterior distribution of the coded aperture by observing DMI-x Specifically, the hyper-net h( M&) approximates P(M by minimizing the Kull- back-Leibler divergence between this posterior and a variational distribution Q(M) parameterized by a. Compared with Jointly train, Self-tuning learns to adapt to different coded apertures and appropriately calibrates the reconstruction network during training, even if there are unseen coded apertures. However, the variational Bayesian learning poses a strict distribution constraint to the sampled coded apertures, which limits the scope of Self-tuning under the practical setting.
To sum up, both of the Jointly train and Self-tuming are representative solutions of centralized learning. where the dataset D and hardware instances with MI...., MM from different sources are presumed to be publicly available. Such a setting has two-fold limitations. (I) Centralized learning does not take the privacy concern into consideration. Hardware configuration and data information sharing across institutions is subject to the rigorous policy constraint. (2) Existing centralized learning methods mainly consider the scenario where coded apertures are sampled from the same distribution, i.e., hardware origin from the same source, which is problematic when it comes to the coded aperture distribution inconsistency especially in the cross-silo case. Bearing the above challenges, in the following, we resort to the federated learning (FL) methods to solve the cooperative learning of reconstruction considering the privacy and hardware configuration inconsistency.
2.3 Federated Learning in SCI
FedAvg. We firstly tailor FedAvg [McMahan et al., 2017], into SCI. Specifically, we exploit a practical setting of cross-silo learning in snapshot compressive imaging. Suppose there are C clients, where each client is packaged with a group of hardware following a specific distribution of Pa
(5) where MC represents k-th sampled coded aperture in e-th client. For simplicity, we use M to denote arbitrary coded aperture sample in e-th client shown in Eq. (5). Based the hardware, each client computes paired dataset DM from the local hyperspectral dataset De
M ~ Pe:
D. = (M, DM = (Y X.HN
(6)
where N₂ represents the number of hyperspectral data in D.. The local learning objective is
N
1 N
(0)
Σ
X
X
where x, f(; Y.M), M model client. FedAvg learns global model Og without sharing the hyperspectral signal dataset Dr,, DMC and M across different clients. Specifically, the global learning objective (G)( is
~ Pa. we use 0 to denote the learnable parameters of reconstruction
= c=1
= (o).
(8)
where C' denotes the number of clients that participate in the current global round and &e represents the aggregation weight. Compared with the centralized learning solutions, FedAvg not only bridges the local hyperspectral data without sharing sensitive information, but also collaborates multi-
hardware with a unified reconstruction model for a better performance (Fig. right comparison between 3 and 4). However, FedAvg shows limitations in two-folds. (1) It has been shown that

---

1. Initialize
2. Local Update (Prompt)
3. Local Update (Adaptor)
4. Aggregation
De
De
NHSA
d
C
Φ1
4
Attendion Nook
HUP
Φ
dbc
Ui
00
Astater
+
(%
Global model
Hardware: CASSI
Prompter
coler
model
Frozen
FC
GEEU
Taned
cone
Previoshned
Client
Client 2
Client c
Figure 2: Learning process of FedHP. We take one global round as example, which consists of (I) Initialize, (2) Local Update (Prompt), (3) Local Update (Adaptor), and (4) Aggregation. For each client, the reconstruction backbone (0g), is initialized as pre-trained model upon local training dataset De and kept as frozen throughout the training. The prompt net upon hardware configuration. i.e., coded aperture, takes effect on the input data of reconstruction, i.e., YM, Adaptors are introduced to enhance the learning, where denotes the parameters of all adaptors.
FedAvg is hard to handle the heterogeneous data [Karimireddy et al., 2020, Khaled et al., 2020, Hsu et al., 2019]. (2) Directly training the reconstruction backbones from scratch would introduce prohibitive computation. Next, we firstly introduce the pordware-induce. data heterogeneity in SCI. Then we develop Federated Hardware-Promp (FedHP) method to achieve cooperative learning without optimizing the client backbones.
Data Heterogeneity. We firstly consider the data heterogeneity stems from the different coded apertures samples, i.e., hardware instances. According to Section 2.1, the optical hardware samples the hyperspectral signal X, from D = {X, JEN and encodes it into a 2D measurement Y.M, which constitutes DM and further serves as the input data for the reconstruction model. To this end, the TENN is vulnerable to the coded aperture variation. A single coded aperture M
modality of (YM)E defines unique input data distribution for the reconstruction, i.e., Y.M PMM(YY))) For arbitrary (Y.M(YMM) (Y Mf + M'. In federated learning, data heterogeneity persistently exists since there is no identical coded aperture across different clients. Such a heterogeneous scenario, i.e., sampling non-ov non-overlapping masks from the same mask distribution, can be caused by lightning distortion optical platform fluttering.
distinct coded apertures, we have PM- (YM') * PMr
We take step further to consider the other type of data heterogeneity stemming from the distinct distributions of coded apertures 2, As formulated in (6), each client collects coded aperture assemble following the distribution P. for e-th client. We have Pa differs from one another, i.e., Pai # Pez for cl * c2, cl, c2 E {1,.... C}. Hardware instances from different clients are produced by distinct manufacturing agencies, so that the distribution Pai and Po2 drastically differs as demonstrated in Fig. 1. This is a more challenging scenario than previous case. As presented in Section 3.2, classic federated learning methods, egg., FedProx [Li et al., 2020b] and SCAFFOLD [Karimireddy et al., 2020] hardly converge while the proposed method enables an obvious performance boost.
2.4 FedHP: Federated Hardware-Prompp Learning
Hardware-Prompl Prompt Learning. Bearing the heterogeneous issue, previous efforts [Li et al., 2020b, Karimireddy et al., 2020] mainly focus on rectifying the global/local gradients upon training, which only takes effect on the learning manifold but fail to solve the heterogeneity rooted in the input data space, whose effectiveness in this low-level vision task may be limited. Since we uncover two types of the heterogeneity in snapshot compressive imaging stemming from the hardware inconsistency (Section. 2.3), this work opts to tackling the client drift issue by directly operating in the input data space. This can be achieved by collaboratively learning the input data alignment given different coded apertures. In light of the visual prompt tuning in large models [Liu et al., 2023b, Bahng et al., 2022], devise
prompt network in the following.
As shown in the Step of Fig, 2, given the input data { (Y MYMis of the reconstruction, the prompt network aligns the input samples, i.e., measurements Y by adding a prompter conditioned on the hardware configuration. Let ((Ø: M) denote the prompt network (e.g., attention block) parameterized
We presume that the hyperspectral single dataset D.. e e= I,.... C, shares the same distribution by generally capturing the natural seenes. Heterogeneity stems from the hyperspectral signal is out of the scope of this work.
5

---

by Φ and Y.M is produced upon coded aperture M. Then, the resulting input sample is aligned as
YM Y. + (6;
In the proposed method, the prompt network collaborates different clients with inconsistent hardware configurations. takes effect by implicitly observing and collecting diverse coded aperture samples of all clients, and jointly learns to react to different hardware settings. The prompter regularizes the input data space and achieves the goal of coping with heterogeneity sourcing from hardware.
Training. As shown in Fig. 2, we demonstrate the training process of proposed FedHP by taking one global round as an example". Since the prompt learning takes effect on pre-trained models, we initialize the e-th backbone parameters with the pre-trained model 0g on local data DM with (7). The global prompt network &G is randomly initialized and distributed to the e-th client
Oe f OG, C 1,... C",
(10)
where the is the local prompt network, and C' denotes the number of clients participated in the current global round. To enable better response of the pre-trained backbonn toward the aligned input data space, we also introduce the adaptors into the transformer backbone. As shown in Fig. Step we show the architecture of the proposed adaptor, which is by a residual connection. We insert the adaptors behind the LN layers.
structure governed
We perform local updates in each global round. It is composed of two stages. Firstly, we update the local prompt network &e for Sp iterations, and fix all the other learnable parameters, The loss is
N
Le
1lf Ec Y." + (MM)) - 1
(1I)
where we use e. represent learnable parameters of all adaptors for c-th client. Secondly, we tune the adaptors for another S₁ iterations. Both of the pre-trained backbone and prompt network are frozen. The loss of e-th client shares the same formulation as (11). After the local update, FedHP uploads and aggregates the learnable parameters &e, c 1,... C' of the prompt network. Since the proposed method does not require to optimize and communicate the reconstruction backbones, the underlying cost is drastically reduced considering the marginal model size of prompt network and adpators compared with the backbone, which potentially serves as a supplied benefit of FedHP.
Compared with FedAvg, FedHP adopts the hardware prompt to explicitly align the input data representation and handle the distribution shift attributing to the coded aperture inconsistency or coded aperture distribution discrepancy.
3 Experiments
3.1 Implementation details
Dataset. Following existing practices [Cai et al., 2022b, Lin et al., 2022, Hu et al., 2022, Huang et al., 2021], we adopt the benchmark training dataset of CAVE [Yasuma et al., 2010], which composed of 32 hyperspectral images with the spatial size as 512 x 512. Data augmentation techniques of rotation, flipping are employed, producing 205 different training scenes. For the federated learning, we equally split the training dataset according to the number of clients C. The local training dataset are kept and accessed confidentially across clients. Note that one specific coded aperture determines a unique dataset according to (2), the resulting data samples for each client can be much more than 203/C. C. We employ the widely-used simulation testing dataset for the quantitative evaluation, which consists of ten 256 x 256 x 28 hyperspectral images collected from KAIST [Choi et al., 2017]. Besides, we use the real testing data with spatial size of 660 660 collected by a SD-CASSI system [Meng et al., 2020] for the perceptual evaluation considering the real- world perturbations.
Hardware. We collect and will release the first Snapshot Spectral Heterogeneous Dataset (SSHD) containing a series of practical SCI systems, from three agencies, each of which offers a series of coded apertures that correspond to a unique distribution' as presented by federated settings in Fig. 2. No identical coded apertures exists among all systems. For the case of inconsistent mask distributions,
We provide an algorithm of FedHP in supplementary.
More illustrations and distribution visualizations of real collected coded apertures are in supplementary.

---

Table I: PSNR(dBySSIM performance comparison. For different clients, sample non-overlapping masks from the same mask distribution to train the model and use unseen masks randomly sampled from all clients for testing. We report ind among 100 trials for all methods.
FedArg
FodProx
SCAPPOLD
PedGST
FodHP
PSNK
SSIM
PSNR
SSIM
SSIM
SSIM
PSNR
SSIM
21.85
21.78.0.21
0.8886-2.085
32.00au014
32.31 2.
60.39
0,8021
20.85
0.8516200007
29:1
30.13an-00
0.85
30.78
10
31 0.27
0,0088
0.0030
20.80
0.896
30.92.0.17
22.0222
20.004
31.1920.32
0.8975
20.00s
31,62,0.25
30.39 0.33
0.9559_...006 0.0001
(19359
30
0.9661
10.0133
39.32,0.30
0.566 -.0003
58.98.0.57
39.78 29.78.0.2
0,9633.0 Lo1
28. n0
0.8821
28. 14
0.8765
28.08 23.04
0.8742200011
28.53
0.8743 20
28.92.0:17
0.89
30.53_6am
0.50054
50
0.91144
29.87
0.9011
30.29.0911
(18849
30.77.00
0.9172cn
50.01=0.20
0.881
29.50.0.00
6.8718
29 29.63
0
29 29.8920-15
0. 1.878622..00
30.44,0.1
0
26.60.6.31
27.74.0.31
88022...08
28.35
1.8752
0,8957
0.0012
10.15
100013
31.22.0.14
0,899 0.8929...004 20.0004
30 30.50
31.34.0.13
2910
0.0000
28.48.0 15
0.8671 005s
28.59g
28.51 15
0,8578 in cons
29.12
0.5835 g000
Arg
0,8059 31,3
30.70 76 i
28900.0.om
30.71
30
0 20
31.35 10
0.9033
Measurement
RGB Reference
FedAvg
FedProx
FedGST
SCAFFOLD
FedHP (Ours)
Ground Truth
503 9wm
57% 3pm
CCAMPOLD 1.000
Follees Ran Fad65: 200 2,09
Patch a
614 4pm
Witweength
45 2-0001
can 3.30 dha
Patch b
Woszlngth
Figure 3: Reconstruction results on simulation data. The density curves compare the spectral consistency of different methods to the ground truth. We the same coded aperture for all methods.
directly assign hardware systems from one source to form client. We simulate the scenario of non-overlapping masks by distributing coded apertures from one source to different clients.
Implementation details. We adopt MST-S [Cai et al., 2022a] as the reconstruction backbone. The prompt network is instantiated by a SwinlR [Liang et al., 2021] block. Limited by the computational resource, set the number of clients as 3 in main comparison. We empirically find that collaborate such amount of clients can be problematic for popular federated learning methods under the very challenging scenario of data heterogeneity (see Section 3.2). For FL methods, we update all clients throughout the training, i.e., C' = C = 3. For the proposed method, we pre-train the client backbones from scratch for 4 x 10tt iterations on their local data. Notably, the total training iterations of different methods are kept as 1.25x x 10" for a fair comparison. The batch is set as 12, We set the initial learning rate for both of the prompt network and adaptor as Opp = ag
10 with step schedulers, i.e., half annealing every 2x 10 iterations. We train the model with an Adam [Kingma and Ba, 2014] optimizer (81 = 0.9, B2 = 0.999). We use Py Torch [Paszke et al., 2017] []] on an NVIDIA A100 GPU.
Compared Methods. We compare FedHP with mainstream FI. methods, including FedAvg [McMa- han et al., 2017], FedProx [Li et al., 2020b], and SCAFFOLD (Karimireddy al., 2020]. Besides, GST [Wang et al., 2022] paves the way for the robustness of the reconstruction toward multiple hardware. Thereby, we integrate this method into the FL framework, dubbed as FedGST. All methods require to train and aggregate the entire client backbones. By comparison, FedHP updates and shares the prompt network, outperforming the others with smaller amount of parameters being optimized and communicated. We adopt PSNR and SSIM [Wang et al., 2004] for the quantitative evaluation.
3.2 Performance
Simulation Results. We quantitatively compare different methods in Table by considering the data heterogeneity stems from non-overggping masks. FedHP performs better than the classic federated
7

---

Table 2: PSNR(dBy/SSIN performance comparison. Masks from each client are sampled from specific distribution for training. We randomly sample non-overlapping masks (unseen to training) from all distributions for testing. We report iadd among 100 trials for all methods.
FedAvg
FodProx
SCAPPOLD
FedGST
FedHP (ours)
PSNK
SSIM
PSNR
SSIM
PSNK
SSIM
SSIM
PSNR
SSIM
29. -0.a0
23:01
22.99
0.5835_a.omm
29.16auee
0.834421
20.37
0.8002_6.0000
20:91
2000
20,99.009
0.447---0015
27.89an.30
2.772822 (o08
28.67.00
28 22.32
0.8464
27157
0:4621 20000
17,58.0.12
0.4618 22.005
28.4520.00
0.8363
29,81 2ouss
0,877
36.93 6.37
(19369
0.9369-00000 0.0006
25
0.4856.0.00
23,00.0.30
0.4848
56.12.0.50
0.9181
37.37
0,9395
to mas
25.84
0.8037
18.99
0:431
18.99
0.4301
26.21
0.7958
27.47
0.5487
27 08.44
0,8655
0.407720
19.10
0.4063
27.52g000
01838420
28.31
0
26 2681
0.5042
0.5042_66.000
20. 20:10
0
30. 14-0..
4883
26.55 27.88
() 1.79577
28.29.0:81
0.22%
25.37_6.3
000000
10.0
0.4002
19.89 L0.05
0,4955
26.22.0.44
1.8206610.000 oien
28. 0.00
000077
18.3
10007
18.30
0,4269 20.077
27.74
0.8199 ao.com
20.3
0.856 Lo
26.54
0,8015
20.06go in
200000
g0
0.3451
25.724668
0.7433
Arg
28.63_6.9
0.84%6 0.0041
20.85
0.540500000
20.00
0.374-
28.04
0.8177
28.98 po
0.8481 g005
RGB Referance
FecdAng
125.1
Tr
Measurement
FedHP
Visualization of reconstruction results on real data. Six representative wavelengths are selected. We use the same unseen coded aperture for both FedAvg and FedHP.
Figure 4:
learning methods. By comparison, FedProx and SCAFFOLD only allows sub-optimal performance, which uncovers the limitations of rectifying the gradient directions in this challenging task. Besides, FedGST works inferior than FedHP, since FedGST approximates the posterior and expects coded apertures strictly follows the identical distribution, which can not be guaranteed in practice. In Fig. 3, we visualize the reconstruction results with sampled wavelengths. FedHP not only enables a more granular retrieval on unseen coded aperture, but also maintains promising spectral consistency as shown by randomly cropped patches (e.g., a, b in Fig. 3).
Challenging Scenario of Heterogeneity. We consider more challenging scenario where the data heterogeneity is caused the distinct coded aperture distributions of different clients. We compare different methods in Table 2. All methods experience large performance degradation, among which FedProx and SCAFFOLD becomes ineffective. Intuitively, it is hard to concur the clients under the large distribution gap, while directly adjusting the input data space better tackles the problem.
Real Results. In Fig. 4, we visually compare the FedAvg with FedHP on the real data. Specifically, both methods are evaluated under unseen hardware configuration, i.e., coded aperture from an uncertain distribution. The proposed method introduces less distortions among different wavelengths. Such an observation endorses FedHP great potential in collaborating hardware systems practically.
3.3 Model Discussion
We conduct model discussion in Table 3. Specifically, accumulate the total cost (e.g., number of parameters, GMACs, and training time) of all clients in a federated system.
Ablation Study. We firstly consider a scenario that trains three clients independently without FL (FedHP w/o FL). For a fair comparison, each client pre-trains the backbone by using the same proce- dure as FedHP and are then enhanced with prompt network and adaptors for efficient fine-tuning. By comparison, FedHP enables an obvious improvement (0.6dB) by implicitly sharing the hardware and data. We then investigate the effectiveness of the prompter and adaptor to the reconstruction, respectively. By observation, directly removing the adaptor leads to limited performance descent. Using prompt network brings significant performance boost. The hardware prompter aligns the input data distributions, potentially solving the heterogeneity rooted in the input data space, considering fact that learning manifold is highly correlated with the coded apertures.
Discussion of the client number. In Table 4a, we discuss the power of FedHP with more real clients under the scenario of Hardware shaking. The performance gap between FedHP and FedAvg consistently remains with the client number increasing, which demonstrates the practicability of the FedHP for the cross-silo spectral system cooperative learning, e.g., 3 ~ 5 clients/insiitutinns.
8

---

Table 3: Ablation study and complexity analysis under the non-overlapping masks. The PSNR (dB)/SSIM are computed among 100 testing trials. We report the model complexity and the accumu-
lative training time of all clients (e.g., C = 3).
Method
Prompter
Adaptor
PL
PSNK
SSIM
#arams (M)
GMACs
Training (days)
FedAvg
41.21
0 00
0.12
10.62
FedHiP woo
30.75+0.11
+.8890
0.27
12.78
2,86
FedlIP w/o Adaptor
31.09
00.997000007
0.15
11.01
2.68
FedHP w/ Prompter
19.1940.01
+0.000
0.12
2.87
2.54
FedHP (Full model)
31,35
0.9033 jn.004
0.27
12.78
2,86
Table 4: Model discussions of the proposed FedHP.
(a) #Client discussion, Averaged values are reported.
(b) Comparison with a deep Unfolding method.
FedAvg
FedHP
Performance gap
Methods
PSNR(dB)
SSIM
#Params (M)
4
31.06
0.8955
31.33
0.9023
0.27
0.0068
GAP-Net
31.07
0.8895
10.0035
3.83
5
31.05
0,9025
31.32
0.9029
0.27
0.0004
FedHP 31.35 0.9033-0.00m
0.27
Comparison with a deep unfolding method. We also compare the proposed FedHP with a repre- sentative deep unfolding method of GAP-Net [Meng et al., 2023] as deep unfolding methods can be adaptable to various hardware configurations. Specifically, we use three clients and keep training and testing settings of GAP-Net the same FedHP. As shown in Table 4b, FedHP improves by 0.28dB with only 7% model size. In fact, despite the adaptability, deep unfolding still shows limitations in solving hardware perturbatiion/epllaemen for a given system [Wang al., 2022].
4 Related Work
Hyperspectral Image Reconstruction. In hyperspectral image reconstruction (HSI), learning deep reconstruction models [Cai al., 2022a,b, Lin al., 2022, Huang al., 2021, Meng et al., 2020, Hu et al., 2022, Miao et al., 2019] has been the forefront among recent efforts due to high- fidelity reconstruction and high-efficiency. Among them, MST [Cai et al., 2022a] devises the first transformer backbone by computing spectral attention. Existing reconstruction learning strategies mainly considers the compatibility toward a single hardware instance. The learned model can be highly sensitive to the variation of hardware. To tackle this practical challenge, GST [Wang et al., 2022] paves the way by proposing a variational Bayesian learning treatment.
Federated Learning. Federated learning [Kairouz et al., 2021, Li et al., 2020a, Wang et al., 2021] collaborates client models without sharing the privacy-sensitive assets. However, FL learning suffers from client drift across clients attributing to the data heterogeneity issue. One mainstream [Karim- ireddy et al., 2020, Li et al., 2020b, Xu et al., 2021, Jhunjhunwala et al., 2023, Reddi et al., 2021] mainly focus on regularizing the global/local gradients. As another direction, personalized FL meth- ods [Collins et al., 2021, Chen and Chao, 2022, Fallah et al., 2020, T Dinh et al., 2020. Jiang and Lin, 2023] propose to fine-tune the global model for better udaptability on clients. However, customizing the global model on client data sacrifices the underlying robustness upon data distribution shift [Wu et al., 2022, Jiang and Lin, 2023], which contradicts with our goal of emphasizing the generality across hardware and thus is not considered. In this work, we propose federated learning framework cooperative learning considering the data privacy and heterogeneity, which to the best knowledge, is the first attempt of empowering spectral SCI with FL. Besides, the principle underlying this method can be potentially extended to broad computational imaging applications [Zheng et al., 2021, Liu et al., 2023a, Goudreault et al., 2023, Robidoux et al., 2021]
to solve the
Conclusion
In this work, we observed unexplored research scenario of multiple hardware cooperative learning in spectral SCI, considering two practical challenges of privacy constraint and the heterogeneity stemming from inconsistent hardware configurations. We developed a Federated Hardware- Prompt (FedHP) learning framework to solve the distribution shift across clients and empower the hardware- software co-optimization. The proposed method serves as first attempt exploit the power of FL in spectral SCI. Besides, we have collected a Snapshot Spectral Heterogeneous Dataset (SSHD) from multiple real spectral SCI systems. Future works may theoretically derive the convergence of FedHP and exploit the behavior of FedHP under a large number of clients. We hope this study will inspire broad explorations this novel direction of hardware collaboration in SCI.

---

References
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modify- ing pixel space to adapt pre-trained models. arXiv preprint arXiv:2203. 2022.
Yuanhao Cai, Jing Lin, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulum Zhang, Radu Timofte, and Luc Van Gool. Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In CVPR, 2022a. 1, 7.9
Yuanhao Cai, Jing Lin, Haoqian Wang. Xin Yuan, Henghui Ding, Yulun Zhang, Radu Timofte, and Luc Van Gool. Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging. NewrlPS, 2022b. 1,3,6 6,9
Hong- You Chen and Wei-Lun Chao. On bridging generic and personalized federated learning for image classification. ICLR, 2022.
Inchang Choi, MH Kim, D Gutierrez, DS Jeon, and G Nam. High-quality hyperspectral reconstruction using spectral prior. Technical report, 2017.
Liam Collins, Hamed Hassani, Aryan Mokhtari, and Sanjay Shakkottai. Exploiting shared represen- tations for personalized federated learning. In ICML, 2021. 9
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In NeurlPS, 2020. 9
Michael E Gchm, Renu John, David J Brady, Rebecca M Willett, and Timothy Schulz. Single- shot compressive spectral imaging with a dual-disperser architecture. Optics express, 15(21): 14013-14027, 2007. 1
Félix Goudreault, Dominik Scheuble, Mario Bijelic, Nicolas Robidoux, and Felix Heide. Lidar-in- the-loop hyperparameter optimization. In CVPR, 2023. 2,9
Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06333. 2019. 2,
Xiaowan Hu, Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool. Hdnet: High-resolution dual-domain learning for spectral compressive imaging. In CVPR, 2022. 1, 9,9
Tao Huang. Weisheng Dong, Xin Yuan, Jinjian Wu, and Guangming Shi. Deep gaussian scale mixture prior for spectral compressive imaging. In CVPR, 2021. 1,3, 6,
Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. Fedexp: Speeding up federated averaging via extrapolation. ICLR, 2023.
Liangze Jiang and Tao Lin. Test-time robust personalization for federated learning. In ICLR, 2023.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and Trends@ in Machine Learning.
14(1-2):1-210, 2021. 2,
Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020. 2, 7,9, 15
Ahmed Khaled, Konstantin Mishchenko, and Peter Richtárik. Tighter theory for local sgd on identical and heterogeneous data. In ICAIS, 2020. 5
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.600 69800, 2014.7
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37(3):50-66 2020a. 2,
10

---

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. In MLSys, 2020b. 2, 7, 15
Yijing Li, Xiaofeng Tao, Xuefei Zhang, Junjie Liu, and Jin Xu. Privacy-preserved federated learning for autonomous driving. IEEE Transactions on Intelligent Transporiation Systems, 23(7):8423- 8434, 2021. 2
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, 2021.
Jing Lin, Yuanhao Cai, Xiaowan Hu, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool. Coarse-to-fi fine sparse transformer for hyperspectral image reconstruction. In ECCV, 2022. 1, 3, 6,
Jiaming Liu, Rushil Anirudh, Jayaraman J Thiagarajan, Stewart He, K Aditya Mohan, Ulugbek S Kamilov, and Hyojin Kim. Dolce: A model- probabilistic diffusion framework for limited- angle reconstruction. In ICCV, 2023a. 9
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhenghao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023b. 5
Patrick Llull, Xuejun Liao, Xin Yuan, Jianbo Yang. David Kittle, Lawrence Carin, Guillermo Sapiro, and David J Brady. Coded aperture compressive temporal imaging. Optics express, 21(9): 10526-10545, 2013. 13
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas. Communication-effiiee learning of deep networks from decentralized data. In AISTATS, 2017.
4,7
Ziyi Meng, Jiawei Ma, and Xin Yuan. End-to-end low cost compressive spectral imaging with spatial-spectral self-attention. In ECCV, 2020. 1, 6, 9
Ziyi Meng, Xin Yuan, and Shirin Jalali. Deep unfolding for snapshot compressive imaging. Interna- lional Journal of Computer Vision, pages 1-26, 2023. 9
Xin Miao, Xin Yuan, Yunchen Pu, and Vassilis Athitsos. X-net: Reconstruct hyperspectral images from snapshot measurement. In ICCV, 2019. 1,9
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Dito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS 2017 Workshop on Autodiff. 2017. 7
Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečný, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization. In ICLR, 2021. 9
Nicolas Robidoux, Luis E Garcia Capel, Dong-eun Seo, Avinash Sharma, Federico Ariza, and Felix Heide. End-to-end high dynamic range camera pipeline optimization. In CVPR, 2021. 2,9
Canh T Dinh, Nguyen Tran, and Josh Nguyen. Personalized federated learning with moreau envelopes. In NeurlPS, 2020. 9
Idalides J Vergara- Laurens, Luis G Jaimes, and Miguel Labrador. for crowdsensing: Survey and research challenges. IEEE Internet of Things Journal, 4(4):855 -69, 2016. 2
mechanisms
Jiamian Wang, Yulun Zhang, Xin Yuan, Ziyi Meng, and Zhiqiang Tao. Modeling mask uncertainty in hyperspectral image reconstruction. In ECCV, 2022. 2, 3, 4, 7,9
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et field guide federated optimization. arXiv preprint 100.:2107.00917. 2021. 2,

---

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from eπror visibility structural similarity. IEEE transactions on image processing, 13(4):600-612. 2004. 7
Shanshan Wu, Tian Li, Zachary Charles, Yu Xiao, Ziyu Liu, Zheng Xu, and Virginia Smith. Motley: Benchmarking heterogeneity and personalization in federated learning. In NeurIPS, 2022.
Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. Fedem: Federated learning with client-level momentum. arXiv preprint arXiv:2106. 10874, 2021. 9
Fumihito Yasuma, Tomoo Mitsunaga, Daisuke Iso, and Shree K Nayar. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE transactions on image processing, 1909):241-2253, 2010.
Xin Yuan, David Brady, and Aggelos Katsaggelos. Snapshot compressive imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine, 38(2):65-88, 2021.
Yucheng Zheng, Yi Hua, Aswin C Samkaranarayanan, and M Salman Asif. A simple framework for 3d lensless imaging with programmable masks. In ICCV, 2021. 2, 9
12

---

A
Appendix supplemental material
We provide more discussions and results of the proposed FedHP as follows
Limitations discussion. (Section A.1).
Broader impacts the proposed FedHP. (Section A.2).
More discussions on new hardware (Section A.3).
Detailed algorithm of FedHP (Section A.4).
More visualizations and analysis (Section A.5).
More discussions on data privacy protection (Section A.6).
More statistical analysis (Section A.7).
A.1
Limitations
One of the limitations of the proposed method is the lack of the real hardares due to the privacy concern. Thus it is hard for us perform the federated learning other tasks like the classification, e.g., C > 100. This in return, motivate us to solve the practical concerns of this field. We working collecting more real data and will continue exploring the power of the proposed method.
large number of the clients in
A.2 Broader Impacts
This work develops a federated learning treatment enable the collaboration of the CASSI systems with different hardware configurations. The proposed method will practically encourage the cross- institution collaborations with emerging optical system designs engaged. By improving the robustness of the pre-trained reconstruction software backend toward optical encoders, this work will help expedite the efficient and widespread deployment of the deep models sensors or platforms.
Table 5: Performance comparison between FedAvg and FedHI on CACTI (e.g., C = 3).
Methods
PSNR (dB)
SSIM
FedAvg
27.35
0,917420.0046
FedHP
27.87
0.919240.0047
A.3
New Hardware
Our key technical contribution provide new multi-hardware optimization framework adapting to hardware shift by only accessing local data. The principle underlying the proposed FedHP can be potentially extended broad SCI applications. This work serves as a proof of concept inspire future endeavors in more general scope. Besides experimental results on CASSI, we also perform additional experiments by applying FedHP to another prevalent SCI system of Coded Aperture Compressive Temporal Imaging (CACTI) [IJull et al., 2013]. The results in Table 5 present performance boost of FedHP over FedAvg baseline (under the same setting as the manuscript), demonstrating that the proposed FedHP does not particularly pertain to CASSI.
A.4 Algorithm
The learning procedure of proposed FedHP provided in Algorithm 1. Let take one global round for example, the learning can be divided into four stages. (1) Initializing the global prompt network from scratch and then distributing it to local clients. Then instantiating the client backbones with the pre-trained models upon the local training dataset. The adaptors are also randomly initialized for a better adaptation of the pre-trained backbones to the aligned input data representation.
(2) Local updating of the prompt network, during which all the other learnable parameters in the system are kept fixed. (3) Local updating of the adaptors. Notably, the parameters of the adaptors is only updated and maintained in local. (4) Global aggregation of the local prompt networks.
13

---

Algorithm FedHP Training Algorithm
Input: Number of global rounds T; Number of clients C; Number of client subset C"; Pre-trained models 0g c = .... C; Number of local update iterations Sp. Sb: Random initialized parameter of prompt network Φc: Random initialized parameter of adaptors of e-th client egi Learning rate ap of prompt network; Learning rate as of adaptors;
Output: Ec, I,.., C,;
1: Server Executes;
2: Randomly choose set of clients of number C";
3: for tort=ll 1,..., do
4: cor C" in parallel do
5: Send global prompt network ΦG to Φei
6: de L-ocaaTraining(0. cc. de);
7: end for
8: @G >'=r" Do
9: end for
10: return do;
LocalTraining(O0,, tor #c):
12: for sus= s, Sp do
Φr. 0. ap VE(0 Cos 6.) using L, N Σi Ilf co Y.M + +(M)) x.
14: end for
for
1,..., Sp
16: E. f Ec - as Ve( co: Φo) using le = 1>E1lll1 (1 cc Y. +*(M)) X.
17: end for
18: return &e to server;
A.5
Visualizations
In this section, we provide more visualization results of different methods. In Figs. 5~6, we present the reconstruction results of different methods under the scenario of hardware shaking, iee, the data heterogeneity is naively induced from the different CASSI instances across clients. FedHP enables more fine-grained details retrieval. Besides, selected representative spatial regions. The higher correlation to the reference, the better spetrum consistency with the ground truth. Figs. 7~9, we show additional real reconstruction results of FedAvg and FedHP on selected wavelengths. By comparison, FedAvg fails to reconstruct some content, while the proposed FedHP allows more granular result.
compare the spectral density curves on
Measurement RGB Reference
FedAvg
FedProx
FedGST
SCAFFOLD
FedHP (Ours)
Ground Truth
be an
ScoeRody din 6ieerr
57 5.3mm
For 0.000
Por
Patch a
600
Whnhrlnnth (rant
Patch b
Fastiig con
GORA
GRAGGRAA
Padc21 5.570
482
500
Waynlength (ym))
Figure 5: Reconstruction results simulation data. The density curves compares the spectral consistency of different methods to the ground truth. We use the same coded aperture for methods.
14

---

Measurement
RGB Reference
FedAvg
FedProx
FedGST
SCAFFOLD
FedHP (Ours)
Ground Truth
471.6nm
64-
Patch
Picttt comn 2006
536.5m 5nm
ReePiin, sen saroo PecCSS 15829
Patch b
567.5 55mm
SOASROUUD der
Fectigg com sells PeecST, 0.008
can
vino
Witvllegghhhp (pm)
Figure 6: Reconstruction results on simulation data. The density curves compares the spectral consistency of different methods to the ground truth. We use the same coded aperture for methods.
Reference
FedAvg 451 6nm
492 Arm
455 Omm
367 Smm
504.4mm 44..frm
625. Tam
Measurement
FedHP
Figure Visualization of reconstruction results on real data. Seven (out of 28) representative wavelengths are selected. We use the same unseen coded aperture for both FedAvg and FedHP.
In Figs. 8, we visualize the different distributions of coded apertures distinct clients under the scenario of the distribution shift of coded apertures among different clients leads to the data het- erogeneity among different local input dataset. This mimics very challenging scenario where in different clients (e.g., research institutions), the corresponding CASSI systems source from different manufacturers. The proposed FedHP allows potential collaboration among different institutions for the hyperspectral data acquisition for the first time despite the large distribution gap. By comparison, classic methods of FedProx et al., 2020b] or SCAFFOLD [Karimireddy al., 2020] fail to provide reasonable retrieval results.
A.6 Data Privacy Protection
FedHP inherently addresses privacy from different perspectives. (1) Hardware decentralization: In the FedHP framework, real hardware configurations (e.g., real masks) remain confidential the local clients. This design makes it difficult to reverse-engineer the pattern or values the real mask without direct sharing. (2) Raw data decentralization: FedHP maintains a private hyperspectral dataset for each client. The hyperspectral images are processed locally (e.g., encoding or data augmentation) and never leaves the client, thereby minimizing the risk of exposure. (3) Training process decentralization: FedHP only collects the local updates from the prompt network, which are then shared with the central server. The local updates are anonymized and aggregated without accessing underlying data, preventing any tracing back to the data source and thus protecting confidentiality. In Table 3, we quantitatively compared
performance of the proposed "FedHP" and "FedHP w/o FL" under privacy- conttraeed environments. FedHP demonstrates a dB average improvement, showcasing its robust model performance and offering a significant privacy advantage that aligns with regulations restricting data sharing.
15

---

10P
110
5.2
3.5
63
08
6.0
02
54
28
10
022
3.5
3.6
18
Figure 8: Coded aperture distributions across Clients under the scenario of manuf vanufacturing discrepancy. The symmetrical logarithm scale is employed for better visualization.
RGB Réference
FedAvg
Anm
Urm
522
557.5mm
694.4nm
625
Figure 9: Visualization of reconstruction results on real data. Seven (out of 28) representative wavelengths are selected. We use the same unseen coded aperture for both FedAvg and FedHP.
A.7 Statistical Analysis
We further conducted a statistical analysis using paired t-test to compare the PSNR and SSIM values from FedHP and FedAvg. We define the hypotheses as follows: (1) Null hypothesis (Hg ): there is no significant difference in the PSNR and SSIM values between FedAvg and proposed FedHP. (2) Alternative hypothesis (H,): there a significant difference the PSNR and SSIM values between FedAvg and proposed FedHP.
We calculated the differences based on the averaged PSNR and SSIM values for each scene from both FedAvg and FedHP, resulting in ten differences values for PSNR (desxa) and SSIM (dsanx). We performed the paired t-test using /a/vi"' where d denotes the mean of the difference values for either PSNR or SSIM, sd is the standard deviations, and is the number of the paired observations.
We calculated the p-value upon the t-distribution for a two-tailed test using the formula p-value = 2 x P(T > [t),, where P(T > |d] denotes the probability that t-distributed random variable with 72 degrees of freedom exceeds the absolute value of the observed t-statistic.
2.50 and p-value is 0.034. Since the p-value is less than the typical significance level of 0.05. Therefore, we reject the null hypothesis (Ho) and conclude that there is statistically significant difference between the PSNR values of FedAvg and FedHP. For SSIM, observe = 7.39 and p-value is 0.00004. The value of is significantly less than 0.05, indicating a very strong statistically significant difference between the SSIM values of FedAvg and FedHP. The test results PSNR and SSIM confirms that the performance gap between FedHP and FedAvg is statistically significant.
For PSNR, we observe
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of the work performed by the authors in the supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We disclose all the information needed to reproduce the main experimental results of the paper in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
18

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The manuscript and the supplementary provides detailed information in reproduce the results. We claim release the dataset, code, and pretrained models in the abstract.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
Beaiiii/guudes//oeeSttttt solliiiiiiiioooooriiioicco/ for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: gww.////e/////gg Saoaaee//oodSSmbaaacaa
/ //nips.c...
onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The manuscript and the supplementary provides detailed information about the experimental setting/dctails.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are accompanied by variances for the experiments that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources has been reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics ptttts://neurpppi cc hetoolii//tiissuiiideeoo
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper will release a new dataset of SSHD. We provide rich details about SSHD in the manuscript and the supplementary.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "Aa3IgBYymf", "reviewer": "Reviewer_BdgG", "summary": "The paper introduces FedHP, a reconstruction method for snapshot compressive imaging systems, which addresses the challenge of cross-hardware learning by proposing a federated learning approach. The key contribution lies in using a hardware-conditioned prompter to align data distributions across different hardware configurations, thereby enhancing the adaptability of pre-trained models without compromising data privacy.", "questions": "1. All the experiments in this paper are based on the SD-CASSI model. Can the same FedHP model be simultaneously applicable to both DD-CASSI and SD-CASSI architectures, which have significantly different designs?\n2. Although the proposed method outperforms other algorithms in terms of performance metrics, there are still many artifacts in the reconstructed images. While I understand that this is maybe due to the precision issues of the CASSI system, it is crucial for evaluating the practical usability of the algorithm. Additionally, I am not sure whether the spectral accuracy of the reconstructed images is also optimal in statistical terms, which is vital for spectral imaging systems.\n3. Furthermore, if possible, I hope the authors can also address the concerns I raised in the Weaknesses section.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing of the paper is good, making it easy to read and follow with clear arguments.\n2. The problem defined in the paper is novel with a clear motivation, providing good inspiration for solving the issue of inconsistent device configurations in snapshot compressive imaging.\n3. The proposed method is clear and the conclusions are relatively convincing. Overall, it is an interesting work.", "weaknesses": "1. There are some typos in the writing. For example, the caption of Figure 3 and the bold parts in the second row of Table 1 and the eighth row of Table 2 are confusing.\n2. The proposed FedHP method is relatively straightforward and lacks deeper insights. Moreover, it does not show a significant performance improvement compared to FedAvg.\n3. The experiments are not comprehensive enough. Given that this work aims to address the snapshot compressive imaging (SCI) problem, I suggest adding experiments to test the applicability of other SCI systems, such as Coded Aperture Compressive Temporal Imaging (CACTI).\n4. There is a lack of sufficient real-world experiments. It would be beneficial to set up multiple independent SCI systems to test the algorithm's performance. Including reconstruction results obtained from these real-world systems is recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that the Reviewer `BdgG` provides valuable comments and finds the problem novel and the method convincing. \n\n`R1.1`: There are some typos in the writing. For example, the caption of Figure 3 and the bold parts in the second row of Table 1 and the eighth row of Table 2 are confusing.\n\n`A1.1`: We will correct the caption of Fig. 3 and the bold parts in Tables 1 and 2 to improve clarity. We will thoroughly proofread the manuscript to eliminate other errors.\n\n`R1.2`: The proposed FedHP method is relatively straightforward and lacks deeper insights. Moreover, it does not show a significant performance improvement compared to FedAvg.\n\n`A1.2`: FedHP offers insightful contributions:\n* It explicitly models hardware variations in computational imaging systems, which is non-trivial and addresses a key practical challenge.\n* The plug-and-play prompt network allows learnable hardware configurations, enabling hardware-software co-optimization.\n* The combination of prompt learning and adaptors reduces communication costs in federated learning systems. \n\nWe empirically find that FedHP enables a consistent performance boost over FedAvg on different number clients, demonstrating the scalability of the proposed method. Considering that FedAvg is a strong baseline that even outperforms prevalent FL methods such as FedProx and SCAFFOLD, it is non-trivial to achieve performance boost over FedAvg. \n\n`R1.3`: I suggest adding experiments to test the applicability of other SCI systems, such as Coded Aperture Compressive Temporal Imaging (CACTI).\n\n| Metrics     | FedAvg     |      FedHP |  \n|:-------------:|:-------------:|:--------------------------:|\n| PSNR      |  $27.35\\pm1.22$    |  $27.87\\pm0.89$   |\n| SSIM       | $0.9174\\pm0.0046$ | $0.9192\\pm0.0047$ | \n\n**Table T1. Comparison between FedAvg and FedHP on CACTI (Client:3)**\n\n`A1.3`: We launch additional experiments by applying FedHP to the real CACTI systems. The results in Table T1 above present a better reconstruction performance of FedHP over FedAvg baseline (under the same setting as Table 1),  demonstrating the generalizability ability of FedHP over a new SCI architecture. We will add the above discussion in the manuscript. \n\nDue to the limited rebuttal time and the heavy workload in deploying real CACTI systems, we can only obtain the intermediate results of both methods (e.g., $1.2\\times 10^4$ out of $4\\times 10^4$). We are committed to updating the latest results during the discussion stage.   \n\n`R1.4`: There is a lack of sufficient real-world experiments. It would be beneficial to set up multiple independent SCI systems to test the algorithm's performance.\n\n`A1.4`: We kindly illustrate that all experiments are performed upon multiple independent real-world SCI systems. For both training and testing, we not only consider sampling non-overlapping masks from the same mask distribution, but also explore a challenging scenario where each client can have real masks sampled from a specific distribution, mimicking the diverse real-world scenarios including hardware perturbation and replacement (L27). \n\n`R1.5`: All the experiments in this paper are based on the SD-CASSI model. Can the same FedHP model be simultaneously applicable to both DD-CASSI and SD-CASSI architectures, which have significantly different designs?\n\n`A1.5`: It is challenging to apply FedHP simultaneously to both DD-CASSI and SD-CASSI architectures due to their different optical designs and forward models. DD-CASSI uses two dispersive elements and a coded aperture, while SD-CASSI uses a single dispersive element and a coded aperture, resulting in distinct measurement formation processes. These differences lead to incompatible forward models, making it difficult to unify them under a single reconstruction model. However, we agree this is an interesting direction for future research. \n\n`R1.6`: There are still many artifacts in the reconstructed images. While I understand that this is maybe due to the precision issues of the CASSI system, it is crucial for evaluating the practical usability of the algorithm. \n\n`A1.6`: Assessing the practical usability of SCI algorithms is inherently difficult due to the complex interplay of hardware limitations, reconstruction algorithms, and application-specific requirements. The proposed FedHP method itself is actually a step towards uncovering and tackling these practical challenges. By explicitly modeling hardware variations and enabling cross-hardware learning, FedHP aims to address real-world issues that arise when deploying SCI systems across multiple devices or institutions. \n\n`R1.7`: I am not sure whether the spectral accuracy of the reconstructed images is also optimal in statistical terms, which is vital for spectral imaging systems.\n\n| Methods | Scene 1 | Scene 2 | Scene 3 | \n|----------|----------|----------|----------|\n| FedAvg | 0.9816 | 0.8905| 0.8634 | \n| FedHP | **0.9901** | **0.9523** | **0.8851** | \n\n**Table T2. Spectral accuracy comparison on three testing scenes.**\n\n`A1.7`: We provide the spectral accuracy comparison in Fig.3, 5, and 6. Specifically, we randomly select a visually informative region (patch) on the hyperspectral images and compute the density values for a specific wavelength (i.e., dividing pixel values on this specific wavelength by the summarized pixel values along all the wavelengths). We statistically measure the spectral accuracy by computing the correlation of the density values between the prediction and the ground truth. FedHP enables better spectral accuracy by comparison. \n\nWe further perform a statistical experiment on three simulation data, on which we randomly choose 10 spatial regions for the averaged correlation computation. As shown in Table T2 above, FedHP consistently enables a higher averaged correlation over FedAvg, indicating a better spectral accuracy. We will add the above discussions into the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QAXeKTfhaH", "reviewer": "Reviewer_Jxiy", "summary": "Most existing reconstruction models in snapshot compressive imaging systems are trained using a single hardware configuration, making them highly susceptible to hardware variations. Previous approaches attempted to address this issue by centralizing data from multiple hardware configurations for training, but this proved difficult due to hardware heterogeneity across different platforms and privacy concerns. This paper proposes a Federated Hardware-Prompt Learning (FedHP) framework, which aligns data distributions across different hardware configurations by correcting the data distribution at the source, thereby enabling the trained model to adapt to multiple hardware configurations. The performance on existing datasets shows an improvement compared to previous popular training frameworks. Additionally, the authors have released their own created dataset and code.", "questions": "Why does the prompter lead to such a significant improvement, while the effect of the adaptor is not as pronounced? Please provide an in-depth analysis.", "limitations": "The generalization to different hardware systems is crucial for deep learning based methods. The current form of this manuscript only reported on a small scale real dataset captured by several systems. A larger dataset captured by more systems is necessary to evaluate the method.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.Previous work focused on the data itself, directly correcting various types of data through network models. In contrast, the authors of this paper focus on the root cause of the differences—hardware. They address the issue from the perspective of learning the differences in hardware.\n2.The method proposed by the authors has achieved excellent performance compared to existing mainstream methods, and the average performance has also improved.", "weaknesses": "1.The number of clients used in the experiments is still relatively small. Although a simple comparison of the impact of different numbers of clients was made, there is not much difference in performance compared to other methods when the number of clients is larger.\n2.Although good results were reported on simulated data, more results on real data should be included to evaluate the effectiveness of the proosed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that the Reviewer `Jxiy` provides valuable comments and finds the proposed method addresses the issue from the new perspective of hardware with good performance. \n\n`R2.1`: The number of clients used in the experiments is still relatively small. Although a simple comparison of the impact of different numbers of clients was made, there is not much difference in performance compared to other methods when the number of clients is larger. \n\n`A2.1`: We find FedHP maintains its performance advantage even with increased client numbers (C=5 and C=10), outperforming FedAvg by 0.21dB and 0.19dB respectively as shown in Table 4 (a). These results demonstrate FedHP's scalability and consistent performance improvements across varying numbers of clients. Besides, it is non-trivial to collect real hardware systems due to the privacy concern and the cost. We are dedicated to facilitating this process by offering a way of cross-silo cooperation using FedHP. We are still working on collecting larger-scale data and real-world hardware systems. \n\n`R2.2`: Although good results were reported on simulated data, more results on real data should be included to evaluate the effectiveness of the proposed method.\n\n`A2.2`: We provide additional comparisons on real data in Fig. 7$\\sim$8. We are still working on collecting more data. \n\n`R2.3`: Why does the prompter lead to such a significant improvement, while the effect of the adaptor is not as pronounced? Please provide an in-depth analysis.\n\n`A2.3`: Thanks for the valuable comment. The prompter plays a crucial role in capturing hardware-specific characteristics and aligning inconsistent data distributions across clients. It directly addresses the heterogeneity rooted in the input data space, which is a key challenge in federated learning for SCI systems. By comparison, the adaptor, while important for efficient fine-tuning, has a more subtle effect on performance. Its primary role is to reduce communication costs in the FL system by allowing us to use pre-trained backbones and only communicate adaptors. \n\n`R2.4`:  The generalization to different hardware systems is crucial for deep learning based methods. The current form of this manuscript only reports on a small scale real dataset captured by several systems. A larger dataset captured by more systems is necessary to evaluate the method.\n\n`A2.4`: We kindly illustrate that current setup can consider a large amount of the real-hardware under two challenging scenarios. \n* Same distribution: As shown in Table 1, we sample non-overlapping masks from the same distribution for different clients. This simulates hardware perturbations or imperfections within the same system design.\n* Different distributions: In Table 2, we explore a more challenging scenario where each client has masks sampled from a specific distribution. This closely mimics diverse real-world scenarios, including hardware replacement or significant variations across institutions.\n\nThese experiments, conducted on real hardware masks with noise (Fig. 9-11 in supplementary), demonstrate FedHP's ability to generalize across various hardware configurations. We appreciate the reviewer's insights on collecting larger dataset. This work laid the foundation for future extension by collecting a Snapshot Spectral Heterogeneous Dataset built upon multiple practical SCI systems for the first time. We are still working on collecting more data and real systems for better evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eO0MWT6rHh", "reviewer": "Reviewer_1X5M", "summary": "The paper addresses the challenges faced in snapshot compressive imaging (SCI) systems due to hardware shifts and the need for adaptability across multiple hardware configurations. By introducing a hardware-prompt network and leveraging federated learning, the framework enhances the adaptability and performance of SCI models across different hardware configurations.", "questions": "1. What is the rationale for using adaptors, and what is their function?\n2. What network models are used in the comparison methods? It is necessary to clearly state the fairness of the validated methods.\n3. The explanation for Figure 3 is not detailed enough. For example, what is \"Patch\"?", "limitations": "1. In the \"Discussion of the client number\" section, the number of clients increases very little, and the metrics slightly decline. However, the authors conclude that the performance is stable with the change in the number of clients. The small variation in the number of clients is unconvincing. A larger difference in the number of clients should be set to demonstrate this more effectively.\n2. The authors mention the \"presence of data privacy\" in the contributions, but there is no further discussion or experimental comparison regarding data privacy in the subsequent sections. This makes it difficult to validate their contribution to data privacy protection.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The manuscript is well-organized with a clear and logical structure that enhances the readability of the content.\n2. The paper provides a detailed background on SCI and FL. The planned release of the Snapshot Spectral Heterogeneous Dataset (SSHD) will significantly aid future research.\n3. Using different coded apertures for different clients closely mirrors real-world scenarios, adding significant practical relevance to the study.", "weaknesses": "1. The literature review on federated learning (FL) heterogeneity in the Introduction section lacks comprehensiveness. There are numerous recent papers addressing heterogeneity in FL that are not cited here. Additionally, the references included are somewhat outdated. Including more current and diverse references would strengthen the review and provide a more accurate context for the study.\n2. the manuscript explains that the coded apertures for each client follow a specific distribution Pc, it does not provide further details about the exact nature or type of this distribution.\n3. There are many ways to partition data to construct heterogeneous scenarios, such as practical and pathological methods. The approach of equally splitting the training dataset according to the number of clients is not very convincing. The authors should try different partitioning methods.\n4. It is unclear which datasets were used to obtain the experimental results in Tables 1 and 2. The authors did not specify this, which creates confusion in the experimental analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that the Reviewer `1X5M` provides valuable comments and finds the proposed work have significant practical relevance to the study and the collected SSHD dataset can benefit the future research. We will release the dataset and the training/testing code. \n\n`R3.1`: The literature review on federated learning (FL) heterogeneity in the Introduction section lacks comprehensiveness.  Additionally, the references included are somewhat outdated. \n\n`A3.1`: We appreciate the reviewer's commitment in helping improve our draft. We will include more current and diverse references to align with the latest development. \n\n`R3.2`: the manuscript explains that the coded apertures for each client follow a specific distribution Pc, it does not provide further details about the exact nature or type of this distribution.\n\n`A3.2`: We visualize the distributions of the real-world masks collected from real CASSI systems in Fig. 5$\\sim$7. The coded apertures for each client follow specific distributions and can reflect the practical variations, including perturbations, shaking, and potential replacements. \n\n`R3.3`: There are many ways to partition data to construct heterogeneous scenarios, such as practical and pathological methods. The approach of equally splitting the training dataset according to the number of clients is not very convincing. The authors should try different partitioning methods.\n\n`A3.3`: We'd like to clarify that our approach already incorporates both practical and pathological methods to construct heterogeneous scenarios by considering heterogeneity stemming from both the dataset and hardware variations. \n* Practical scenario (Table 1): Different clients have non-overlapping masks sampled from the same distribution. This simulates real-world hardware perturbations or imperfections within the same system design, which is a common practical scenario.\n* Pathological scenario (Table 2): Each client has masks sampled from a specific distribution. This represents a more extreme case of significant hardware variations or replacements across institutions, which can be considered pathological.\n\nBesides, the equal splitting of the training dataset across clients, combined with these hardware variations, creates a comprehensive heterogeneous environment.\n\n`R3.4`: It is unclear which datasets were used to obtain the experimental results in Tables 1 and 2. The authors did not specify this, which creates confusion in the experimental analysis.\n\n`A3.4`: All our experiments use the CAVE dataset for the training and the KAIST dataset for testing, both of which are prevalent benchmarks in the field. We will make it clear in the manuscript about the dataset settings. \n\n`R3.5`: What is the rationale for using adaptors, and what is their function?\n\n`A3.5`: The adaptor in FedHP (1) allows efficient fine-tuning of pre-trained backbones, enhancing the model's ability to adapt to new clients. (2) The adaptor significantly reduces communication costs in the FL system by only requiring the transmission of adaptors rather than entire model backbones. As shown in Table 3, the adaptor brings a 0.07dB improvement in PSNR and 0.0029 boost in SSIM. The adaptor is a CONV-GELU-CONV  structure governed by a residual connection (L207).  \n\n`R3.6`: What network models are used in the comparison methods? It is necessary to clearly state the fairness of the validated methods.\n\n`A3.6`: All compared methods use the same network architecture MST [R1] for the reconstruction backbones and a SwinIR [R2] block for the prompt network (see L240\\~L241). The key differences lie in how each method handles the federated learning aspect and hardware variations. In Table 3, we demonstrate the comparable complexity and cost between FedHP and FedAvg. \n\n[R1] Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction. In CVPR 2022. \n\n[R2]  Swinir: Image restoration using swin transformer. In ICCV 2021. \n\n`R3.7`: The explanation for Figure 3 is not detailed enough. For example, what is \"Patch\"?\n\n`A3.7`: The ``Patch'' corresponds to the small regions (such as a, b in Fig.3 top-left) that are randomly cropped for the spectral accuracy evaluation. We will provide more detailed explanations in Figure 3. \n\n`R3.8`: In the \"Discussion of the client number\" section, the number of clients increases very little, and the metrics slightly decline. However, the authors conclude that the performance is stable with the change in the number of clients. The small variation in the number of clients is unconvincing. A larger difference in the number of clients should be set to demonstrate this more effectively.\n\n`A3.8`: While the performance slightly declines with more clients, FedHP maintains its advantage over FedAvg (0.21dB improvement for C=5 and 0.19dB for C=10). We conjecture that the slight descent tendency might attribute to more complex corporations among the clients and less sufficient training samples for each client. It is non-trivial to collect real hardware systems due to the privacy concern and the cost. FedHP is dedicated to facilitate this process by offering a way of cross-silo cooperation. We are also working on collecting larger-scale data and real hardware.\n\n`R3.9`: The authors mention the \"presence of data privacy\" in the contributions, but there is no further discussion or experimental comparison regarding data privacy in the subsequent sections. This makes it difficult to validate their contribution to data privacy protection.\n\n`A3.9`: Thanks for the valuable comment. While we don't explicitly discuss data privacy in our experiments, it is inherent to the federated learning framework we employ. FedHP, like other FL methods, ensures that raw data never leaves the local clients, addressing privacy concerns by design. We will add a dedicated subsection to discuss how our method preserves data privacy and compare it with centralized approaches in terms of privacy protection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OYqo48ZEnI", "reviewer": "Reviewer_aeby", "summary": "The authors present a Federated Hardware-Prompt learning (FedHP) framework to address the fact that compressive snapshot spectral imaging devices may not be easily tuneable against changes in the coded aperture, and that in fact the said access to coded apertures may not be possible due to privacy reasons. The authors solve this by hardware prompt learning, which essentially learns from observing diverse coded aperture samples of all clients, regularizing the input data space and achieving the goal of coping with heterogeneity sourcing from hardware. The results show on a specific dataset improvement across all 10 samples in terms of spectral reconstruction quality. The comparison is primarily in the sense of federated learning approaches.\n\nTypo: figure 3  caption -> colorblue shouldn’t be there", "questions": "Can the authors extend this to any other compressive imaging scheme? Or perhaps disentangle the improvements in terms of FedHP, from those specific to the application? This would also broaden the data available for validating the experiments.", "limitations": "The addressed setup assumes that the problem the authors propose to tackle is meaningfully posed, I.e., that federated learning in the chosen formulation is practically meaningful. The reviewer is not sure whether this is a practically relevant problem considering that CASSI systems are arguably scientific instrumentation/experimental devices whose calibration is likely done per case anyways.\n\nIn addition the topic would appear to be more meaningful for publications that cover CASSI systems such as IEEE TGARRS or the like. It is hard for this reviewer to disentangle the margin of novelty of this paper in terms of federated learning approach vs. impact on the target application.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The presentation is somewhat accessible to a generally knowledgeable non-expert in federated learning, in that the purposes are clear.", "weaknesses": "The biggest weakness is arguably that the paper covers a somewhat very niche topic, which is the application of a federated learning scheme to compressive snapshot spectral imaging. To some extent one would expect the technique to abstract away from the specific case of CASSI, as the solution does not particularly pertain to CASSI.\n\nIn addition, due to limited data available in this setup and to very limited size datasets, it is difficult to ascertain the significance of the findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We much appreciate that the Reviewer `aeby` provides valuable comments and finds the method is designed with clear purpose. \n\n`R4.1`: The biggest weakness is arguably that the paper covers a somewhat very niche topic, which is the application of a federated learning scheme to compressive snapshot spectral imaging. To some extent one would expect the technique to abstract away from the specific case of CASSI, as the solution does not particularly pertain to CASSI.\n\n| Metrics     | FedAvg     |      FedHP |  \n|:-------------:|:-------------:|:--------------------------:|\n| PSNR      |  $27.35\\pm1.22$    |  $27.87\\pm0.89$   |\n| SSIM       | $0.9174\\pm0.0046$ | $0.9192\\pm0.0047$ | \n\n**Table T1. Comparison between FedAvg and FedHP on CACTI (Client:3)**\n\n`A4.1`: Our key technical contribution is to provide a new multi-hardware optimization framework adapting to hardware shift by only accessing local data. The principle underlying the proposed FedHP can be potentially extended to broad SCI applications. However, due to the practical cost of data acquisition and building optics systems, this study explores one specific direction, where we focus on spectral SCI and collecting optical masks and real data from multiple hardware.\n\nExploiting the hardware collaboration of computational imaging systems is still in an early stage. This work serves as a proof of concept to inspire future endeavors in a more general scope. We list several potential related applications that might benefit from the proposed method, such as Lensless camera [1], LiDAR [2], HDR camera [3], or CT-Reconstruction [4], cooperating multiple imaging systems via aligning forward models, etc. \n\nPer reviewer's inspiration, we tried our best to launch additional experiments by applying FedHP into another prevalent SCI system of Coded Aperture Compressive Temporal Imaging (CACTI). The results in Table T1 above present a performance boost of FedHP over FedAvg baseline (under the same setting as Table 1),  demonstrating that the proposed FedHP does not particularly pertain to CASSI. Due to the limited rebuttal time and the heavy workload in deploying real CACTI systems, we can only obtain the intermediate results of both methods (e.g., $1.2\\times10^4$ out of $4\\times 10^4$). We are committed to updating the latest results during the discussion stage.  We will add the above discussion in the manuscript. \n\n[1] A simple framework for 3D lensless imaging with programmable masks. CVPR 2021.\n\n[2] LiDAR-in-the-loop Hyperparameter Optimization. CVPR 2023.\n\n[3] End-to-end high dynamic range camera pipeline optimization. CVPR 2021.\n\n[4] DOLCE: A model-based probabilistic diffusion framework for limited-angle ct reconstruction. ICCV 2023.\n\n[5] Deep unfolding for snapshot compressive imaging. IJCV 2023.\n\n\n`R4.2`: In addition, due to limited data available in this setup and to very limited size datasets, it is difficult to ascertain the significance of the findings.\n\n`A4.2`: This work has constructed  the Snapshot Spectral Heterogeneous Dataset (SSHD), which includes multiple practical spectral SCI systems. This dataset will be released to support further research and validation. Despite the current dataset size, our experiments demonstrate consistent performance improvements using FedHP, even under challenging heterogeneous settings. This provides a strong indication of the robustness and potential of the proposed method.\n\n`R4.3`: Can the authors extend this to any other compressive imaging scheme? Or perhaps disentangle the improvements in terms of FedHP, from those specific to the application? This would also broaden the data available for validating the experiments.\n\n`A4.3`: As shown in `Table T1` and `A4.1`, we tried our best to launch additional experiments by extending FedHP into another prevalent SCI system of Coded Aperture Compressive Temporal Imaging (CACTI), which is used to compress and recover temporal information. By comparison, FedHP outperforms FedAvg by 0.52dB/0.0018 in terms of the PSNR/SSIM despite both only training for $1.2\\times10^4$ iterations out of $4\\times 10^4$. The results indicate that FedHP demonstrates the generalization ability on new SCI systems. We are committed to updating the latest results during the discussion stage.  We will add the above discussion in the manuscript. \n\n`R4.4`: The addressed setup assumes that the problem the authors propose to tackle is meaningfully posed, I.e., that federated learning in the chosen formulation is practically meaningful. The reviewer is not sure whether this is a practically relevant problem considering that CASSI systems are arguably scientific instrumentation/experimental devices whose calibration is likely done per case anyways.\n\n`A4.4`: While CASSI systems are specialized scientific instruments, the challenge of hardware heterogeneity and data privacy remains significant. Training models for each new hardware configuration can be impractical due to data-starving clients and high computational costs, and the cross-silo cooperation can be impractical without solving privacy concerns for the hardware. FedHP provides a solution and can facilitate broader and more practical deployment of SCI systems in diverse settings.\n\n`R4.5`: In addition the topic would appear to be more meaningful for publications that cover CASSI systems such as IEEE TGARRS or the like. It is hard for this reviewer to disentangle the margin of novelty of this paper in terms of federated learning approach vs. impact on the target application.\n\n`A4.5`: Our work intersects federated learning and computational imaging, addressing challenges in both fields. The novelty of our approach lies in the integration of a hardware prompt network within a federated learning framework to handle hardware heterogeneity and data privacy issues. This contribution is relevant to both federated learning and SCI communities, and we believe it provides valuable insights and advancements applicable to various imaging systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents a hardware focused method for performing snapshot compressive imaging in cases where differences in hardware are adjusted in the trained model. Federated learning is used towards this end. Four reviewers considered the paper, with two voting Weak Accept and two Borderline Accept. There was good engagement in the discussion period to indicate that they think the idea a good one and the results are potentially of interest. While the average score for this paper technically puts it on the threshold between accept and reject, the small variance between reviewers, and the fact that all consider the paper above threshold for accept, is a strong signal that it will contribute to the conference and is worth accepting.
Decision: Accept (poster)