Title: Cooperative Hardware-Prompt Learning for Snapshot Compressive Imaging
Venue: NeurIPS 2024
Accepted: yes
Keywords: snapshot compressive imaging

hyperpectral imaging

prompt learning

federated learning
Abstract: Existing reconstruction models in snapshot compressive imaging systems (SCI) are trained with a single well-calibrated hardware instance, making their perfor- mance vulnerable to hardware shifts and limited in adapting to multiple hardware configurations. To facilitate cross-hardware learning, previous efforts attempt to directly collect multi-hardware data and perform centralized training, which is impractical due to severe user data privacy concerns and hardware heterogeneity across different platforms/institutions. In this study, we explicitly consider data privacy and heterogeneity in cooperatively optimizing SCI systems by proposing a Federated Hardware-Prompt learning (FedHP) framework. Rather than mitigating the client drift by rectifying the gradients, which only takes effect on the learning manifold but fails to solve the heterogeneity rooted in the input data space, FedHP learns a hardware-conditioned prompter to align inconsistent data distribution across clients, serving as an indicator of the data inconsistency among different hardware (e.g., coded apertures). Extensive experimental results demonstrate that the proposed FedHP coordinates the pre-trained model to multiple hardware con- figurations, outperforming prevalent FL frameworks for 0.35dB under challenging heterogeneous settings. Moreover, a Snapshot Spectral Heterogeneous Dataset has been built upon multiple practical SCI systems. Data and code are aveilable at https://github.com/Jiamian-Wang/FedHP-Snapshot-Compressive-Imaging.git


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a Federated Hardware-Prompt learning (FedHP) framework to address the fact that compressive snapshot spectral imaging devices may not be easily tuneable against changes in the coded aperture, and that in fact the said access to coded apertures may not be possible due to privacy reasons. The authors solve this by hardware prompt learning, which essentially learns from observing diverse coded aperture samples of all clients, regularizing the input data space and achieving the goal of coping with heterogeneity sourcing from hardware. The results show on a specific dataset improvement across all 10 samples in terms of spectral reconstruction quality. The comparison is primarily in the sense of federated learning approaches.\n\nTypo: figure 3  caption -> colorblue shouldn’t be there\n\nSTRENGTHS:\nThe presentation is somewhat accessible to a generally knowledgeable non-expert in federated learning, in that the purposes are clear.\n\nWEAKNESSES:\nThe biggest weakness is arguably that the paper covers a somewhat very niche topic, which is the application of a federated learning scheme to compressive snapshot spectral imaging. To some extent one would expect the technique to abstract away from the specific case of CASSI, as the solution does not particularly pertain to CASSI.\n\nIn addition, due to limited data available in this setup and to very limited size datasets, it is difficult to ascertain the significance of the findings.\n\nQUESTIONS:\nCan the authors extend this to any other compressive imaging scheme? Or perhaps disentangle the improvements in terms of FedHP, from those specific to the application? This would also broaden the data available for validating the experiments.\n\nLIMITATIONS:\nThe addressed setup assumes that the problem the authors propose to tackle is meaningfully posed, I.e., that federated learning in the chosen formulation is practically meaningful. The reviewer is not sure whether this is a practically relevant problem considering that CASSI systems are arguably scientific instrumentation/experimental devices whose calibration is likely done per case anyways.\n\nIn addition the topic would appear to be more meaningful for publications that cover CASSI systems such as IEEE TGARRS or the like. It is hard for this reviewer to disentangle the margin of novelty of this paper in terms of federated learning approach vs. impact on the target application.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenges faced in snapshot compressive imaging (SCI) systems due to hardware shifts and the need for adaptability across multiple hardware configurations. By introducing a hardware-prompt network and leveraging federated learning, the framework enhances the adaptability and performance of SCI models across different hardware configurations.\n\nSTRENGTHS:\n1. The manuscript is well-organized with a clear and logical structure that enhances the readability of the content.\n2. The paper provides a detailed background on SCI and FL. The planned release of the Snapshot Spectral Heterogeneous Dataset (SSHD) will significantly aid future research.\n3. Using different coded apertures for different clients closely mirrors real-world scenarios, adding significant practical relevance to the study.\n\nWEAKNESSES:\n1. The literature review on federated learning (FL) heterogeneity in the Introduction section lacks comprehensiveness. There are numerous recent papers addressing heterogeneity in FL that are not cited here. Additionally, the references included are somewhat outdated. Including more current and diverse references would strengthen the review and provide a more accurate context for the study.\n2. the manuscript explains that the coded apertures for each client follow a specific distribution Pc, it does not provide further details about the exact nature or type of this distribution.\n3. There are many ways to partition data to construct heterogeneous scenarios, such as practical and pathological methods. The approach of equally splitting the training dataset according to the number of clients is not very convincing. The authors should try different partitioning methods.\n4. It is unclear which datasets were used to obtain the experimental results in Tables 1 and 2. The authors did not specify this, which creates confusion in the experimental analysis.\n\nQUESTIONS:\n1. What is the rationale for using adaptors, and what is their function?\n2. What network models are used in the comparison methods? It is necessary to clearly state the fairness of the validated methods.\n3. The explanation for Figure 3 is not detailed enough. For example, what is \"Patch\"?\n\nLIMITATIONS:\n1. In the \"Discussion of the client number\" section, the number of clients increases very little, and the metrics slightly decline. However, the authors conclude that the performance is stable with the change in the number of clients. The small variation in the number of clients is unconvincing. A larger difference in the number of clients should be set to demonstrate this more effectively.\n2. The authors mention the \"presence of data privacy\" in the contributions, but there is no further discussion or experimental comparison regarding data privacy in the subsequent sections. This makes it difficult to validate their contribution to data privacy protection.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMost existing reconstruction models in snapshot compressive imaging systems are trained using a single hardware configuration, making them highly susceptible to hardware variations. Previous approaches attempted to address this issue by centralizing data from multiple hardware configurations for training, but this proved difficult due to hardware heterogeneity across different platforms and privacy concerns. This paper proposes a Federated Hardware-Prompt Learning (FedHP) framework, which aligns data distributions across different hardware configurations by correcting the data distribution at the source, thereby enabling the trained model to adapt to multiple hardware configurations. The performance on existing datasets shows an improvement compared to previous popular training frameworks. Additionally, the authors have released their own created dataset and code.\n\nSTRENGTHS:\n1.Previous work focused on the data itself, directly correcting various types of data through network models. In contrast, the authors of this paper focus on the root cause of the differences—hardware. They address the issue from the perspective of learning the differences in hardware.\n2.The method proposed by the authors has achieved excellent performance compared to existing mainstream methods, and the average performance has also improved.\n\nWEAKNESSES:\n1.The number of clients used in the experiments is still relatively small. Although a simple comparison of the impact of different numbers of clients was made, there is not much difference in performance compared to other methods when the number of clients is larger.\n2.Although good results were reported on simulated data, more results on real data should be included to evaluate the effectiveness of the proosed method.\n\nQUESTIONS:\nWhy does the prompter lead to such a significant improvement, while the effect of the adaptor is not as pronounced? Please provide an in-depth analysis.\n\nLIMITATIONS:\nThe generalization to different hardware systems is crucial for deep learning based methods. The current form of this manuscript only reported on a small scale real dataset captured by several systems. A larger dataset captured by more systems is necessary to evaluate the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FedHP, a reconstruction method for snapshot compressive imaging systems, which addresses the challenge of cross-hardware learning by proposing a federated learning approach. The key contribution lies in using a hardware-conditioned prompter to align data distributions across different hardware configurations, thereby enhancing the adaptability of pre-trained models without compromising data privacy.\n\nSTRENGTHS:\n1. The writing of the paper is good, making it easy to read and follow with clear arguments.\n2. The problem defined in the paper is novel with a clear motivation, providing good inspiration for solving the issue of inconsistent device configurations in snapshot compressive imaging.\n3. The proposed method is clear and the conclusions are relatively convincing. Overall, it is an interesting work.\n\nWEAKNESSES:\n1. There are some typos in the writing. For example, the caption of Figure 3 and the bold parts in the second row of Table 1 and the eighth row of Table 2 are confusing.\n2. The proposed FedHP method is relatively straightforward and lacks deeper insights. Moreover, it does not show a significant performance improvement compared to FedAvg.\n3. The experiments are not comprehensive enough. Given that this work aims to address the snapshot compressive imaging (SCI) problem, I suggest adding experiments to test the applicability of other SCI systems, such as Coded Aperture Compressive Temporal Imaging (CACTI).\n4. There is a lack of sufficient real-world experiments. It would be beneficial to set up multiple independent SCI systems to test the algorithm's performance. Including reconstruction results obtained from these real-world systems is recommended.\n\nQUESTIONS:\n1. All the experiments in this paper are based on the SD-CASSI model. Can the same FedHP model be simultaneously applicable to both DD-CASSI and SD-CASSI architectures, which have significantly different designs?\n2. Although the proposed method outperforms other algorithms in terms of performance metrics, there are still many artifacts in the reconstructed images. While I understand that this is maybe due to the precision issues of the CASSI system, it is crucial for evaluating the practical usability of the algorithm. Additionally, I am not sure whether the spectral accuracy of the reconstructed images is also optimal in statistical terms, which is vital for spectral imaging systems.\n3. Furthermore, if possible, I hope the authors can also address the concerns I raised in the Weaknesses section.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}