Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper addresses the practical application-level problem with joint optimization from front-end sensor to back-end CNN algorithms. Authors validate the proposed method's advantages with comprehensive experiments on different tasks, including image classification, face/pose detection, image segmentation, and image translation. Lastly, the authors claim to contribute its source codes to the research community upon acceptance. Overall, I think this research paper is well-written with no flaw identified; more importantly, I believe this investigation will substantially impact many real-world applications via pushing CNN to edge devices.
One suggestion is to specify the search space for NAS, as the search space is usually application dependent. As it is not specified in the paper, I assume you adopt the default search space, which contains only the 10 operations, and I wonder what is the best set of operations among them that best for the sensor data?
review_title: This paper presents a joint optimization on signal acquisition front-end and back-end CNN algorithms to achieve deep learning's practical significance on edge devices. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: #################################

Summary:

The paper proposed to adopt differentiable network architecture search (DARTS) for the co-design of the sensor (a lensless camera) and the deep model for visual recognition tasks, so as to maximize the accuracy and minimize the energy consumption. The key idea is to include the sensor configuration, in this case the phase mask of a lensless camera modeled as 2D convolutions, as additional parameters in architecture search.  The proposed method was evaluated on simulated data for a number of vision tasks (image classification, face recognition and head pose estimation), as well as using fabricated masks on a real world camera. The results demonstrated significantly increase recognition performance given the same energy level.  

#################################

Pros:
* The high-level idea of using NAS for the co-design of the sensor and the deep model is quite exciting.
* The experiments are extensive, including both several vision tasks in simulation and a classification task using real-world sensor implementation.
 
#################################

Cons:
* Sensor Design: Generality vs Specificity 

The paper proposed to tailor a physical sensor and bundle it with a deep model for a target vision task. Once realized, it is unclear if the sensor is able to adapt to a different task or even a different model. Would the phase mask identified on one model / task generalize to different tasks/models? Generality is an important property of visual sensors (e.g., RGB / ToF cameras). The same camera can be used for different tasks (e.g., image classification, face recognition, etc). And the captured images can be examined by various models. It is probably not surprising that higher accuracy and lower energy can be achieved if a sensor is specifically designed for a single task and for a single type of model. The question is why do we need such a high specificity. It seems very limited even in the IoT setting. For example, what if the backend model needs to be updated. 

* Lack of modeling for fabrication error

Fabrication error is quite significant and is not modeled. For example, on CIFAR 10, fabrication error accounts for a over 4% drop in accuracy, canceling out most of the gains from the co-design (using a Gabor mask has 2-3% drop in accuracy in comparison to the co-design). Is it possible to model fabrication error and encode that into the loss function for NAS? For example, a good design should avoid those patterns that are likely to create issues in fabrication. 

#################################

Minor comments:

The paper has a strong vision flavor and might be a better fit to vision / computational photography conferences. 

#################################

Justification for score:

Overall, this is a paper with a very interesting idea and solid experiments. Yet, the problem setting is limited and the modeling has some issues. I am more positive about this paper after reading the authors' response. 

review_title: An interesting idea with some issues

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################
Summary:
 
SACoD presents a novel attempt to integrate the computational capabilities of a lensless imaging system, PhlatCam, with the search for the optimal convolutional neural network design for a given task. SACoD provides a framework which enables joint optimization of sensor and CNN resulting in IoT devices that achieve higher task accuracy’s with limited resource budgets of a typical IoT system. The authors present a new an optical layer design that enables above described features. Detailed experiments comparing SACoD sensor + CNN with other baseline models covering past papers, demonstrate the superiority of SACoD’s accuracy/efficiency curve over that of separately optimizing CNN arch or sensor/CNN joint-optimizations that do not vary network architecture. Additionally, ablation studies and results from measurements from actual phase masks fabricated help breakdown the accuracy/efficiency benefits of SACoD while analyzing the noise limitations of mask fabrication process.

##########################################################################
Reasons for score: 
 
The paper presents a sound theoretical description of the SACoD approach along with their optical layer design. The results presented through experiments clearly show the superiority of SACoD approach over other baselines that do not utilize the cost-free computational capabilities of the PhlatCam sensor. With the concerns raised in the Cons, sections answered I recommend that this paper be accepted.
 
##########################################################################
Pros:

1.	Section 2 presents a good overview of the different approaches attempting sensor/CNN optimization to improve accuracy for given hardware resources. The proposed solution is unique in attempting to utilize the computation capability of PhlatCam imaging system.
2.	Section 3 presents a good theoretical description of the SACoD framework including the steps involved in its training to estimate the optimal mask, network arch and weights.
3.	It is commendable that as noted at the end of Section 3, authors try out both differentiable and reinforcement learning (RL) based NAS approaches and achieve similar accuracies. Highlighting, the effectiveness of SACoD irrespective of the NAS approach utilized.
4.	Section 4 describes the detailed experiments that the authors used to assess the accuracy/efficiency benefits of SACoD over two other baseline approaches for 6 datasets and 4 tasks. Figures 4/5/6 are clear and help demonstrate the superior accuracy vs efficiency curves plotted for SACoD.
5.	Ablation studies discussed in Section 4.6 on mask flexibility influence and optical layer effectiveness are useful to illustrate the importance of joint optimization of sensor and network as well as the effectiveness of SACoD optical layer.
6.	Figures showing a comparison of the performance of different approaches on vision tasks are helpful to show the superiority of SACoD.
 
##########################################################################
Cons: 
 
1.	Section 3, The optical sensing frontend subsection, describes how the object that is being imaged is at a distance d to the camera while trying to formulate the output of the masks in terms of a 2D convolution operation. Further in this subsection, the authors note that the mask is fixed at a distance d from the sensor. It would be useful to rephrase either of these sentences to clarify what is the distances of the object from the mask and the sensor.
2.	Authors make an effort to describe the impact α on w* and m*, it is not obvious to me as to why the dependence changes for w* and m*. Perhaps an example illustrating the indirect influence of α on m* would be help clarify this.
3.	The experiments in Section 4, present a range of datapoints for SACoD and other baselines for each task/benchmark. In my understanding, these points are obtained by tuning the accuracy vs efficiency tradeoff. How does the size of network change with these experiments? I worry that the efficiency of the GPU utilized for Flops and energy measurements might have biased the results somewhat against SACoD. Testing on a hardware platform with smaller on-chip memory and parallelism might be interesting exercise to show SACoDs superiority.
4.	Section 4.5 is commendable for confirming through measurements that the desired PSF can indeed be generated through fabricated masks. Further the ablation studies discussed in section 4.6 and appendix attempt to model the noise added due to fabrication process in the masks and the resultant accuracy loss. 
5.	Its not clear that the magnitude of normal noise assumed by the authors is sufficiently capturing the fabrication noise measured. Considering that the accuracy drops reported in Table 1 and Figure 10 are for different benchmarks. It would be great if authors could use the same benchmark to model the measured 4% reduction in accuracy and then compare with other baselines. Cause for CIFAR 10, the SACoD approach does not have enough margin to dominate Gabor mask baseline despite a 4% accuracy loss. 
6.	This dependence on the noise during fabrication is a critical weakness of the SACoD approach. Since it relies on the computations carried out by these potentially noisy masks. The authors should consider developing a full-proof solution to this problem by utilizing noise model based training for their models such that the final network can be immune to the fabrication noises.

##########################################################################
Questions during rebuttal period: 

Kindly address the concerns noted in the Cons section

review_title: Review for Submission 2602

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper presents a method called SACoD to develop a more efficient CNN-powered Phlatcam. The proposed method optimizes both the PhlatCam sensor and the backend CNN model simultaneously.  That is, the coded mask in Phlatcam and neural network weights are regarded as learnable parameters. The coded mask (the optical layer) can be considered as a special convolution layer. As a result, it achieves energy saving, model compressing as well as good accuracy. Extensive experiments and ablation studies are presented to show the effectiveness of the method.

Overall,  I think the paper is interesting and properly designed. I also believe that various experiments and analyses of this paper can influence future related studies. However, I hope many experiments using raw images obtained from the real-world lensless imaging system will be included.

Strength

This paper is the first attempt to optimize both sensor and CNN-model at the same time. The derivation and formulation look interesting.


Comments and Weakness

In figure3, the input seems like a normal RGB image. Also, data in datasets such as CIFAR10 and Cityscape are common RGB images. Then, wasn't it applied to the real image obtained using the lensless imaging system? Even in Table 1, reported results were from common RGB images in CIFAR-10. In both optimized and experimental results, source images for testing are not real-world data from lensless imaging systems.

Is there any comparative test of the three methods in Figure3? Is there a reason why they are mentioned? The reasons why design (a) was chosen are mentioned in the text, it is also better to show them experimentally.

To what level can the Flops and energy be lowered based on the same baseline CNN model, and is it enough to put it on the real-world IoT system?

It would be better to compare the proposed method with more than two baseline methods. For example, I wonder what happens when regular CNN without optical layers is used. Also, it would be good to have an experiment on various types of mask filters except for Gabor.

About table2, What is the reason for the big difference in CIFAR-100 while there is not much increase in CIFAR-10?

Font sizes in Figure4 and Figure 6 are too small to read.

review_title: Review on Paper2602 

Meta-review: This paper attempts to jointly search for the sensor and the neural network architecture. More specifically, the proposed approach jointly optimizes the parameters governing the PhlatCam sensor and the backend CNN model. In terms of the approach, the paper follows a well known DARTS formulation for the differentiable architecture search. A very straightforward solution was proposed for the problem.

Although all the reviewers place that the paper is marginally above the acceptance threshold, none of them strongly support the paper and the reviewers point out that the paper is limited in terms of the setting and data. The problem formulation of the paper itself is interesting, but the AC agrees with the reviewers that the paper is limited and lacks enough technical contributions to warrant the acceptance to ICLR.

Decision: Reject