SACoD: Sensor Algorithm Co-Design Towards Efficient CNN-powered Intelligent PhlatCam
Authors: Yonggan Fu
Yang Zhang
Yue Wang
Zhihan Lu
Vivek Boominathan
Ashok Veeraraghavan
Yingyan Lin
Venue:  (2021)
Keywords: Sensor Network Co-design
neural architecture search

Abstract:  There has been a booming demand for integrating Convolutional Neural Networks (CNNs) powered functionalities into Internet-of-Thing (IoT) devices to enable ubiquitous intelligent "IoT cameras”. However, more extensive applications of such IoT systems are still limited by two challenges. First, some applications, especially medicine- and wearable-related ones, impose stringent requirements on the camera form factor. Second, powerful CNNs often require considerable storage and energy cost, whereas IoT devices often suffer from limited resources. PhlatCam, with its form factor potentially reduced by orders of magnitude, has emerged as a promising solution to the first aforementioned challenge, while the second one remains a bottleneck. Existing compression techniques, which can potentially tackle the second challenge, are far from realizing the full potential in storage and energy reduction, because they mostly focus on the CNN algorithm itself. To this end, this work proposes SACoD, a Sensor Algorithm Co-Design framework to develop more efficient CNN-powered PhlatCam. In particular, the mask coded in the PhlatCam  sensor and the backend CNN model are jointly optimized in terms of both model parameters and architectures via differential neural architecture search. Extensive experiments including both simulation and physical measurement on manufactured masks show that the proposed SACoD framework achieves aggressive model compression and energy savings while maintaining or even boosting the task accuracy, when benchmarking over two state-of-the-art (SOTA) designs with six datasets on four different tasks. We also perform visualization for better understanding the superiority of SACoD generated designs. All the codes will be released publicly upon acceptance. 

Under review as a conference paper at ICLR 2021
SACoD: SENSOR GLGORITHM CO-DESIGN TOWARDS EFFICIENT CNN-POWEREE INTELLIGENT PHLATCAM
Anonymous authors Paper under double-blind review
ABSTRACT
There has been booming demand for integrating Convolutional Neural Networks (CNNs) powered functionalities into Internet-of-Thing (IoT) devices to enable ubiquitous intelligent "IoT cameras". However, more extensive applications of such IoT systems are still limited by two challenges. First, some applications, especially medicine- and wearable-related ones, impose stringent requirements on the camera form factor. Second, powerful CNNs often require considerable storage and energy cost, whereas IoT devices often suffer from limited resources. PhlatCam, with its form factor potentially reduced by orders of magnitude, has emerged promising solution the first aforementioned challenge, while the second one remains a bottleneck. Existing compression techniques, which can potentially tackle the second challenge, are far from realizing the full potential in storage and energy reduction, because they mostly focus on the CNN algorithm itself. To this end, this work proposes SACoD, Sensor Algorithm Co-Design framework to develop more efficient CNN-powered PhlatCam. In particular, the mask coded in the PhlatCam sensor and the backend CNN model are jointly opti- mized in terms of both model parameters and architectures via differential neural architecture search. Extensive experiments including both simulation and physical measurement manufactured masks show that the proposed SACoD framework achieves aggressive model compression and energy savings while maintaining or even boosting the task accuracy, when benchmarking (SOTA) designs with six datasets on four different tasks. We also evaluate the per- formance of SACoD on the actual PhlatCam imaging system with visualizations and experiment results. All the codes will be released publicly upon acceptance.
two state-of-the-art
INTRODUCTION
Recent CNN breakthroughs trigger growing demand for intelligent IoT devices, such wearables and biology devices (e.g., swallowed endoscopes). However, two major challenges are hamper- ing more extensive applications of CNN-powert IoT devices. First, some applications, especially medicine- and biology-related ones, impose strict requirements on the form factor, especially the thickness, which are often too stringent for existing lens-based imaging systems. Second, powerful CNNs require considerable hardware costs, whereas IoT devices only have limited resources.
For the first challenge, lensless imaging systems (Asif et al. 2015 Shimano al. 2018 Adams ett al. 2017 Antipa et al., al. (2018: Boominathan et al. (2020) have emerged as a promising rescue. For example, PhlatCam (Boominathan et al. (2020) replaces the focal lenses with set of phase masks, which encodes the incoming light instead of directly focusing it. The encoded information can be either computationally decoded reconstruct the images or processed specifically for different applications. Such lensless imaging systems can be made much smaller and thinner, because the phase masks are smaller than the focal lens, and they can be placed much closer the sensors and fabricated with much lower costs. For the second challenge, many recent works focus designing CNNs with improved hardware efficiency, i.c., by applying generic neural architecture search (NAS) to find efficient CNNs.
As such, a naive way to address the two aforementioned challenges simultaneously is to introduce lensless cameras the signal acquisition frontend and then apply NAS to optimize the backend CNN. However, such approaches would result in disjoint optimization that can be far from optimal. A generic NAS would treat the camera given, and only optimize the CNN. Likewise, existing

---

Under review as a conference paper at ICLR 2021
Input 9 User demands (Acc., energy,
The Proposed SACoD Framework
Output
Sensor Search
Algorithm Search
Image
Hardware resources
si Plase Moak
Mhase Mast
Updare
Dask
Integrated System
Figure 1: An overview of the proposed SACoD framework.
phase mask designs for lensless cameras treat the CNNs as given, and only optimize the masks. Such disjoint optimization fails to (I) take advantage of the masks' potential computational capacity, with which the NAS optimization can be fundamentally improved, and (2) perform end2end optimization.
It is shown in (Boominathan al. 2020) that, under some assumptions, the phase masks in Phlat- Cam essentially perform 2D convolutions on the incoming lights, and the convolution kernel is encoded in the masks. Moreover, unlike other convolutional layers, the phase masks' convolutions are almost free - they do not consume additional energy, computation power, or storage, regardless of what value each mask takes. Therefore, we aim incorporate the phase mask design into NAS to enable an end2end optimization of the proeiinggoossinggggg pipeline, while exempting a portion of the pipeline from the efficiency penalties. Such co-designs are expected to achieve better tradeoffs between accuracy and efficiency.
To this end, we propose a Sensor Algorithm Co-Design (SACoD) framework to enable more energy- efficient CNN- CoW-red IoT devices. While SACoD is general and can be applied to different sensing and intelligent processing systems, it is developed and evaluated in the context of PhlatCam (Boom- inathan et al. 2020) based imaging systems. Our main contributions are:
We propose SACoD, novel co-design framework that jointly optimizes the sensor and neural networks to enable more energy energy-efficient CNN- CNN-powered IoT devices. To our best knowledge, SACoD
first to propose sensor algorithm co-design for CNN inferences.
We develop effective design of the optical layer to (1) exploit its potential computation capability and (2) enable co-search of the optical layer and backend algorithm. We then characterize the trade-off between accuracy and the required area of the corresponding imaging systems to demonstrate its effectiveness under practical size constraints.
Extensive experiments and ablation studies validate that the proposed SACoD consistently achieves reduced hardware costs/area while offering a comparable or even better task ac- curacy, when evaluated over two SOTA lensless imaging systems on four tasks and six datasets. And part of the experiments are further evaluated with fabricated masks to vali- date SACoD's effectiveness in the physical measurement besides simulation.
RELATED WORKS
Neural architecture search. Recently NAS has attracted increasing attention. It eliminates the handcrafting process and automatically designs neural architectures. Existing NAS techniques can be divided into three categories, evolution- based, reinforcememt-lermingg (RL)-based, and one-shot RL-based approaches can be unacceptably high, many techniques (Brock et al. 2017 Cai et al. 2018a Liu et al. 2017 2018 Pham et al.) 2018 Xie et al. 2018) have been proposed to reduce the searching cost, among which differentiable archi- tecture search (DARTS) has gained intensive interests. While being conceptually general, SACoD in this paper adopts the DARTS method, where a super-network is optimized during search and the strongest sub-network is preserved and then retrained.
NAS. As the computational overheads of evolution-
Lensless imaging systems. To eliminate the size or thickness burden caused by the lens, various lensless imaging systems have been developed. While lensless imaging systems have been widely
used for capturing X-ray and gamma-ray (Dicke 1968 (Crolii et al. 1987), it is still in an exploring
stage for visible spectrum uses (Asif et al. 2015 Shimano al. 2018 Antipa al. 2018
Boom-
2

---

Under review as a conference paper at ICLR 2021
inathan et al., 2020). In general, lensless imaging systems capture the scene either directly on the sensor or after being modulated by a mask element.
In this paper, focus imaging system based on phase masks called PhlatCam (Boominathan al. 2020). which is a general-purpose framework to create phase masks that can achieve desired sharp point- spread-functions (PSFs). A phase mask mod- ulates the phase of incident lights, and allows most of the light to pass through, providing a high signal-to-noise ratio. Hence, they are desirable for low light scenarios and photon- limited imaging. Fig. shoows a fabricated phase mask, which is essentially a transparent material with different thicknesses at different locations. Based on this lensless imaging system, we develop our SACoD framework, aiming to enable more energy-efficient CNN-powered IoT devices.
specific lensless
5.mm
pm
Figure 2: A fabricated phase mask used in the PhlatCam lensless imaging system (Boominathan jet al. 2020).
Sensor-algorithm co-training. There have recently been some attempts that try to jointly optimize the sensor parameters and the neural network backend. For lens-based image systems, novel lens designs are introduced and trained concurrently with the neural network backend to jointly optimize for image reconstruction (Sitzmann et al. 2018), depth estimation (Chang & Wetzstein 2019). and
high-dynamic-rang imaging (Metzler et 2019). Similar approaches have been applied other imaging systems as well, including cameras with color multiplexing (Chakrabarti] 2016), Phase- Cam3D (Wu et al.] (2099)) and Single Photon Avalanche Photodiodes cameras (Sun et al. 2020).
Yet these methods still consider the neural network architecture as fixed, and do not explore the potential of applying the co-design principle the neural architecture design.
THE PROPOSED SACOD FRAMEWORK
This section presents the SACoD framework. We will first outline the framework and explain why the optical layer can be considered as convolutional layer, and then introduce how we implement SACoD's optimization algorithm. Finally, we describe our developed optical layer design.
Framework setup. The SACoD framework shown in Fig. [ consists of two modules, an optical sensing frontend and a neural network backend. The coded mask PhlatCam in the sensor and the backend are jointly optimized using SOTA differential NAS algorithm (Liu et al.) 2018), where the coded masks, together with the neural network weights, are regarded as network parameters.
Specifically, the first module, i.e. the optical sensing frontend, is denoted as O(:,),, where vn = m(x., v) denotes the phase mask values. The optical layer is based on the PhlatCam sys- tem (Boominathan et al.) (020)). It receives the light signal from the object in front of the camera, processes the signal using the phase masks, and generates the sensor output. The second module, i.e. the neural network backend, is denoted F(:; a), where w represents weights of the neu- ral network, and a parameterizes the architecture. The neural network backend receives the sensor signal and produces an output for the intended applications.
Formally, denote the light signal as I(x,)) E B HxWx3 where and are coordinate indices and H and W represent the height and width of the range of light the camera can receive. The light signal contains RGB channel, and hence the last dimension is 3. Denote the signal received at sensor as (zx w) E R H''w''xN 'N where H' and w' represent the height and width, and N represent the number of channels. Denote Y as the final output of the neural network backend. Then we have
Z = 0(I,),
F(Z; w. o).
The following subsections will introduce the form of O(:: m) and how to determine m, w, and a.
The optical sensing frontend. Assuming that the light signal I(x,)) comes from an object whose distance the camera d, and that the depth of the object is relatively small, it can be shown (Boominathan et al. 2020) that O(:; m) takes the following convolutional form:
2(x,8) v) O(I; m)) = (zz, m, d) I(x,y),
where * denotes 2D convolution, p(x, yı m, =) is called the point spread function of the phase mask, which is determined by the phase mask m(x,)) y) and the distance d. Once we optimize the PSF, the
3

---

Under review as a conference paper at ICLR 2021
phase masks are designed for the PSF and chosen d. The fabricated mask then produces the PSF at the given d. For the fabricated system shown in 4.5 d is set to be 2 mm for making our system much thinner than conventional cameras (thickness ranges between 7-20 mm). The mask is fixed at distance d to the sensor during operation, and thus the convolution property will continue to hold. According to Eq. equation [ the optical layer can be regarded as special convolutional layer. Note that one phase mask can only perform a single-channel convolution with a positive kernel. It takes two phase masks to implement a single-channel convolution with a real-valued kernel, where one implements the positive part of the kernel and the other implements the negative part. Therefore, in order to construct a convolutional layer with three channels and real-valued kernels, we need six masks in the imaging system. Also, the input light has three color channels (R, G, and B), and each phase mask operates on all the color channels. Therefore, a three-channel convolution will produce a total of nine feature maps (FMs).
We propose three different designs for using the rendered FMs, as shown in Fig. 3 Specifically, the design in Fig. (a) accumulates the FMs across the same color and outputs a 3-channel FM, which is still in an RGB shape; the design in Fig. B (b) accumulates the FMs from one mask across three colors and outputs 2-channel FM; and the design in Fig. (c) simply concatenates all the FMs from different colors. These three different designs extract different amounts or types of information from the scene which are then passed the following neural network.
Experiments under various settings show that SACoD based on Fig. 3 achieves higher accura- cies than those based Fig. (b), and similar accuracies as the designs based on Fig. (c). We conjecture the reason is that design (a) applies independent transformations on the RGB channels to maintain the original channel-wise discriminative information; and maintaining all the information as design (c) does not contribute to higher accuracy than design (a), since the accumulated infor- mation across the same color has provided sufficient information for the following processing under the constrained number of masks. So in our experiments, we adopt the optical layer design (a).
The SACoD formulation and algorithm. This subsection introduces the formulation and optimiza- tion of SACoD which aims to simultaneously optimize the phase mask m, and the neural network's s architecture a, and the neural network's weights w. Formally, SACoD aims to solve:
min Lent (m" (ax), w" (a)) + (a)) ca
(3)
m (ax), w (ax) arg min Cer (m, w.)x).
(4)
(m.w))
for and Leai are task-specific performance losses evaluated on the training and validation set, respec- tively, L. is the efficiency loss, e.g. model size, computational cost, energy consumption, and A is the tuning parameter trading-off the accuracy and efficiency. Following the same parameterization scheme in DARTS (Liu et al. 2018), a denotes the weights of different candidate operations.
There are two major modifications compared to the original DARTS framework. The first difference is that the efficiency loss L., measured by the sum of each layer's FLOPs weighted by the network parameter a, is introduced. More importantly, the second and major difference is that the phase mask is optimized jointly in the framework. is worth pointing out that although mathematically similar, m and w" have different degrees of dependencies on a. w" is directly impacted by a because a governs which subset of the w is ultimately used, m' only indirectly influenced by a. Therefore. incorporating m will significantly improve the tradeoff between performance and model complexity. Besides, SACoD naturally compatible with other NAS methods. We adopt differential NAS for the fast generation of the optical mask and network. When using other NAS methods, c.g., reinforcement learning- baseid NAS (Zoph & 2017), we still observe similar system performance (within 0.3% accuracy CIFAR-100), but the search time increases to 8 GPU-days from 0.5 GPU-days.
The whole co-design process can be divided into two stages, searching stage and training stage. In the searching stage, we apply the alternate gradient descent of Eqs. equation Bnd equation to search for the optimal network architecture x In the training stage, the optimal mask and weights are determined by optimizing Eq. equation 4 conditioning on the optimal network architecture a..
4 EXPERIMENTS
This section presents evaluation results of SACoD applied to PhlatCam lensless imaging sys- tems. We first describe the experiment settings, and then benchmark SACoD over SOTA lensless
4

---

Under review as a conference paper at ICLR 2021
(a)
(b)
Figure 3: An illustration of the proposed three different optical layer designs.
imaging systems on standard classification tasks, IoT applications, and other vision tasks in Sec-
tions 4.2
4.4
respectively. Finally, we provide ablation studies of SACoD in Section 4.6
4.1 EXPERIMENT SETUP
Optical layer constraints. As mentioned, the optical layer first performs convolutional operations on the input scene optically, the outputs which are then processed by the following neural network. The physical device construction imposes design constraints the optical layer design. Specifically, since the phase mask is placed closer to the sensor, the optically achievable kernel size cannot be arbitrarily small. Here, adopt kernel sizes that are not smaller than 7x7. Additionally, since all the designed masks are sharing the same sensor area, the number of designed masks cannot be large due to the limited sensor area. Here, we constrain the number of masks to be no more than 6.
Algorithm setting, Datasets: we evaluate SACoD on total of four vision tasks with six datasets: two classification datasets CIFAR-10/100, two IoT datasets including FlatCam Face (Tan et al. 2019) and Head Pose Detection (Gourier & Crowley) 2004), one segmentation dataset Cityscapes (Cordts et al.,, (2066), and one unsupervised image translation dataset horse2zebra (Zhu al. 2017). The same and standard data augmentation methods (e.g., random crop and normaliza- tion) are adopted for both SACoD and the baselines.
Baselines: we benchmark SACoD over two SOTA lensless imaging systems:
Gabor-mask System: we fix the optical layer to be the Gabor-mask (Chen et al. 2016) and search for networks using the same NAS method as SACoD.
Co-train System: we fix the network to SOTA ToT CNN MobileNetV2 (Sandler al.) 2018 and jointly train the optical layer and the backend network.
Efficiency metrics: consider both FLOPs (Floating Point Operations) and energy cost based on real-device measurements the efficiency metrics. Specifically, adopt NVIDIA JETSON TX2 (NVIDIA Inc.). a popular IoT GPU, as the target platform, which is connected to laptop with the real-time energy cost being obtained via the sysfs (Patrick Mochel and Mike Murphy.) of the (Texas Instruments Inc.) power rails monitor.
embedded INA3221
44.1%
Accuracy (%)
Accuracy (R))
462
-48.7%
Accurocc
+1.6%
-70.9%
+1.8%
20
+7.9%
Recuume
70
+7.9%
-50.%
SACTO (Proposeet
66
56
66
150
100
12.5
17.5
PLOPs (x30F)
Energy (mj)
PLOPs (x109)
Energy (m))
CIFAR 10
CIFAR 100
Figure Accuracy vs. FLOPs/energy trade-offs of SACoD and the baselines on CIFAR-10/100.
4.2 SACOD OVER SOTA IMAGING SYSTEMS CLASSIFICATION TASKS
Search and training setting. To search neural networks CIFAR-10/100 for both the SACoD and Gabor-mask systems, we quantize all the operations to 8-bit using a SOTA quantization training method (Banner al. 2018), which is common practice considering the constrained sources loT devices. We adopt the search settings in (Wu et al. 2019a) with minor changes discussed in appendix. Specifically, search for 50 epochs with 64 batch size, and update the supernet weights
5

---

Under review as a conference paper at ICLR 2021
on half of the training dataset using an SGD optimizer with a 0.9 momentum, and an initial learning rate of 0.025 with the cosine decay, and update network architecture parameters on the other half of the training dataset using an Adam optimizer with 0.9 momentum, and fixed learning rate of 3e-4. For training the derived network architectures from scratch, we adopt an SGD optimizer with 0.9 momentum, and initial learning rate of 0.01 with cosine decay for 600 epochs with 96 batch size.
To benchmark SACoD over SOTA imaging systems, we fix the number of masks to be six among all the settings, and then study their accuracy under different FLOPs and energy costs. We control the FLOPs of the SACoD and Gabor-mask systems by controlling X in Eq. equation 3 and that of the Co-train system by changing the width multiplier (Howard et al. 2017).
Results analysis. Fig. shows the trade-off between the accuracy and required hardware costs in terms of both FLOPs and energy cost for the SACoD and the two baseline lensless imaging systems on CIFAR-10/100. We can observe that SACoD consistently requires reduced FLOPs or energy cost while achieving a comparable or higher accuracy over the baselines. On CIFAR- 10, SACoD achieves 44. 1..1% % and 70.9% reduction in FLOPs, and 27.6% and 48.4% reduction in energy, while offering a +0.01% % and + +1.58% higher accuracy, compared with the Co-train and Gabor-mask baselines, respectively; On CIFAR-100, SACoD reduces the FLOPs by 62.9% and 64.1%, and en- ergy cost by 49.7% and 50.9%, while achieving a +0.71% and +7.92% higher accuracy. compared to the Co-train and Gabor-mask baselines, respectively. This set of experiments validate that the end-to-end optimization engine in SACoD indeed can lead to superior performance in both task performance and hardware efficiency.
Considering that the form factor or area is an- other influential design factor in lensless loT imaging systems, we evaluate SACoD over the baselines in terms of the trade-off between ac- curacy and area by controlling the number of masks in the optical layer, and summarize the results in Fig. 5 We can see that the pro- posed SACoD achieves the best accuracy- tradeoff's among all the designs under the same number of masks (and thus area) and the same model size. In particular, SACoD achieves 60.0% and 80.0% reduction in area while offer- ing 0.01% % and 1.05% higher accuracy, com- pared with the Co-train and Gabor-mask hase- lines, respectively.
96
SACOD (Propased)
95
Co-train
Accuracy (%)
Gabor
25
Area
94
60.0%
20
Area (mm
93
1.05%
80.0%
15
92
Impractical sensor area
10
91
90
2
4
5
8
9
10
Number of masks
Figure 5: No. of masks vs. accuracy and sen- sor/mask
CIFAR-10.
4.3 SACoD OVER SOTA IMAGING SYSTEMS ON IoT APPLICATIONS
Here benchmark SACoD the SOTA baselines two IoT applications (including FlatCam Face recognition (Tan et al. 2019) and Head-pose task (Gourier & Crowley] (004)) to evaluate its effectiveness on real-world ToT tasks. In this set of experiments, we further constrain FLOPs of the derived neural networks to see if SACoD is still applicable to extremely mnegyyconsttraied sce- narios. As shown in Fig. 6 we can see that again SACoD consistently outperforms the baselines under all settings in terms of accuracy-cost tradeoffs. Specifically, compared with the Co-train base- line, SACoD achieves 59.5% and 57.1% reduction in FI.OPs, 32.9% and 30. 1% reduction energy cost with +0.1 1.11% % and +0.07% higher accuracy, on the Flatcam Face and Head-pose tasks, re- spectively, Meanwhile, compared with the Gabor-mask bascline, SACoD shows better scalability to more energy-constrained scenarios. In particular, when the FLOPs or energy constraint is extremely low, SACoD achieves an 8.75% and 5.85% higher accuracy, under the same FLOPs/energy cost on the Flatcam Face and Head-pose pose tasks. These results show the effectiveness of SACoD extends to read-world IoT applications and the superior scalability of SACoD over SOTA IoT imaging systems.
4.4 SACOD OVER SOTA IMAGING SYSTEMS ON OTHER VISION TASKS
Considering the diverse applications of IoT devices, also evaluate SACoD on other vision tasks including one segmentation task and two unpaired amage-to-imaat translation tasks, which require more challenging tradeoff on -NN-powered intelligent 1oT devices. We show the quantitative comparison. visualization, and detailed experiment settings in the appendix. Note that segmentation tasks are commonly evaluated in terms of both FID (Heusel al. 2017) and image visualization as

---

Under review as a conference paper at ICLR 2021
100
100
25.5%
Accuracy 150)
17.15
Accuracy
25
Accuracy
Accurncy
-077%%
SAGOD (Poposed)
-6
+-00%
6.72%
80
80
50
150
17.5 20
FLOPs (x10*)
Energy (mj)
FLOPS (x10*)
Energy (mj)
FlatCam
Head-pose
Figure Accuracy vs. FLOPs/energy of SACoD over the baselines on IoT tasks.
Optimaed
Experimental
Optimined
Experimenti
Optimzed
Experimental
Posttive Components
Negative Components
Figure 7: Physical Masks
Figure 8: PSF comparison between optimized and experimental ones
Table I: Performance of SACoD the actual PhlatCam imaging system.
Datasets
Setting
Accuracy (%)
Gap (%)
Optimized
94.43
CIFARIO
Experimental
90.02
4.41
the former cannot always capture the image quality. From the results, SACoD requires the smallest FLOPs under all the six cases, and it shows that SACoD provides notably the best visualization effect among all the methods. particular, compared the Gabor-mask baseline, SACoD achieves 12.0% 50.1% reduction in FLOPs and +1.26 +20.27 improvement in FID, while providing notably better visualization effect; compared to the Co-train baseline, SACoD reduces the FLOPS by 92.56% 93.4% and offers notably better visualization effect, while achieving better FID on the zebra2horse dataset. Note that although the Co-train baseline achieves better FID than SACoD on the horse2zebra dataset, its visualization images (see Fig. 12 the appendix) suffer greatly from color shifts, distortion and thare-board effect, similar to those of the Gabor-mask baseline.
4.5 EVALUATION ON PHYSICAL FABRICATED MASKS
To evaluate the performance of SACoD the actual PhlatCam imaging system besides simulation, we fabricate the actual masks with the PSF from the optimized optical layer. We first visualize the fabricated masks and compare them with the optimized PSF Fig. 7 shows 6 fabricated masks setting on CIFAR-100 with a microscope image, in which the top row is positive masks and bottom ones are negative, and columns from left right represent RGB-channels respectively. Then we compare the experimental PSF generated by the fabricated masks with the optimized PSF in Fig From the comparison, we find experimental PSF basically keeps the original shape, although the brightness of some pixels is changed. We further compare the achieved accuracy based one simulated and fabricated masks on CIFAR-10. As shown in Table [ the accuracy gap is 4.41% % which is resulting from possible alignment issues and other experimental error sources in the mask fabrication process and has been shown by other optical computing systems, i.e., (Lin et al. 2018).
4.6 ABLATION STUDIES ON SACOD
Influence of mask flexibility. better understand SACoD's s superior performance, we perform ab- lation experiments where the optical mask is fixed at either or both of the search and training stages. The experiment setting the same as described Section 4.2 which the models are all under comparable model sizes (1.0 M), and the pre-trained optical mask is obtained by co-training with MobileNetV2 (Sandler et al., 2018). We explore four cases: (mask) fixed during both the search and training stages, fixed only during the training stage, fixed only during the search stage, and unfixed

---

Under review as a conference paper at ICLR 2021
Table 2: Ablation: whether to fix the optical mask during search and training using small models.
CIFAR-100
CIFAR-10
Search
fixed
unfixed
fixed
unfixed
fixed
unfixed
fixed
unfixed
Improv.
Improv.
Training
fixed
fixed
unfixed
unfixed
fixed
fixed
unfixed
unfixed
Acc (%)
67.45
67.69
68.03
69.64
+2.19
92.02
92.37
92.84
92.95
+0.93
during both (i.e., SACoD). From the results in Table 2 we can see that (1) fixing the mask during both the search and training stages causes the biggest accuracy drop (more than 0.93% CIFAR- 10 and CIFAR-100 datasets); and (2) fixing the mask at only one of the stages would also significantly impact the performance. This study verifies that the success of the co-design principle be cribed to two causes. First, with the mask included as hyperparameters, the optimization (during fine-tuning) can reach better minimum. Second and more importantly, the trainable mask as- sist finding a better architecture because the neural network backend can entrust more computation responsibilities to the 'costless' optical layer and thus can be made more efficient.
Effectiveness of the optical layer. To fur- ther explore the reason behind the success of SACoD, we compare the discriminative power of the features captured by the optical layers of SACoD and the Gabor-mass mask baseline. Specifi- cally, following (Suau et al. (020), we average the optical layer's ss activations over the output channels to obtain a vector and use the corre- sponding softmax value the feature distribu- tion for each input image. We then calculate the KL divergence between the feature distribution from different classes to see how discriminative the features are. Fig. visualizes the average KL divergence (over 100 random selected im- ages) between every two classes on the test dataset of CIFAR- 10. We can see that the feature dis- tribution difference of SACoD between different classes is notably and consistently larger than that of the Gabor-mast baseline, which further verifies that the optical layer of SACoD undertakes more computations to extract the discriminative information from the input, as to save the computations for the neural network backend.
SACoD
0.40
0.35
0.50
020
6:15
0.15
0.00
Figure 9: KL divergence the output distri- bution between different classes captured by the searched optical layer of SACoD and Gabor-mase on CIFAR- 10. where the x-axis and y-axis the class id, and the heatmap value denotes the mag- nitude of KL divergence.
Considering the potential defection when fabricating the PhlatCam sensors, we study the robustness of the SACoD, Co-train, and Gabor-mask systems when injecting noises into the masks. To simulate the noise effect, sample noises from standard normal distribution and add it to the masks; after that, we finetune the backend network for restoring the accuracy until convergence. By controlling the standard deviation (σ) of the standard normal distribution, i.c., the noises' dynamic range, evaluate the robustness the three lensless systems under different noise intensity and summarize the results. We can observe that (1) as the increases (i.e., noises' intensity increases), the restored accuracy of all the three lensless systems decreases while the accuracy of SACoD is consistently higher than that of both the Co-train and Gabor-mask baselines; and (2) the Co-train baseline is the most sensitive to the noises injected to the masks, and suffers from up to 2.71% accuracy loss, the Gabor-man mask baseline shows the least accuracy degradation (1%1 under noises yet offering the lowest overall accuracy, and our proposed SACoD achieves favorable tradeoff with the highest task accuracy and a medium level accuracy drops under noises. More details are provided in appendix.
5 CONCLUSION
We propose SACoD, a sensor algorithm co-design framework, to enable more energy-efficien CNN- powered loT devices based PhlatCam. A novel end-to-end co-search algorithm is presented to jointly optimize the coded mask of PhlatCam in the sensor and the neural network backend. Extensive experiments and ablation studies validate the superiority of the proposed SACoD in terms of both task performance and hardware efficiency as well as the optical layer effectiveness,, when evaluated over SOTA lensless imaging systems various tasks and datasets.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jesse K Adams, Vivek Boominathan, Benjamin W Avants, Daniel G Vercosa, Fan Ye, Richard G Baraniuk, Jacob Robinson, and Ashok Veeraraghavan. Single-frame 3d fluorescence mi- croscopy with ultraminiature lensless flatscope. Science advances, 3(12) 701 548, 2017.
Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng. and Laura Waller. Diffusercam: lensless single-exposure 3d imaging. Optica, 5(1):1-9, Jan 2018. doi: 10.166/OPTICA.5.0000000 URL http://wwwt wwwppoollliiiiooooooooooooooaoaa
abst ract. wttp::/cocnIcoooiaaaa5-111
Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard Bara- niuk. Flatcam: Thin, bare-sensor cameras using coded aperture and computation. 08 2015.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In Advances in neural information processing systems, pp- 5145-5153, 2018.
V. Boominathan, J. Adams, J. Robinson, and A. Veeraraghavan, Phlatcam: Designed phase-mask based thin lensless camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp- 2020.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint arXiv: /ea.i/0084..444 2017.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Thirty-Second AAAI conference on artificial intelligence 2018a.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. arXiv preprint arXiv:1812.00333 2018b.
Ezio Caroli, J. Stephen, G. Cocco, L. Natalucci, and A. Spizzichino. Coded aperture imaging in x- and gamma-ray astronomy. Space Science Reviews, 403, 1987. doi: 10.1007/ BF00171998.
Ayan Chakrabarti. Learning sensor multiplexing design through back- pack-propagation Advances in Neural Information Processing Systems, pp. 3081 3031-3,89, 2016.
Julie Chang and Gordon Wetzstein. Deep optics for monocular depth estimation and 3d object detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10193- 10202. 2019.
Huaijin Chen, Suren Jayasuriya, Jiyue Yang, Judy Stephen, Sriram Sivaramakrishman, Ashok Veeraraghavan, and Alyosha Molnar. Asp vision: Optically computing the first layer of convo- lutional neural networks using angle sensitive pixels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 903-912, 2016.
Liang-Chich Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1766.05887 2017.
Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, and Zhangyang Wang. Fasterseg: Searching for faster real-time semantic segmentation. arXiv preprint arXiv: /912. 19111 2019.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
R. H. Dicke. Scatter-Hole Cameras for X-Rays and Gamma Rays. 153:L101, August 1968. doi: 10.1086/180230.
Xinyu Gong, Shiyu Chang. Yifan Jiang, and Zhangyang Wang. Autogan: Neural architecture search for generative adversarial networks. Proceedings of the IEEE International Conference on Computer Vision pp. 3224-3234 2019.

---

Under review as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- 2014.
mation processing systems, pp.
Nicolas Gourier and James Crowley. Estimating face orientation from robust detection of salient facial structures. FG Net Workshop on Visual Observation of Deictic Gestures, 01 2004.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint ar 1503000300.,,31,, 2015.
Andrew G Howard, Mengiong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference computer vision, pp. 694-711. Springer, 2016.
Xing Lin, Yair Rivenson, Nezih T. Yardimci, Muhammed Veli. Yi Luo, Mona Jarrahi, and Aydogan Ozcan. All-optical machine learning using diffrrative deep neural networks, Science, (6466)))): 1004-1008, 2018 8. ISSN 0036-8075. doi: 10. 20.1126//cience.aatt/. aat8084. URL
sciencemag. http://em....rrreeoooeee///////e /ttp://eennn/666666///6//
Hanxiao Liu, Karen Simonyan, Oriol Vinyals, Chrisantha Fernando, and Koray Kavukcuoglu. Hier- archical representations for efficient architecture search. arXiv preprint arXiv:171..00436 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:1806.0900 2018.
Christopher A Metzler, Hayato Ikoma, Yifan Peng, and Gordon Wetzstein. Deep optics for single- shot bhigh-dynamic-range imaging. arXiv preprint 19X1v:1908.00662. 2019.
NVIDIA Inc.
https://wwi www.waviiia...mmm.mm com/enn-ue aut hww.c/uuussmmccceeaaeeeeecneeeee www.bbeddddddsttsseeeeeeeeeeeeee setts//sssssstemttttttttot///// accessed 2019-09-
NVIDIA Jetson TX2.
01.
Patrick Mochel and Mike Murphy. sysfs - The filesystem for exporting kernel ob-
jects. https: /wwww kernel. org/ doc/ www.meeettaaaooo//eeeeseeeeeeeee
txt
accessed 2019-11-21.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: / 800203368, 2018.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks, 2018.
Takeshi Shimano, Yusuke Nakamura, Kazuyuki Tajima, Mayu Sao, and Taku Hoshizawa. Lensless light-field imaging with fresnel zone aperture: quasi-coherent coding. Appl. Opt., 57(11)22841 2850, Apr 2018. doi: 10. 10.136//A0.57.0022 0028411. cfm?URI=a0-57-
URL
org abstracts.........
Han Shu, Yunhe Wang, Xu Jia, Kai Han, Hanting Chen, Chunjing Xu, Qi Tian, and Chang Xu. Co-evolutionary compression for unpaired image translation. In Proceedings of the IEEE Inter- national Conference Computer Vision, pp. 3235-3244 2019.
10

---

Under review as a conference paper at ICLR 2021
Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd. Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG), 1- 2018.
Xavier Suau, Nicholas Apostoloff, et al. Filter distillation for network compression. In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 3129-3138. IEEE, 2020.
Qilin Sun, Jian Zhang, Xiong Dun, Bernard Ghanem, Yifan Peng, and Wolfgang Heidrich. End- to-end learned, optically coded super-resolution spad camera. ACM Transactions Graphics (TOG), 39(2):1-14, 2020.
J. Tan, L. Niu, J. K. Adams, V. Boominathan, J. T. Robinson, R. G. Baraniuk, and A. Veeraraghavan. Face detection and verification using lensless cameras. IEEE Trans. Comput. Imag., 5(2):180- 194, June 2019.
Texas Instruments Inc. INA3221 Triple-Channll, High-Side Measurement, Shunt and Bus Voltage Monitor. http::/www. ti. com/pprodict///A3222 INA3222 accessed 2019-11-21.
Yue Wang, Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang- E2-train: Training state-of-the-art cnns with over 80% energy savings. In Advances in Neural Information Processing Systems, pp. 5139-5151, 2019.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019a.
Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraragha- van. gnneccamdd-learinng phase masks for passive single view depth estimation. In 2019 IEEE International Conference on Computational Photography (ICCP), pp. 1-12. IEEE, 2019b.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint arXiv: 10/2::/8/2..9929 2018
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE Interna- tional Conference on, 2017.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning, 2017.

---

Under review as a conference paper at ICLR 2021
A EXPERIMENT DETAILS FOR SEARCHING ON CIFAR-10/100
For the experiments on CIFAR- we adopt the same search space as FBNet (Wu et al. 20 (0)
except each group's 's stride settings, i.e. a set of blocks with the same number pf output channels, for adapting to the resolution of images on CIFAR-10/100 In particular. we follow the strides settings for MobileNetV2 for all the seven groups. In addition, following (Wu et al.) (019a). we apply gumbel softmax on each architecture parameter option's contribution weights to the supernet, where the initial temper- ature is and decays by 0.92 at the end of each epoch.
CIFAR- 10/100 as described in (Wang al. 2019), which is [1, 1, 2, 2, 2, 1]
B MORE ABLATION STUDIES FOR SACOD: ROBUSTNESS AGAINST NOISE
Considering the potential defection when fabri- cating the PhlatCam sensors, study the ro- bustness of the SACoD, Co-train, and Gabor- mask systems when injecting noises into the masks. To simulate the noise effect, we sample noises from standard normal distribution and add it to the masks; after that, we finetume the backend network for restoring the accuracy un- til convergence. By controlling the standard de- viation (or) of the standard normal distribution, i... the noises' dynamic range, we evaluate the robustness of the three lensless systems under different noise intensity and summarize the re- sults in Fig. T0] We can observe that (1) as the a increases (i.e., noises' intensity increases), the restored accuracy of all the three lensless sys- tems decreases while the accuracy of SACoD is consistently higher than that of both the Co- train and Gabor-mask baselines; and (2) the Co-
76
Accuracy (%)
74
72
SACoDiPrapaced)
70
Co-train
68
-1.19
0.0
0.3
0.5
0.7
0.9 1.0
Standard Deviation
Figure 10: The recovered accuracy after finetun- ing the SACoD, Co-train, and Gabor-mask sys- tems subject to different noise levels in the masks.
train baseline the most sensitive to the noises injected to the masks, and suffers from up to 2.71 % accuracy loss, the Gabor-mast maak baseline shows the least accuracy degradation (1.19%) under noises yet offering the lowest overall accuracy, and our proposed SACoD achieves a favorable tradcoff with the highest task accuracy and medium level accuracy drops under noises.
C EXPERIMENT RESULTS WHEN USING DIFFERENT MASK DESIGNS
As illustrated in Fig. 3 of the main content, three different mask designs can be adopted. We here provide more experiment results when using the three mask designs for SACoD. Specifically, we co-train MobileNetV2 with the three different mask designs on CIFAR-10/100 and find that the design with the masks in Fig. (a) (in the main content) achieves a 0.86% and 1.78% higher accuracy compared with that based the masks in Fig. (b). while offering a comparable accuracy 00.11% and -0.02%) with a 66.67% reduced rendered feature maps, on CIFAR- 10 and CIFAR- 100, respectively. This set of experiments motivates to adopt the masks' design based the in Fig. 3(a) for experiments shown in the main content.
D SACOD VS. LENS-BASED SYSTEMS
Here we provide experiments for benchmarking SACoD over lens-based systems under the same search space (Wu et al., 2019a) and datasets (CIFAR- 10--1-I-I--R-O)) For designing the lens-based systems, we release the FLOPs constraints for the first layer of the network, i.e., removing the optical layer and its associated constraints, and search for the optimal network given the search space and datasets. We find that under a slightly reduced FLOPs (154M FLOPs vs. 158M FLOPs), SACoD achieves 0.39% and 0.62% lower accuracy on CIFAR-10 10 and CIFAR- 100, respectively, while reducing the thickness of the imaging systems by 10x which makes possible to integrated into more loT applications. This of experiments show that proposed SACoD can offer similar
12

---

Under review as a conference paper at ICLR 2021
task performance and hardware efficiency compared to bass-based systems, while being able to shrink the thickness of the system by one order.
E MORE DETAILS AND RESULTS ON THE UNPAIRED IMAGE-TO-IMAGE TRANSLATION TASKS
In this subsection, we provide more details for the experiments on the unpaired image-to-image translation tasks, results of which are summarized in Table and Fig. of the main content.
E. THE SEARCH SPACE AND METHOD
For the search space, drew inspiration from existing works and build sequential supernet with hardware-friendly regular connection patterns where each candidate operators are sequentially con- nected. For the search method, we adopt SOTA differential neural architecture search (NAS) method (Liu et al., 2018) to search for both the operators and their widths to maximize efficiency improvements.
Search for the operators. The supernet consists of a total of nine operators from the following options:
The Conv layer: Conv 1xl, Conv 3x3;
The residual block (i.e., Conv 3x3 layers with skip connection);
The depthwise (DW) block (Conv x1 DW Conv 3x3 Conv 1x1 with skip connec- tion).
The above options cover popular and efficient building blocks in SOTA GAN generators (Zhu et al, 2017 Shu et al. 2019 Gong al. 2019). In particular, following SOTA differential NAS search method, we use the architecture parameter a;j to control the probability of choosing the j- operator in the i-th layer, and treat its softmax values as its contributing weight the supernet.
Search for the widths. Considering our target IoT applications, we also search for the widths (i.e., the number of output channels) of each operator for more efficient GAN models. Naively building set of independent convolutional kernels with different widths for each operator is not practical due to the required large memory consumption. We thus build superkernel with a maximal width, and then search for the expansion ratio & to make use of only subset of the superkernel's s input/output dimensions. In particular, we set Φ E [ 1 3 1] and use the architecture parameter % to control 2 6
the probability of choosing each expansion ratio for the i-th layer; Meanwhile, we apply gumbel softmax to approximate differentiable sampling for ø based on 7 as (Cai et al. 2018b). Therefore, during the searching process only one expansion ratio will be activated in one iteration, saving both the required memory and computational costs.
E.2 PROXY TASK
We search for efficient generator via distillation as formulated below:
min G.
Σ
((G(na y), Go(z:)) + AF(a, 1-1
(5)
N
Where d(.,)) is distance metric for the knowledge distillation (Hinton et al. 2015 between G (the searched generator) and Go (the original generator in CycleGAN (Zhu et al. [217]] ). which is the perceptual loss (Johnson et al. 2016) in our case, F is the computational budget determined by the network architecture (FLOPs our case), and a and 7 are the architecture parameters controlling the operator and width of each layer, respectively. Note that, the objective function in Eqq. 5 is independent of any trained discriminators since in practice the discriminator is often discarded after the generator is trained and thus not necessarily available when compressing the generator.
E.3 MORE VISUALIZATION RESULTS
Here we show more visualization results of SACoD as compared with the two lensless basclines on the unpaired image-to-image translation tasks (horse2zebra and zebra2horse) in Fig. 12 We can
13

---

Under review as a conference paper at ICLR 2021
Source image
Co-train 54.17 GFLOPs
Gabor-mask 7.15 GFLOPs
SACoD 3.57 GFLOPs
Source image
Co-train 54.17 GFLOPs
Gabor-mask 4.72 GFLOPs
SACoD 4.03 GFLOPs
Figure 11: More visualization examples the zebra2horse (row 1) and horse2zebra (row 2-3) tasks when using six masks. Columns from left to right: source images, and translation results of the Co-train, Gabor-mase and SACoD methods, respectively, where the required FLOPs of each method on each task are annotated above the images.
Table 3: SACoD over SOTA baselines on a segmentation task with the Cityscapes dataset.
masks
masks
masks
Setting
mIOU (%)
GFLOPs
mIOU (%)
GFLOPs
mIOU (%)
GFLOPs
Co-train
69.0
435.0
69.6
435.0
68.8
435.0
Segmentation
Gabor-mask
65.8
45.64
66.1
38.32
67.3
36.34
-Cityscapes
SACoD
36.17
70.4
71.6
SACoD Improv.
+0.80-+4.0
20.7% 9917%
+0.80 +4.3
12.4%
-92.3% %
+2,8
18.8% -93.2%
see that again the Gabor-mask baseline suffers greatly from color shift and distortion especially on the horse2zebra task, and the proposed SACoD provides better competitive visualization effects with high-contrast textures compared with the Co-train baseline while achieving 92.6% % reduction in FLOPs.
E.4 SACOD OVER SOTA IMAGING SYSTEMS ON OTHER VISION TASKS
Segmentation tasks. Real-time segmentation on resource-constrained platforms have growing de- mands many applications such as medical image segmentation, so we explore its combination with PhlatCam lensless imaging systems for scenarios with constrained imaging environments. To search for efficient segmentation networks, we adopt the search space and search method of SOTA NAS method (Chen et al.] 2019). For fair comparison, all the searching and training settings follow the original ones in (Chen et al. 2019). Specifically, the co-train baseline adopts the DeepLabV3 (Chen) al. 2017) model with ResNet50 (He et al. 2016) backbone. Table 3 with 2048x 1024 images show that under all the mask constraints SACoD achieves the highest mean Intersection Over Union (mIOU) while requiring the smallest FLOPs. Specifically, SACoD achieves 0.8%~4.3% higher mIOU and 12.4%~93.1 reduction in FLOPs over the Co-train and Gabor-mask baselines, respec- tively, under all the mask settings.
14

---

Under review as a conference paper at ICLR 2021
Source image
Co-train 54.17 GFLOPs
Gabor-mask 7.15 GFLOPs
SACoD 3.57 GFLOPs
Source image
Co-train 54.17 GFLOPs
Gabor-mask 4.72 GFLOPs
SACoD 4.03 GFLOPs
Figure 12: Visualizations on the zebra2horse (rows 1) and horse2zebra (row 2) tasks with six masks. Columns from left right: source image, translation results for the Co-train, Gabor-mask and SACoD methods, respectively. FLOPs of each method on each task are annotated above the images.
Table 4: SACoD over SOTA haselines on unpaired image-to-image translation tasks.
masks
masks
masks
Setting
Methods
FID
GFLOPs
FID
GFLOPs
FID
GFLOPs
Co-train
147.03
54.17
140.70
54.17
139.83
54.17
zebra2horse
Gabor-mask
137.79
6.89
141.11
5.04
145.87
7.15
SACoD
136.35
5.93
136.41
3.89
138.23
3,57
Co-train 66.82
54.17
61.21
54.17
68.26
54.17
horse2zebra
Gahor-mask
91.87
5.87
106.27
4.34
88.36
4.72
SACoD 89.80
3.70
86.00
3.82
87.10
4.03
Unpaired image-to-image translation tasks. To apply GAN (Goodfellow et al. 2014) to lensless imaging systems under constrained imaging environments, evaluate SACoD on unpaired image- to-image translation tasks (Zhu et al. 2017). one of the most popular applications GAN.
Experiment settings: since no prior NAS works search for image-to-image GANs, we design new search space (see details in the supplement) targeting extremely efficient GAN architectures, Fol- lowing the settings of CycleGAN (Zhu et al. 2017), we keep the stem and header structures in generator and search for both the operator type and channel width within the supernet which in- clude nine sequential operators. between the searched architecture and the pre-trained CycleGAN generator as the loss function. More search settings are in the supplement. The Co-train bascline system consists of an original CycleGAN following the optical layer.
addition, we use the knowledge distillation (Hinton et al.) 2015)
15


Reviews:
[{"review_id": "kbhy32PIto", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThis paper presents a method called SACoD to develop a more efficient CNN-powered Phlatcam. The proposed method optimizes both the PhlatCam sensor and the backend CNN model simultaneously.  That is, the coded mask in Phlatcam and neural network weights are regarded as learnable parameters. The coded mask (the optical layer) can be considered as a special convolution layer. As a result, it achieves energy saving, model compressing as well as good accuracy. Extensive experiments and ablation studies are presented to show the effectiveness of the method.\n\nOverall,  I think the paper is interesting and properly designed. I also believe that various experiments and analyses of this paper can influence future related studies. However, I hope many experiments using raw images obtained from the real-world lensless imaging system will be included.\n\nStrength\n\nThis paper is the first attempt to optimize both sensor and CNN-model at the same time. The derivation and formulation look interesting.\n\n\nComments and Weakness\n\nIn figure3, the input seems like a normal RGB image. Also, data in datasets such as CIFAR10 and Cityscape are common RGB images. Then, wasn't it applied to the real image obtained using the lensless imaging system? Even in Table 1, reported results were from common RGB images in CIFAR-10. In both optimized and experimental results, source images for testing are not real-world data from lensless imaging systems.\n\nIs there any comparative test of the three methods in Figure3? Is there a reason why they are mentioned? The reasons why design (a) was chosen are mentioned in the text, it is also better to show them experimentally.\n\nTo what level can the Flops and energy be lowered based on the same baseline CNN model, and is it enough to put it on the real-world IoT system?\n\nIt would be better to compare the proposed method with more than two baseline methods. For example, I wonder what happens when regular CNN without optical layers is used. Also, it would be good to have an experiment on various types of mask filters except for Gabor.\n\nAbout table2, What is the reason for the big difference in CIFAR-100 while there is not much increase in CIFAR-10?\n\nFont sizes in Figure4 and Figure 6 are too small to read.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TIPG2MxCR5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the practical application-level problem with joint optimization from front-end sensor to back-end CNN algorithms. Authors validate the proposed method's advantages with comprehensive experiments on different tasks, including image classification, face/pose detection, image segmentation, and image translation. Lastly, the authors claim to contribute its source codes to the research community upon acceptance. Overall, I think this research paper is well-written with no flaw identified; more importantly, I believe this investigation will substantially impact many real-world applications via pushing CNN to edge devices.\nOne suggestion is to specify the search space for NAS, as the search space is usually application dependent. As it is not specified in the paper, I assume you adopt the default search space, which contains only the 10 operations, and I wonder what is the best set of operations among them that best for the sensor data?", "extra_scores": {}, "extra_text": {}}, {"review_id": "LF5INkkF36", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\nSummary:\n \nSACoD presents a novel attempt to integrate the computational capabilities of a lensless imaging system, PhlatCam, with the search for the optimal convolutional neural network design for a given task. SACoD provides a framework which enables joint optimization of sensor and CNN resulting in IoT devices that achieve higher task accuracy’s with limited resource budgets of a typical IoT system. The authors present a new an optical layer design that enables above described features. Detailed experiments comparing SACoD sensor + CNN with other baseline models covering past papers, demonstrate the superiority of SACoD’s accuracy/efficiency curve over that of separately optimizing CNN arch or sensor/CNN joint-optimizations that do not vary network architecture. Additionally, ablation studies and results from measurements from actual phase masks fabricated help breakdown the accuracy/efficiency benefits of SACoD while analyzing the noise limitations of mask fabrication process.\n\n##########################################################################\nReasons for score: \n \nThe paper presents a sound theoretical description of the SACoD approach along with their optical layer design. The results presented through experiments clearly show the superiority of SACoD approach over other baselines that do not utilize the cost-free computational capabilities of the PhlatCam sensor. With the concerns raised in the Cons, sections answered I recommend that this paper be accepted.\n \n##########################################################################\nPros:\n\n1.\tSection 2 presents a good overview of the different approaches attempting sensor/CNN optimization to improve accuracy for given hardware resources. The proposed solution is unique in attempting to utilize the computation capability of PhlatCam imaging system.\n2.\tSection 3 presents a good theoretical description of the SACoD framework including the steps involved in its training to estimate the optimal mask, network arch and weights.\n3.\tIt is commendable that as noted at the end of Section 3, authors try out both differentiable and reinforcement learning (RL) based NAS approaches and achieve similar accuracies. Highlighting, the effectiveness of SACoD irrespective of the NAS approach utilized.\n4.\tSection 4 describes the detailed experiments that the authors used to assess the accuracy/efficiency benefits of SACoD over two other baseline approaches for 6 datasets and 4 tasks. Figures 4/5/6 are clear and help demonstrate the superior accuracy vs efficiency curves plotted for SACoD.\n5.\tAblation studies discussed in Section 4.6 on mask flexibility influence and optical layer effectiveness are useful to illustrate the importance of joint optimization of sensor and network as well as the effectiveness of SACoD optical layer.\n6.\tFigures showing a comparison of the performance of different approaches on vision tasks are helpful to show the superiority of SACoD.\n \n##########################################################################\nCons: \n \n1.\tSection 3, The optical sensing frontend subsection, describes how the object that is being imaged is at a distance d to the camera while trying to formulate the output of the masks in terms of a 2D convolution operation. Further in this subsection, the authors note that the mask is fixed at a distance d from the sensor. It would be useful to rephrase either of these sentences to clarify what is the distances of the object from the mask and the sensor.\n2.\tAuthors make an effort to describe the impact α on w* and m*, it is not obvious to me as to why the dependence changes for w* and m*. Perhaps an example illustrating the indirect influence of α on m* would be help clarify this.\n3.\tThe experiments in Section 4, present a range of datapoints for SACoD and other baselines for each task/benchmark. In my understanding, these points are obtained by tuning the accuracy vs efficiency tradeoff. How does the size of network change with these experiments? I worry that the efficiency of the GPU utilized for Flops and energy measurements might have biased the results somewhat against SACoD. Testing on a hardware platform with smaller on-chip memory and parallelism might be interesting exercise to show SACoDs superiority.\n4.\tSection 4.5 is commendable for confirming through measurements that the desired PSF can indeed be generated through fabricated masks. Further the ablation studies discussed in section 4.6 and appendix attempt to model the noise added due to fabrication process in the masks and the resultant accuracy loss. \n5.\tIts not clear that the magnitude of normal noise assumed by the authors is sufficiently capturing the fabrication noise measured. Considering that the accuracy drops reported in Table 1 and Figure 10 are for different benchmarks. It would be great if authors could use the same benchmark to model the measured 4% reduction in accuracy and then compare with other baselines. Cause for CIFAR 10, the SACoD approach does not have enough margin to dominate Gabor mask baseline despite a 4% accuracy loss. \n6.\tThis dependence on the noise during fabrication is a critical weakness of the SACoD approach. Since it relies on the computations carried out by these potentially noisy masks. The authors should consider developing a full-proof solution to this problem by utilizing noise model based training for their models such that the final network can be immune to the fabrication noises.\n\n##########################################################################\nQuestions during rebuttal period: \n\nKindly address the concerns noted in the Cons section\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "60SpRWrROA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#################################\n\nSummary:\n\nThe paper proposed to adopt differentiable network architecture search (DARTS) for the co-design of the sensor (a lensless camera) and the deep model for visual recognition tasks, so as to maximize the accuracy and minimize the energy consumption. The key idea is to include the sensor configuration, in this case the phase mask of a lensless camera modeled as 2D convolutions, as additional parameters in architecture search.  The proposed method was evaluated on simulated data for a number of vision tasks (image classification, face recognition and head pose estimation), as well as using fabricated masks on a real world camera. The results demonstrated significantly increase recognition performance given the same energy level.  \n\n#################################\n\nPros:\n* The high-level idea of using NAS for the co-design of the sensor and the deep model is quite exciting.\n* The experiments are extensive, including both several vision tasks in simulation and a classification task using real-world sensor implementation.\n \n#################################\n\nCons:\n* Sensor Design: Generality vs Specificity \n\nThe paper proposed to tailor a physical sensor and bundle it with a deep model for a target vision task. Once realized, it is unclear if the sensor is able to adapt to a different task or even a different model. Would the phase mask identified on one model / task generalize to different tasks/models? Generality is an important property of visual sensors (e.g., RGB / ToF cameras). The same camera can be used for different tasks (e.g., image classification, face recognition, etc). And the captured images can be examined by various models. It is probably not surprising that higher accuracy and lower energy can be achieved if a sensor is specifically designed for a single task and for a single type of model. The question is why do we need such a high specificity. It seems very limited even in the IoT setting. For example, what if the backend model needs to be updated. \n\n* Lack of modeling for fabrication error\n\nFabrication error is quite significant and is not modeled. For example, on CIFAR 10, fabrication error accounts for a over 4% drop in accuracy, canceling out most of the gains from the co-design (using a Gabor mask has 2-3% drop in accuracy in comparison to the co-design). Is it possible to model fabrication error and encode that into the loss function for NAS? For example, a good design should avoid those patterns that are likely to create issues in fabrication. \n\n#################################\n\nMinor comments:\n\nThe paper has a strong vision flavor and might be a better fit to vision / computational photography conferences. \n\n#################################\n\nJustification for score:\n\nOverall, this is a paper with a very interesting idea and solid experiments. Yet, the problem setting is limited and the modeling has some issues. I am more positive about this paper after reading the authors' response. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper attempts to jointly search for the sensor and the neural network architecture. More specifically, the proposed approach jointly optimizes the parameters governing the PhlatCam sensor and the backend CNN model. In terms of the approach, the paper follows a well known DARTS formulation for the differentiable architecture search. A very straightforward solution was proposed for the problem.

Although all the reviewers place that the paper is marginally above the acceptance threshold, none of them strongly support the paper and the reviewers point out that the paper is limited in terms of the setting and data. The problem formulation of the paper itself is interesting, but the AC agrees with the reviewers that the paper is limited and lacks enough technical contributions to warrant the acceptance to ICLR.

Decision: Reject