SACoD: Sensor Algorithm Co-Design Towards Efficient CNN-powered Intelligent PhlatCam

1 INTRODUCTION . Recent CNN breakthroughs trigger a growing demand for intelligent IoT devices , such as wearables and biology devices ( e.g. , swallowed endoscopes ) . However , two major challenges are hampering more extensive applications of CNN-powered IoT devices . First , some applications , especially medicine- and biology-related ones , impose strict requirements on the form factor , especially the thickness , which are often too stringent for existing lens-based imaging systems . Second , powerful CNNs require considerable hardware costs , whereas IoT devices only have limited resources . For the first challenge , lensless imaging systems ( Asif et al. , 2015 ; Shimano et al. , 2018 ; Adams et al. , 2017 ; Antipa et al. , 2018 ; Boominathan et al. , 2020 ) have emerged as a promising rescue . For example , PhlatCam ( Boominathan et al. , 2020 ) replaces the focal lenses with a set of phase masks , which encodes the incoming light instead of directly focusing it . The encoded information can be either computationally decoded to reconstruct the images or processed specifically for different applications . Such lensless imaging systems can be made much smaller and thinner , because the phase masks are smaller than the focal lens , and they can be placed much closer to the sensors and fabricated with much lower costs . For the second challenge , many recent works focus on designing CNNs with improved hardware efficiency , i.e. , by applying generic neural architecture search ( NAS ) to find efficient CNNs . As such , a naive way to address the two aforementioned challenges simultaneously is to introduce lensless cameras as the signal acquisition frontend and then apply NAS to optimize the backend CNN . However , such approaches would result in disjoint optimization that can be far from optimal . A generic NAS would treat the camera as given , and only optimize the CNN . Likewise , existing phase mask designs for lensless cameras treat the CNNs as given , and only optimize the masks . Such disjoint optimization fails to ( 1 ) take advantage of the masks ’ potential computational capacity , with which the NAS optimization can be fundamentally improved , and ( 2 ) perform end2end optimization . It is shown in ( Boominathan et al. , 2020 ) that , under some assumptions , the phase masks in PhlatCam essentially perform 2D convolutions on the incoming lights , and the convolution kernel is encoded in the masks . Moreover , unlike other convolutional layers , the phase masks ’ convolutions are almost free – they do not consume additional energy , computation power , or storage , regardless of what value each mask takes . Therefore , we aim to incorporate the phase mask design into NAS to enable an end2end optimization of the sensing-processing pipeline , while exempting a portion of the pipeline from the efficiency penalties . Such co-designs are expected to achieve better tradeoffs between accuracy and efficiency . To this end , we propose a Sensor Algorithm Co-Design ( SACoD ) framework to enable more energyefficient CNN-powered IoT devices . While SACoD is general and can be applied to different sensing and intelligent processing systems , it is developed and evaluated in the context of PhlatCam ( Boominathan et al. , 2020 ) based imaging systems . Our main contributions are : • We propose SACoD , a novel co-design framework that jointly optimizes the sensor and neural networks to enable more energy-efficient CNN-powered IoT devices . To our best knowledge , SACoD is the first to propose sensor algorithm co-design for CNN inferences . • We develop an effective design of the optical layer to ( 1 ) exploit its potential computation capability and ( 2 ) enable co-search of the optical layer and backend algorithm . We then characterize the trade-off between accuracy and the required area of the corresponding imaging systems to demonstrate its effectiveness under practical size constraints . • Extensive experiments and ablation studies validate that the proposed SACoD consistently achieves reduced hardware costs/area while offering a comparable or even better task accuracy , when evaluated over two SOTA lensless imaging systems on four tasks and six datasets . And part of the experiments are further evaluated with fabricated masks to validate SACoD ’ s effectiveness in the physical measurement besides simulation . 2 RELATED WORKS . Neural architecture search . Recently NAS has attracted increasing attention . It eliminates the handcrafting process and automatically designs neural architectures . Existing NAS techniques can be divided into three categories , evolution-based , reinforcement-learning ( RL ) -based , and one-shot NAS . As the computational overheads of evolution- or RL-based approaches can be unacceptably high , many techniques ( Brock et al. , 2017 ; Cai et al. , 2018a ; Liu et al. , 2017 ; 2018 ; Pham et al. , 2018 ; Xie et al. , 2018 ) have been proposed to reduce the searching cost , among which differentiable architecture search ( DARTS ) has gained intensive interests . While being conceptually general , SACoD in this paper adopts the DARTS method , where a super-network is optimized during search and the strongest sub-network is preserved and then retrained . Lensless imaging systems . To eliminate the size or thickness burden caused by the lens , various lensless imaging systems have been developed . While lensless imaging systems have been widely used for capturing X-ray and gamma-ray ( Dicke , 1968 ; Caroli et al. , 1987 ) , it is still in an exploring stage for visible spectrum uses ( Asif et al. , 2015 ; Shimano et al. , 2018 ; Antipa et al. , 2018 ; Boom- inathan et al. , 2020 ) . In general , lensless imaging systems capture the scene either directly on the sensor or after being modulated by a mask element . In this paper , we focus on a specific lensless imaging system based on phase masks called PhlatCam ( Boominathan et al. , 2020 ) , which is a general-purpose framework to create phase masks that can achieve desired sharp pointspread-functions ( PSFs ) . A phase mask modulates the phase of incident lights , and allows most of the light to pass through , providing a high signal-to-noise ratio . Hence , they are desirable for low light scenarios and photon- limited imaging . Fig . 2 shows a fabricated phase mask , which is essentially a transparent material with different thicknesses at different locations . Based on this lensless imaging system , we develop our SACoD framework , aiming to enable more energy-efficient CNN-powered IoT devices . Sensor-algorithm co-training . There have recently been some attempts that try to jointly optimize the sensor parameters and the neural network backend . For lens-based image systems , novel lens designs are introduced and trained concurrently with the neural network backend to jointly optimize for image reconstruction ( Sitzmann et al. , 2018 ) , depth estimation ( Chang & Wetzstein , 2019 ) , and high-dynamic-range imaging ( Metzler et al. , 2019 ) . Similar approaches have been applied to other imaging systems as well , including cameras with color multiplexing ( Chakrabarti , 2016 ) , PhaseCam3D ( Wu et al. , 2019b ) , and Single Photon Avalanche Photodiodes cameras ( Sun et al. , 2020 ) . Yet these methods still consider the neural network architecture as fixed , and do not explore the potential of applying the co-design principle to the neural architecture design . 3 THE PROPOSED SACOD FRAMEWORK . This section presents the SACoD framework . We will first outline the framework and explain why the optical layer can be considered as a convolutional layer , and then introduce how we implement SACoD ’ s optimization algorithm . Finally , we describe our developed optical layer design . Framework setup . The SACoD framework shown in Fig . 1 consists of two modules , an optical sensing frontend and a neural network backend . The coded mask of PhlatCam in the sensor and the backend are jointly optimized using a SOTA differential NAS algorithm ( Liu et al. , 2018 ) , where the coded masks , together with the neural network weights , are regarded as network parameters . Specifically , the first module , i.e . the optical sensing frontend , is denoted as O ( ⋅ ; m ) , where m = m ( x , y ) denotes the phase mask values . The optical layer is based on the PhlatCam system ( Boominathan et al. , 2020 ) . It receives the light signal from the object in front of the camera , processes the signal using the phase masks , and generates the sensor output . The second module , i.e . the neural network backend , is denoted as F ( ⋅ ; w , α ) , where w represents weights of the neural network , and α parameterizes the architecture . The neural network backend receives the sensor signal and produces an output for the intended applications . Formally , denote the light signal as I ( x , y ) ∈ RH×W×3 , where x and y are coordinate indices and H and W represent the height and width of the range of light the camera can receive . The light signal contains RGB channel , and hence the last dimension is 3 . Denote the signal received at sensor as Z ( x , y ) ∈ RH ′×W ′×N , where H ′ and W ′ represent the height and width , and N represent the number of channels . Denote Y as the final output of the neural network backend . Then we have Z = O ( I ; m ) , Y = F ( Z ; w , α ) . ( 1 ) The following subsections will introduce the form of O ( ⋅ ; m ) and how to determine m , w , and α . The optical sensing frontend . Assuming that the light signal I ( x , y ) comes from an object whose distance to the camera is d , and that the depth of the object is relatively small , it can be shown ( Boominathan et al. , 2020 ) that O ( ⋅ ; m ) takes the following convolutional form : Z ( x , y ) = O ( I ; m ) = p ( x , y ; m , d ) ∗ I ( x , y ) , ( 2 ) where ∗ denotes 2D convolution , p ( x , y ; m , z ) is called the point spread function of the phase mask , which is determined by the phase mask m ( x , y ) and the distance d. Once we optimize the PSF , the phase masks are designed for the PSF and a chosen d. The fabricated mask then produces the PSF at the given d. For the fabricated system shown in 4.5 , d is set to be 2 mm for making our system much thinner than conventional cameras ( thickness ranges between 7-20 mm ) . The mask is fixed at distance d to the sensor during operation , and thus the convolution property will continue to hold . According to Eq . equation 2 , the optical layer can be regarded as a special convolutional layer . Note that one phase mask can only perform a single-channel convolution with a positive kernel . It takes two phase masks to implement a single-channel convolution with a real-valued kernel , where one implements the positive part of the kernel and the other implements the negative part . Therefore , in order to construct a convolutional layer with three channels and real-valued kernels , we need six masks in the imaging system . Also , the input light has three color channels ( R , G , and B ) , and each phase mask operates on all the color channels . Therefore , a three-channel convolution will produce a total of nine feature maps ( FMs ) . We propose three different designs for using the rendered FMs , as shown in Fig . 3 . Specifically , the design in Fig . 3 ( a ) accumulates the FMs across the same color and outputs a 3-channel FM , which is still in an RGB shape ; the design in Fig . 3 ( b ) accumulates the FMs from one mask across three colors and outputs a 2-channel FM ; and the design in Fig . 3 ( c ) simply concatenates all the FMs from different colors . These three different designs extract different amounts or types of information from the scene which are then passed to the following neural network . Experiments under various settings show that SACoD based on Fig . 3 ( a ) achieves higher accuracies than those based on Fig . 3 ( b ) , and similar accuracies as the designs based on Fig . 3 ( c ) . We conjecture the reason is that design ( a ) applies independent transformations on the RGB channels to maintain the original channel-wise discriminative information ; and maintaining all the information as design ( c ) does not contribute to higher accuracy than design ( a ) , since the accumulated information across the same color has provided sufficient information for the following processing under the constrained number of masks . So in our experiments , we adopt the optical layer design ( a ) . The SACoD formulation and algorithm . This subsection introduces the formulation and optimization of SACoD which aims to simultaneously optimize the phase mask m , and the neural network ’ s architecture α , and the neural network ’ s weights w. Formally , SACoD aims to solve : min α Lval ( m∗ ( α ) , w∗ ( α ) , α ) + λLe ( α ) , ( 3 ) m ∗ ( α ) , w∗ ( α ) = argmin { m , w } Ltr ( m , w , α ) . ( 4 ) Ltr and Lval are task-specific performance losses evaluated on the training and validation set , respectively , Le is the efficiency loss , e.g . model size , computational cost , or energy consumption , and λ is the tuning parameter trading-off the accuracy and efficiency . Following the same parameterization scheme in DARTS ( Liu et al. , 2018 ) , α denotes the weights of different candidate operations . There are two major modifications compared to the original DARTS framework . The first difference is that the efficiency loss Le , measured by the sum of each layer ’ s FLOPs weighted by the network parameter α , is introduced . More importantly , the second and major difference is that the phase maskm is optimized jointly in the framework . It is worth pointing out that although mathematically similar , m∗ and w∗ have different degrees of dependencies on α. w∗ is directly impacted by α because α governs which subset of the w is ultimately used . m∗ is only indirectly influenced by α . Therefore , incorporating m will significantly improve the tradeoff between performance and model complexity . Besides , SACoD is naturally compatible with other NAS methods . We adopt differential NAS for the fast generation of the optical mask and network . When using other NAS methods , e.g. , reinforcement learning-based NAS ( Zoph & Le , 2017 ) , we still observe similar system performance ( within 0.3 % accuracy on CIFAR-100 ) , but the search time increases to 8 GPU-days from 0.5 GPU-days . The whole co-design process can be divided into two stages , a searching stage and a training stage . In the searching stage , we apply the alternate gradient descent of Eqs . equation 3 and equation 4 to search for the optimal network architecture α∗ . In the training stage , the optimal mask and weights are determined by optimizing Eq . equation 4 conditioning on the optimal network architecture α∗ .

Summary: SACoD presents a novel attempt to integrate the computational capabilities of a lensless imaging system, PhlatCam, with the search for the optimal convolutional neural network design for a given task. SACoD provides a framework which enables joint optimization of sensor and CNN resulting in IoT devices that achieve higher task accuracy’s with limited resource budgets of a typical IoT system. The authors present a new an optical layer design that enables above described features. Detailed experiments comparing SACoD sensor + CNN with other baseline models covering past papers, demonstrate the superiority of SACoD’s accuracy/efficiency curve over that of separately optimizing CNN arch or sensor/CNN joint-optimizations that do not vary network architecture. Additionally, ablation studies and results from measurements from actual phase masks fabricated help breakdown the accuracy/efficiency benefits of SACoD while analyzing the noise limitations of mask fabrication process.