SACoD: Sensor Algorithm Co-Design Towards Efficient CNN-powered Intelligent PhlatCam

1 INTRODUCTION . Recent CNN breakthroughs trigger a growing demand for intelligent IoT devices , such as wearables and biology devices ( e.g. , swallowed endoscopes ) . However , two major challenges are hampering more extensive applications of CNN-powered IoT devices . First , some applications , especially medicine- and biology-related ones , impose strict requirements on the form factor , especially the thickness , which are often too stringent for existing lens-based imaging systems . Second , powerful CNNs require considerable hardware costs , whereas IoT devices only have limited resources . For the first challenge , lensless imaging systems ( Asif et al. , 2015 ; Shimano et al. , 2018 ; Adams et al. , 2017 ; Antipa et al. , 2018 ; Boominathan et al. , 2020 ) have emerged as a promising rescue . For example , PhlatCam ( Boominathan et al. , 2020 ) replaces the focal lenses with a set of phase masks , which encodes the incoming light instead of directly focusing it . The encoded information can be either computationally decoded to reconstruct the images or processed specifically for different applications . Such lensless imaging systems can be made much smaller and thinner , because the phase masks are smaller than the focal lens , and they can be placed much closer to the sensors and fabricated with much lower costs . For the second challenge , many recent works focus on designing CNNs with improved hardware efficiency , i.e. , by applying generic neural architecture search ( NAS ) to find efficient CNNs . As such , a naive way to address the two aforementioned challenges simultaneously is to introduce lensless cameras as the signal acquisition frontend and then apply NAS to optimize the backend CNN . However , such approaches would result in disjoint optimization that can be far from optimal . A generic NAS would treat the camera as given , and only optimize the CNN . Likewise , existing phase mask designs for lensless cameras treat the CNNs as given , and only optimize the masks . Such disjoint optimization fails to ( 1 ) take advantage of the masks ’ potential computational capacity , with which the NAS optimization can be fundamentally improved , and ( 2 ) perform end2end optimization . It is shown in ( Boominathan et al. , 2020 ) that , under some assumptions , the phase masks in PhlatCam essentially perform 2D convolutions on the incoming lights , and the convolution kernel is encoded in the masks . Moreover , unlike other convolutional layers , the phase masks ’ convolutions are almost free – they do not consume additional energy , computation power , or storage , regardless of what value each mask takes . Therefore , we aim to incorporate the phase mask design into NAS to enable an end2end optimization of the sensing-processing pipeline , while exempting a portion of the pipeline from the efficiency penalties . Such co-designs are expected to achieve better tradeoffs between accuracy and efficiency . To this end , we propose a Sensor Algorithm Co-Design ( SACoD ) framework to enable more energyefficient CNN-powered IoT devices . While SACoD is general and can be applied to different sensing and intelligent processing systems , it is developed and evaluated in the context of PhlatCam ( Boominathan et al. , 2020 ) based imaging systems . Our main contributions are : • We propose SACoD , a novel co-design framework that jointly optimizes the sensor and neural networks to enable more energy-efficient CNN-powered IoT devices . To our best knowledge , SACoD is the first to propose sensor algorithm co-design for CNN inferences . • We develop an effective design of the optical layer to ( 1 ) exploit its potential computation capability and ( 2 ) enable co-search of the optical layer and backend algorithm . We then characterize the trade-off between accuracy and the required area of the corresponding imaging systems to demonstrate its effectiveness under practical size constraints . • Extensive experiments and ablation studies validate that the proposed SACoD consistently achieves reduced hardware costs/area while offering a comparable or even better task accuracy , when evaluated over two SOTA lensless imaging systems on four tasks and six datasets . And part of the experiments are further evaluated with fabricated masks to validate SACoD ’ s effectiveness in the physical measurement besides simulation . 2 RELATED WORKS . Neural architecture search . Recently NAS has attracted increasing attention . It eliminates the handcrafting process and automatically designs neural architectures . Existing NAS techniques can be divided into three categories , evolution-based , reinforcement-learning ( RL ) -based , and one-shot NAS . As the computational overheads of evolution- or RL-based approaches can be unacceptably high , many techniques ( Brock et al. , 2017 ; Cai et al. , 2018a ; Liu et al. , 2017 ; 2018 ; Pham et al. , 2018 ; Xie et al. , 2018 ) have been proposed to reduce the searching cost , among which differentiable architecture search ( DARTS ) has gained intensive interests . While being conceptually general , SACoD in this paper adopts the DARTS method , where a super-network is optimized during search and the strongest sub-network is preserved and then retrained . Lensless imaging systems . To eliminate the size or thickness burden caused by the lens , various lensless imaging systems have been developed . While lensless imaging systems have been widely used for capturing X-ray and gamma-ray ( Dicke , 1968 ; Caroli et al. , 1987 ) , it is still in an exploring stage for visible spectrum uses ( Asif et al. , 2015 ; Shimano et al. , 2018 ; Antipa et al. , 2018 ; Boom- inathan et al. , 2020 ) . In general , lensless imaging systems capture the scene either directly on the sensor or after being modulated by a mask element . In this paper , we focus on a specific lensless imaging system based on phase masks called PhlatCam ( Boominathan et al. , 2020 ) , which is a general-purpose framework to create phase masks that can achieve desired sharp pointspread-functions ( PSFs ) . A phase mask modulates the phase of incident lights , and allows most of the light to pass through , providing a high signal-to-noise ratio . Hence , they are desirable for low light scenarios and photon- limited imaging . Fig . 2 shows a fabricated phase mask , which is essentially a transparent material with different thicknesses at different locations . Based on this lensless imaging system , we develop our SACoD framework , aiming to enable more energy-efficient CNN-powered IoT devices . Sensor-algorithm co-training . There have recently been some attempts that try to jointly optimize the sensor parameters and the neural network backend . For lens-based image systems , novel lens designs are introduced and trained concurrently with the neural network backend to jointly optimize for image reconstruction ( Sitzmann et al. , 2018 ) , depth estimation ( Chang & Wetzstein , 2019 ) , and high-dynamic-range imaging ( Metzler et al. , 2019 ) . Similar approaches have been applied to other imaging systems as well , including cameras with color multiplexing ( Chakrabarti , 2016 ) , PhaseCam3D ( Wu et al. , 2019b ) , and Single Photon Avalanche Photodiodes cameras ( Sun et al. , 2020 ) . Yet these methods still consider the neural network architecture as fixed , and do not explore the potential of applying the co-design principle to the neural architecture design . 3 THE PROPOSED SACOD FRAMEWORK . This section presents the SACoD framework . We will first outline the framework and explain why the optical layer can be considered as a convolutional layer , and then introduce how we implement SACoD ’ s optimization algorithm . Finally , we describe our developed optical layer design . Framework setup . The SACoD framework shown in Fig . 1 consists of two modules , an optical sensing frontend and a neural network backend . The coded mask of PhlatCam in the sensor and the backend are jointly optimized using a SOTA differential NAS algorithm ( Liu et al. , 2018 ) , where the coded masks , together with the neural network weights , are regarded as network parameters . Specifically , the first module , i.e . the optical sensing frontend , is denoted as O ( ⋅ ; m ) , where m = m ( x , y ) denotes the phase mask values . The optical layer is based on the PhlatCam system ( Boominathan et al. , 2020 ) . It receives the light signal from the object in front of the camera , processes the signal using the phase masks , and generates the sensor output . The second module , i.e . the neural network backend , is denoted as F ( ⋅ ; w , α ) , where w represents weights of the neural network , and α parameterizes the architecture . The neural network backend receives the sensor signal and produces an output for the intended applications . Formally , denote the light signal as I ( x , y ) ∈ RH×W×3 , where x and y are coordinate indices and H and W represent the height and width of the range of light the camera can receive . The light signal contains RGB channel , and hence the last dimension is 3 . Denote the signal received at sensor as Z ( x , y ) ∈ RH ′×W ′×N , where H ′ and W ′ represent the height and width , and N represent the number of channels . Denote Y as the final output of the neural network backend . Then we have Z = O ( I ; m ) , Y = F ( Z ; w , α ) . ( 1 ) The following subsections will introduce the form of O ( ⋅ ; m ) and how to determine m , w , and α . The optical sensing frontend . Assuming that the light signal I ( x , y ) comes from an object whose distance to the camera is d , and that the depth of the object is relatively small , it can be shown ( Boominathan et al. , 2020 ) that O ( ⋅ ; m ) takes the following convolutional form : Z ( x , y ) = O ( I ; m ) = p ( x , y ; m , d ) ∗ I ( x , y ) , ( 2 ) where ∗ denotes 2D convolution , p ( x , y ; m , z ) is called the point spread function of the phase mask , which is determined by the phase mask m ( x , y ) and the distance d. Once we optimize the PSF , the phase masks are designed for the PSF and a chosen d. The fabricated mask then produces the PSF at the given d. For the fabricated system shown in 4.5 , d is set to be 2 mm for making our system much thinner than conventional cameras ( thickness ranges between 7-20 mm ) . The mask is fixed at distance d to the sensor during operation , and thus the convolution property will continue to hold . According to Eq . equation 2 , the optical layer can be regarded as a special convolutional layer . Note that one phase mask can only perform a single-channel convolution with a positive kernel . It takes two phase masks to implement a single-channel convolution with a real-valued kernel , where one implements the positive part of the kernel and the other implements the negative part . Therefore , in order to construct a convolutional layer with three channels and real-valued kernels , we need six masks in the imaging system . Also , the input light has three color channels ( R , G , and B ) , and each phase mask operates on all the color channels . Therefore , a three-channel convolution will produce a total of nine feature maps ( FMs ) . We propose three different designs for using the rendered FMs , as shown in Fig . 3 . Specifically , the design in Fig . 3 ( a ) accumulates the FMs across the same color and outputs a 3-channel FM , which is still in an RGB shape ; the design in Fig . 3 ( b ) accumulates the FMs from one mask across three colors and outputs a 2-channel FM ; and the design in Fig . 3 ( c ) simply concatenates all the FMs from different colors . These three different designs extract different amounts or types of information from the scene which are then passed to the following neural network . Experiments under various settings show that SACoD based on Fig . 3 ( a ) achieves higher accuracies than those based on Fig . 3 ( b ) , and similar accuracies as the designs based on Fig . 3 ( c ) . We conjecture the reason is that design ( a ) applies independent transformations on the RGB channels to maintain the original channel-wise discriminative information ; and maintaining all the information as design ( c ) does not contribute to higher accuracy than design ( a ) , since the accumulated information across the same color has provided sufficient information for the following processing under the constrained number of masks . So in our experiments , we adopt the optical layer design ( a ) . The SACoD formulation and algorithm . This subsection introduces the formulation and optimization of SACoD which aims to simultaneously optimize the phase mask m , and the neural network ’ s architecture α , and the neural network ’ s weights w. Formally , SACoD aims to solve : min α Lval ( m∗ ( α ) , w∗ ( α ) , α ) + λLe ( α ) , ( 3 ) m ∗ ( α ) , w∗ ( α ) = argmin { m , w } Ltr ( m , w , α ) . ( 4 ) Ltr and Lval are task-specific performance losses evaluated on the training and validation set , respectively , Le is the efficiency loss , e.g . model size , computational cost , or energy consumption , and λ is the tuning parameter trading-off the accuracy and efficiency . Following the same parameterization scheme in DARTS ( Liu et al. , 2018 ) , α denotes the weights of different candidate operations . There are two major modifications compared to the original DARTS framework . The first difference is that the efficiency loss Le , measured by the sum of each layer ’ s FLOPs weighted by the network parameter α , is introduced . More importantly , the second and major difference is that the phase maskm is optimized jointly in the framework . It is worth pointing out that although mathematically similar , m∗ and w∗ have different degrees of dependencies on α. w∗ is directly impacted by α because α governs which subset of the w is ultimately used . m∗ is only indirectly influenced by α . Therefore , incorporating m will significantly improve the tradeoff between performance and model complexity . Besides , SACoD is naturally compatible with other NAS methods . We adopt differential NAS for the fast generation of the optical mask and network . When using other NAS methods , e.g. , reinforcement learning-based NAS ( Zoph & Le , 2017 ) , we still observe similar system performance ( within 0.3 % accuracy on CIFAR-100 ) , but the search time increases to 8 GPU-days from 0.5 GPU-days . The whole co-design process can be divided into two stages , a searching stage and a training stage . In the searching stage , we apply the alternate gradient descent of Eqs . equation 3 and equation 4 to search for the optimal network architecture α∗ . In the training stage , the optimal mask and weights are determined by optimizing Eq . equation 4 conditioning on the optimal network architecture α∗ .

Summary: The paper proposed to adopt differentiable network architecture search (DARTS) for the co-design of the sensor (a lensless camera) and the deep model for visual recognition tasks, so as to maximize the accuracy and minimize the energy consumption. The key idea is to include the sensor configuration, in this case the phase mask of a lensless camera modeled as 2D convolutions, as additional parameters in architecture search.  The proposed method was evaluated on simulated data for a number of vision tasks (image classification, face recognition and head pose estimation), as well as using fabricated masks on a real world camera. The results demonstrated significantly increase recognition performance given the same energy level.