ItNet: iterative neural networks for fast and efficient anytime prediction

1 INTRODUCTION . For massively-parallel hardware accelerators ( Schemmel et al. , 2010 ; Merolla et al. , 2014 ; Yao et al. , 2020 ; Graphcore , 2020a ) , every neuron and synapse in the network model has its physical counterpart on the hardware system . Usually , by design , memory and computation is not separated anymore , but neuron activations are computed next to the memory , i.e . the parameters , and fully in parallel . This is in contrast to the rather sequential data processing of CPUs and GPUs , for which the computation of a network model is tiled and the same arithmetic unit is re-used multiple times for different neurons . Since the computation is performed fully in parallel and in memory , the throughput of massively-parallel accelerators is usually much higher than for CPUs and GPUs . This can be attributed to the fact that the latency and power consumption for accessing local memory , like for in-memory computing , are much lower than for computations on CPUs and GPUs that require the frequent access to non-local memory like DRAM ( Sze et al. , 2017 ) . However , the network graph has to fit into the memory of the massively-parallel hardware accelerators to allow for maximal throughput . If the network graph exceeds the available memory , in principle , the hardware has to be re-configured at high frequency as it is the case for CPUs and GPUs and the throughput would be substantially reduced . Mixed-signal massively-parallel hardware systems usually operate on shorter time scales than digital ones , e.g . compare Schemmel et al . ( 2010 ) and Yao et al . ( 2020 ) to Merolla et al . ( 2014 ) and Graphcore ( 2020a ) , and would allow for even higher throughputs . In order to achieve neural networks with tiny computational graphs , in which nodes are operations and edges are activations , we heavily re-use a single building block of the network ( see the iterative block in Figure 1a ) . Not only the structure of computations , i.e . the type of network layers including their dimensions and connectivity , is identical for each iteration of this building block , but also the weights are shared between iterations . In the computational graphs of these so-called iterative neural networks ( ItNet ) , the re-used building blocks with shared weights can be represented by nodes with self-loops . Compared to conventional feed-forward networks , loops simplify the graph by reducing the number of unique nodes and , consequently , its computational footprint . However , the restriction of sharing weights usually decreases the number of free parameters and , hence , the accuracy of networks . To isolate and quantify this effect we compare networks with weight sharing to networks , for which the parameters of the building blocks are chosen to be independent between iterations of the building block . In contrast to the above proposal , conventional deep neural networks for image processing usually do not share weights and have no ( e.g . Huang et al. , 2017 ) or few ( e.g . one building block for each scale like by Greff et al. , 2017 ) layers of identical structure . Liao & Poggio ( 2016 ) share weights between re-used building blocks , but use multiple unique building blocks . To improve the training of networks , which contain loops in their graphs , and to reduce the latency of networks during inference we use multiple intermediate outputs . Multi-output networks that heavily re-use intermediate activations are beneficial for a wide range of applications , especially in the mobile domain . In an online manner , they allow to trade off latency versus accuracy with barely any overhead ( e.g . Huang et al. , 2018 ) . From an application point of view , the benefit of this trade-off can be best described in the following two scenarios ( Huang et al. , 2018 ) : In the so-called anytime prediction scenario , the prediction of a network is progressively updated , whereas the first output defines the initial latency of the network . In a second scenario , a limited computational budget can be unevenly distributed over a set of samples with different `` difficulties '' in order to increase the average accuracy . Since all nodes in the network graph are computed in parallel on massively-parallel hardware systems ( e.g . Esser et al. , 2016 ) , the latency for inference is dominated by the depth of the network , i.e . the longest path from input to output ( Fischer et al. , 2018 ) . Consequently , we prefer networks that compute all scales in parallel ( similar to Huang et al. , 2018 ; Ke et al. , 2017 ) and increase their depth for each additional intermediate output to networks that keep the depth constant and progressively increase their width ( Yu et al. , 2019 ) . Furthermore , multi-scale networks are also beneficial for the integration of global information , as especially required by dense prediction tasks like semantic segmentation ( Zhao et al. , 2017 ) . To further reduce the latency we also reduce the depth of the building blocks for each scale . In deep learning literature , the computational costs are usually quantified by counting the parameters and/or the multiply-accumulate operations ( MACs ) required for the inference of a single sample . For fully convolutional networks , the number of parameters is independent of the spatial resolution of the network ’ s input and the intermediate feature maps . Especially for large inputs as commonly used for semantic segmentation , the number of parameters does not cover the main workload and is , hence , not suited as a measure for computational costs . MACs have the advantage that they can be easily calculated and are usually a good approximation for the latency and throughput on CPUs and even GPUs . However , for most novel hardware accelerators , not the MACs , but memory transfers are dominating the computational costs in terms of power consumption ( Chen et al. , 2016 ; Sze et al. , 2017 ; Chao et al. , 2019 ) . These memory transfers are minimized on massivly-parallel hardware systems as long as the network graph fits into the in-computation memory of these systems , i.e . the memory of their arithmetic units . Since both the power consumption during inference and the production cost scale with the size of this memory , in addition to MACS , we also compare the size of the computational graphs between networks . Note that the practical benefits of ItNets can not be demonstrated on conventional CPUs or GPUs , since these hardware systems do not support the processing of neural networks in a fully-parallel and , hence , low-latency fashion . Note that , in this study , we focus on network models in the low-power regime of only few billion MACs , while processing large-scale images as commonly used for semantic segmentation ( for datasets , see Section 2.4 ) . The key contributions of this study are : • We introduce efficient networks with tiny computational graphs that heavily re-use a single building block and provide multiple intermediate outputs ( Sections 2.1 , 2.2 and 2.5 ) . • We search for the best hyperparameters of this model and investigate the effect of multiple outputs and weight sharing on the network training ( Sections 2.3 , 3.1 and 3.2 ) . • To our knowledge , we set the new state-of-the-art in terms of accuracy over the size of the computational graph and discuss the potential benefits for low-power applications . We will release the source code upon acceptance for publication . 2 METHODS . The following networks process images of size x× y × 3 and output N semantic maps mn of size x× y × C with C being the number of classes . 2.1 NETWORK ARCHITECTURE . We are interested in network architectures with a small computational footprint facilitating their application in mobile devices . To this end , we design a neural network that heavily re-uses the intermediate activations and weights ( Figure 1a ) . Conceptionally , the network model can be split into three main building blocks : the data block , the iterative block and the classification block ( for an overview and details , see Figure 1a and Figure 2 , respectively ) . While the data block is executed only once for each image , the iterative block can be executed multiple times in a row by feeding back its output as the input for the next iteration . The classification block outputs the prediction of the semantic map by processing the intermediate activations of the feedback signal . While the weights of the iterative block are shared between iterations , the weights of the classification block are unique for each iteration . We consider the size of the computational graph and the number of MACs as meaningful indicators for the computational footprint ( see also Section 1 ) . To obtain a high-accuracy network architecture under these objectives we reduce the size of the computational graph by introducing loops and optimize the following architectural hyperparameters : the number of scales L , the number of iterations N , and the number of bottleneck residual blocks K . 2.2 NETWORK TRAINING . For training , we use a joint cost function for all outputs of the network : L = ∑ n āncn ( m̃n , mn ) , where cn is the categorical cross entropy between the true labels m̃n and the network predictions mn . The weight factors an are normalized as follows : ān = an/ ( ∑ i ai ) . We use the Adam optimizer with β1 = 0.9 , β2 = 0.999 and a learning rate 0.001 that we multiply with 0.1 after 70 % and 85 % of the number of overall training epochs . We use a batch size of 8 and train the network for 2000 ( 4000 for Figure 5 ) and 900 for the CamVid and Cityscapes datasets , respectively . For Figure 3 , Figure 4 and the appendix , we report the mean values and the errors of the means across 5 trials . For Figures 1 and 5 , we report the trial with the highest peak accuracy over 3 trials . For the results shown in Figures 1 and 5 , we use dropout with rate 0.1 after the depth-wise convolutions in the bottleneck residual blocks and an L2 weight decay of 10−5 in all convolutional layers . For all other results , we do not use dropout and weight decay . 2.3 NETWORK EVALUATION . Throughout this study , we measure the quality of semantic segmentation by calculating the mean intersection-over-union ( mIoU Jaccard , 1912 ) , which is the ratio of the area of overlap and the area of union IoU = label ∩ prediction label ∪ prediction averaged over all classes . We consider a network to perform well if it achieves a high mIoU while requiring few MACs . To this end , we calculate the area under the curve of the mIoU ( yn ) over MACs ( xn ) with output index n as follows : AUC = N−1∑ n=0 ( xn+1 − xn ) ( ( yn+1 − y0 ) + ( yn − y0 ) 2 ) with ( x0 , y0 ) = ( 0 , 0.00828 ) , where y0 denotes the mIoU at chance level for the CamVid dataset . To compensate for different maximum numbers xN of MACs for different sets of hyperparameters , we normalize as follows : AUC = AUCxN . The size of the computational graph is computed by accumulating the memory requirements of all nodes , i.e . network layers , in the network graph . For each layer , the total required memory is the sum of the memory for parameters , input feature maps and output feature maps . The theoretical latency of a network if executed fully in parallel is determined by the length , i.e . the depth , of the path from input to output of this network ( see also Section 1 ) . For both the size of the computational graph and the latency , we only consider convolutional layers like commonly done in literature ( e.g . Paszke et al. , 2016 ; Wu et al. , 2018 ; Mehta et al. , 2019 ) . This means , we ignore other network layers like normalizations , activations , concatenations , additions and spatial resizing , for which we assume that they can be fused with the convolutional layers .

Summary: This paper proposes a homogeneous network structure for semantic segmentation, which optimizes for prediction accuracy, latency as well as memory footprint.  The paper studies anytime prediction setting and designs a re-usable single building block to reduce the memory footprint. Experimental results on CamVid data shows that it's possible to use a homogeneous network architecture to achieve competitive mIoU compared to previous work at the cost of increased MACs.  Experimental evaluation on larger datasets such as Cityscapes is prohibited due to memory constraints of the available GPUs.