SkipW: Resource Adaptable RNN with Strict Upper Computational Limit

1 INTRODUCTION . Since Recurrent Neural Networks ( RNN ) have been introduced Williams et al . ( 1986 ) , they have become one of the reference methods to process sequences . A typical architecture is the Long-ShortTerm-Memory neural network ( LSTM ) which allowed improvement in natural language processing such as large-vocabulary speech recognition ( Sak et al. , 2014 ; Li & Wu , 2015 ) . Used with CNNs they have also reached state of the art in automatic image captioning ( Vinyals et al. , 2015 ) . Deep learning models are now brought closer to the user rather than running in a distant cloud , helping to reduce latency , network congestion , and improving data security and privacy . However , smartphones and user devices impose additional constraints such as limited computation or energy . Handling these constraints has become an active research topic ( Zhang et al. , 2017 ; 2018 ; Howard et al. , 2019 ; Wu et al. , 2019 ; Cai et al. , 2020 ) . User devices can also host multiple processes running at the same time and starting or stopping abruptly , modifying the constraints affecting the processes . Few works have considered models that can be modified at run time to adapt to an evolving computational limit ( Yu et al. , 2019 ; Yu & Huang , 2019 ; Guerra et al. , 2020 ; Jin et al. , 2020 ) . However , none of these focus on sequences and therefore none address the problem of adapting the model in the middle of a sequence . In this context , this paper introduces Skip-Window ( SkipW ) , a flexible recurrent neural network architecture : its computational cost can be dynamically adapted during a sequence analysis to meet real time constraints changes . The proposed architecture can be combined with any RNN cell and allows to strictly limit the computational resources used to avoid exceeding a given budget . Furthermore , empirical experiments on four data sets ( Adding Task , MNIST , IMDB and HAR-2D-POSE ) demonstrate that this subsampling architecture is interesting in itself . Skip-Window matches or exceed the accuracy of existing approaches for a given computational cost . In addition , measurements on specific processors highlight that SkipW produces real computational and energy savings . 2 RELATED WORK . Typically , RNNs maintain a “ state ” , a vector of variables , over time . This state is supposed to accumulate relevant information and is updated recursively . Each input of the sequence is typically a ) processed by some deep layers and b ) then combined with the previous state through some other deep layers to compute the new state . Hence , the RNN can be seen as a function taking a sequence of inputs x = ( x1 , . . . , xT ) and recursively computing a set of states s = ( s1 , . . . , sT ) . Each state st is computed from st−1 and xt by a cell S of the RNN . As neural networks are increasingly run on limited hardware , recent research has focused on controlling their computational cost . 2.1 FLEXIBLE NEURAL NETWORKS . A few architectures have recently been designed to adapt the computational complexity of a Deep Neural Network ( DNN ) without reloading the whole model . This can be achieved by removing/adding neurons ( Yu et al. , 2019 ; Yu & Huang , 2019 ) or by modifying the quantization of the weights ( Guerra et al. , 2020 ; Jin et al. , 2020 ) . An efficient embedding of a mixture of Convolutional Neural Network ( CNNs ) also allows to add or remove several models at the same time , hence changing the computational cost ( Ruiz & Verbeek , 2019 ) . 2.1.1 THRRNN . For RNNs specifically , ThrRNN ( Lambert et al. , 2020 ) aims to control computation time by not processing some inputs . This is controlled by an update gate ut . The tradeoff between the average accuracy and the average number of updates can be modified during inference by changing a single parameter thr . ThrRNN can wrap any RNN cell S : ut = fbinarize ( ũt , thr ) = { 0 if ũt < thr 1 otherwise ( 1 ) ∆ũt = σ ( Wst + b ) ( 2 ) ũt+1 = ut∆ũt + ( 1− ut ) ( ũt +min ( ∆ũt , 1− ũt ) ) ( 3 ) st = utS ( st−1 , xt ) + ( 1− ut ) st−1 . ( 4 ) When an input is processed , an update gate computes the quantity ∆ũt that determines how many inputs will be skipped . In practice the ∆ũt are accumulated in ũt until ũt ≥ thr . 2.2 RECURRENT NEURAL NETWORK WITH LOW COMPUTATIONAL COMPLEXITY . Several architectures have been proposed to limit or reduce the computational cost of RNNs , but this cost can not be adapted at inference . A first class of architectures dynamically reduces computation based on the input . SkipRNN ( Campos et al. , 2018 ) predates and is similar to ThrRNN , except that the binarization function does not change . A similar mechanism has been proposed by Zhang et al . ( 2019 ) . Other architectures directly select the next input to process ( Yeung et al. , 2016 ; Yu et al. , 2017 ; Hansen et al. , 2019 ; Song et al. , 2018 ) . Early exit has also been investigated by Dennis et al . ( 2019 ) . Tao et al . ( 2019 ) also use xt as input to an update gate . So do Seo et al . ( 2018 ) ; Jernite et al . ( 2017 ) ; Li et al . ( 2020 ) . However , they do not skip any input but perform partial state updates . A second class of architectures focuses on reducing the overal cost of the RNN . FastRNN is an RNN augmented with a residual connection with two extra scalar parameters and FastGRNN is an improved FastRNN : the residual connection is extended to a gate and RNN matrices are low rank , sparse and quantized ( Kusupati et al. , 2018 ) . Other architectures reduce the RNN length . Chan et al . ( 2016 ) train an encoder to reduce the input length . Yeung et al . ( 2016 ) ; Shan et al . ( 2018 ) ; Chen et al . ( 2018 ) propose various mechanisms to summarize subsequences of windows of inputs . 2.3 RECURRENT NEURAL NETWORK WITH HIERARCHICAL-DEPENDENT COMPLEXITY . A class of architectures focuses on hierarchy level concept to reduce the complexity . These methods are mainly used in the context of multi-layer RNNs where each layer is supposed to model a different level in the hierarchy ( e.g . for a corpus the levels could be documents , paragraphs , sentences , words , letters ) . These approaches are based on the fact that a hierarchical separation exists within a sequence of inputs , which might not always be the case . In ( Koutnik et al. , 2014 ) , the hidden state is partitioned into different modules , each module has its own clock period , meaning that they will be updated at different times . Skipping updates of part of the hidden state decreases the computational cost . In ( Koutnik et al. , 2014 ) , the update periods are chosen arbitrarily , for example using an exponential series . For stacked-RNNs , Chung et al . ( 2017 ) ; Chang et al . ( 2017 ) conditionally update each layer based on a feature level criterion , or by dilating a skip connection . Layers close to the inputs would model lower feature levels and be updated more frequently . Further layers would model higher level features . In ( Chung et al. , 2017 ) , a layer modeling sentences would be updated only when a word is processed entirely ( in a model fed character by character ) , from the layer modeling words . Before the end of a word is reached , the state of the former layer is copied across input steps . 2.4 RELATIONSHIP TO OUR WORK . ThrRNN is the closest model to SkipW . Both are flexible RNNs and skip some inputs . However , ThrRNN optimizes computational cost on average over sequences . This induces two variabilities : a ) inter-sequence variability : the model will not use the same number of updates for every sequence ; and b ) intra-sequence variability : the number of updates will not be uniform across time steps , updates may be concentrated in a certain part of the sequence . These two variabilities can cause the model to exceed its computational budget and , therefore , to either shut down or delay the output . SkipW does not have this problem as it strictly enforces a computational constraint over each window of inputs . Other strategies for flexible models are not straightforward to apply to RNN . They require specialized training algorithms . They have never been applied to models processing inputs of an RNN or to make an RNN flexible and it is not clear how they would need to be modified . Furthermore , these models adapt between independent inputs whereas , for sequences , adaptation is necessary between time steps . RNN architectures with low complexity are orthogonal to our approach . They do not offer flexibility . They could be combined with and benefit from our approach . However , SkipRNN ( which we are based on ) and related methods have one big advantage over others : by skipping inputs , they also skip any modification of an input , such as processing by an expensive CNN for images . As SkipW makes decision over a window of inputs , it has some superficial similarity to methods summarizing windows or hierarchical RNNs . However , SkipW a ) does not summarize windows and b ) does not even look at these inputs before deciding what to skip . 3 SKIP-WINDOWS ARCHITECTURE . Skip-Windows ( SkipW ) is a wrapper for a RNN cell S. It uses a conditional computation mechanism to skip some updates . Rather than at each input xt , update gates are computed at the beginning of windows of inputs , that is , every L time steps ( Figure 1 ) . In other words , before any new L-size window of inputs , a L-size vector ũW is computed . ũW [ i ] can be seen as the importance of input i in the window . Then , the architecture includes a selectK mechanism . This function takes as input the vector ũW and outputs the vector ũKW , setting L − K elements to a value that ensures the associated inputs are not processed ( 0 in Figure 2 ) . Therefore , it ensures that at most K out of every L inputs will be processed . In other words , it forces the RNN cell to skip ( L − K ) out of every L inputs . This ensures a strict upper bound on the computational cost of the model for a sequence and for each window , therefore alleviating the inter-sequence variability and intra-sequence variability issues . Similarly to other works , the binary state update L-size vector , uW , is then obtained by binarizing the remaining values as in equation 1 . For example , by setting all values below a threshold to a value that ensures the associated inputs are not processed ( 0 in Figure 2 ) . An example of the Skip-Window cell implementation is represented in Figure 2 . In this case , selectK is implemented as a topK function . This enforces the strict constraint on the number of updates . The topK operation keeps unchanged the K highest value in ũW , t , and sets the ( L −K ) others to 0 . The corresponding architecture can be characterized as follows : st = ut · S ( st−1 , xt ) + ( 1− ut ) · st−1 ( 5 ) ũW , t+1 = γ · σ ( Ww ( st−1 , t ) + bw ) + ( 1− γ ) · ũW , t ( 6 ) γ = { 1 if i == 0 0 otherwise ( 7 ) i = t mod L ( 8 ) ũKW , t = topK ( ũW , t ) ( 9 ) ut = fbinarize ( ũ K W , t [ i ] , thr ) = { 0 if ũKW , t [ i ] < thr 1 otherwise ( 10 ) where Ww is a weight matrix of size ( N + 1 ) × L , N is the number of hidden units as defined by the RNN cell S , bW is a L-vector bias , σ is the sigmoid function and mod is the modulo operation . Instead of a topK function for selectK , it is also possible to use a stochastic sampling mechanism that randomly selects ( without replacement ) K out of L elements of ũW where the probability of selecting each element of index i is proportional to ũW [ i ] . Some selectK alternatives are discussed and evaluated in Appendix H. Including the time step t in equation 6 is also optional and can be replaced by a value ensuring the state is not static if no update is made in a window . For example , the number of inputs since the last update or the number of windows already processed . Training the model The model is trained to minimize a two-part loss , similarly to Campos et al . ( 2018 ) . The first term measures the accuracy of the task , and the second one penalizes inputs used : Lbudget = λ T∑ t=1 ut , ( 11 ) where λ is the cost associated to the use of a single input . More experimental details are provided in Appendix B . Error gradients The model is differentiable except for the fbinarize and topK function . To train the model using standard backpropagation the straight-through estimator is used as done in Campos et al . ( 2018 ) for fbinarize . Other alternatives might involve reinforcement learning such as REINFORCE ( Williams , 1992 ) or , in the case of the topK function , the usage of a differentiable topK as proposed in Xie et al . ( 2020 ) . Early experiments using a differentiable topK ( Xie et al. , 2020 ) have shown worse results than the straight-through estimator . This suggests that constraining computation may be an interesting approximation for a topK operation . Adapting computational cost at inference During inference , adapting the tradeoff between model performance and computational cost can be done using two factors : the K in equation 9 and the thr in equation 10 . These two parameters can be modified together or one at a time . Increasing/lowering thr parameter in [ 0 , 1 ] encourages the model to process fewer/more inputs . Changing K in { 0 . . . L } forces the model to process at most K/L of the window . Choice of the window size hyper-parameter By the nature of the model , the task can influence the choice of L. It can be hand tuned or computed using typical hyper-parameter search methods such as grid search . Choosing small L allows the model to make update decisions for the near future only but offers less choice in operating points . Similarly a bigger L requires the model to predict its update decisions for a bigger time span but offers more flexibility . At the extreme when L = 1 , each window consists of a single input .

Summary: The paper proposes Skip-Window or SkipW an abstraction encapsulating RNN cells to actively skip updates similar to some earlier works like Skip-RNN, Skim-RNN, and ThrRNN. The novelty of the method comes is in having control over the total updates to control the overall computational budget compared to previous methods which didn't provide deterministic upper bounds and varied depending on the inputs. The idea is very simple and straightforward and can be looked at as a logical extension to the Skip-RNN line of work combined with a windowed approach on time series as used in ShaRNN (Dennis et al., NeurIPS 2019). The entire time series is divided into Windows of length L (which is a tunable parameter) and each window has a precomputed (from the final hidden state of the previous window) per-time step (update inside the window) importance vector which can be used as an indicator to update or not to update following the binarization as done in previous methods. The strict sparsification of this per-window importance vector to have only K non-zeros per window helps reduce compute to an upper bound ratio of K/L. The method further uses another threshold term over the sparsified importance vector to control finer budget requirements if needed. The experimentation is done on 2 tasks HAR-2D-Pose (with 32 time steps) and Adding task with 50 timesteps. The evaluation shows that Skip-Window shows good performance./accuracy compared to previous flexible RNNs with a reduction in the total number of updates. Finally, the impressive part of the paper is the real-world evaluation on Jetson Nano with a more complex workflow involving pose estimation from images for HAR-2D-Pose.