SkipW: Resource Adaptable RNN with Strict Upper Computational Limit

1 INTRODUCTION . Since Recurrent Neural Networks ( RNN ) have been introduced Williams et al . ( 1986 ) , they have become one of the reference methods to process sequences . A typical architecture is the Long-ShortTerm-Memory neural network ( LSTM ) which allowed improvement in natural language processing such as large-vocabulary speech recognition ( Sak et al. , 2014 ; Li & Wu , 2015 ) . Used with CNNs they have also reached state of the art in automatic image captioning ( Vinyals et al. , 2015 ) . Deep learning models are now brought closer to the user rather than running in a distant cloud , helping to reduce latency , network congestion , and improving data security and privacy . However , smartphones and user devices impose additional constraints such as limited computation or energy . Handling these constraints has become an active research topic ( Zhang et al. , 2017 ; 2018 ; Howard et al. , 2019 ; Wu et al. , 2019 ; Cai et al. , 2020 ) . User devices can also host multiple processes running at the same time and starting or stopping abruptly , modifying the constraints affecting the processes . Few works have considered models that can be modified at run time to adapt to an evolving computational limit ( Yu et al. , 2019 ; Yu & Huang , 2019 ; Guerra et al. , 2020 ; Jin et al. , 2020 ) . However , none of these focus on sequences and therefore none address the problem of adapting the model in the middle of a sequence . In this context , this paper introduces Skip-Window ( SkipW ) , a flexible recurrent neural network architecture : its computational cost can be dynamically adapted during a sequence analysis to meet real time constraints changes . The proposed architecture can be combined with any RNN cell and allows to strictly limit the computational resources used to avoid exceeding a given budget . Furthermore , empirical experiments on four data sets ( Adding Task , MNIST , IMDB and HAR-2D-POSE ) demonstrate that this subsampling architecture is interesting in itself . Skip-Window matches or exceed the accuracy of existing approaches for a given computational cost . In addition , measurements on specific processors highlight that SkipW produces real computational and energy savings . 2 RELATED WORK . Typically , RNNs maintain a “ state ” , a vector of variables , over time . This state is supposed to accumulate relevant information and is updated recursively . Each input of the sequence is typically a ) processed by some deep layers and b ) then combined with the previous state through some other deep layers to compute the new state . Hence , the RNN can be seen as a function taking a sequence of inputs x = ( x1 , . . . , xT ) and recursively computing a set of states s = ( s1 , . . . , sT ) . Each state st is computed from st−1 and xt by a cell S of the RNN . As neural networks are increasingly run on limited hardware , recent research has focused on controlling their computational cost . 2.1 FLEXIBLE NEURAL NETWORKS . A few architectures have recently been designed to adapt the computational complexity of a Deep Neural Network ( DNN ) without reloading the whole model . This can be achieved by removing/adding neurons ( Yu et al. , 2019 ; Yu & Huang , 2019 ) or by modifying the quantization of the weights ( Guerra et al. , 2020 ; Jin et al. , 2020 ) . An efficient embedding of a mixture of Convolutional Neural Network ( CNNs ) also allows to add or remove several models at the same time , hence changing the computational cost ( Ruiz & Verbeek , 2019 ) . 2.1.1 THRRNN . For RNNs specifically , ThrRNN ( Lambert et al. , 2020 ) aims to control computation time by not processing some inputs . This is controlled by an update gate ut . The tradeoff between the average accuracy and the average number of updates can be modified during inference by changing a single parameter thr . ThrRNN can wrap any RNN cell S : ut = fbinarize ( ũt , thr ) = { 0 if ũt < thr 1 otherwise ( 1 ) ∆ũt = σ ( Wst + b ) ( 2 ) ũt+1 = ut∆ũt + ( 1− ut ) ( ũt +min ( ∆ũt , 1− ũt ) ) ( 3 ) st = utS ( st−1 , xt ) + ( 1− ut ) st−1 . ( 4 ) When an input is processed , an update gate computes the quantity ∆ũt that determines how many inputs will be skipped . In practice the ∆ũt are accumulated in ũt until ũt ≥ thr . 2.2 RECURRENT NEURAL NETWORK WITH LOW COMPUTATIONAL COMPLEXITY . Several architectures have been proposed to limit or reduce the computational cost of RNNs , but this cost can not be adapted at inference . A first class of architectures dynamically reduces computation based on the input . SkipRNN ( Campos et al. , 2018 ) predates and is similar to ThrRNN , except that the binarization function does not change . A similar mechanism has been proposed by Zhang et al . ( 2019 ) . Other architectures directly select the next input to process ( Yeung et al. , 2016 ; Yu et al. , 2017 ; Hansen et al. , 2019 ; Song et al. , 2018 ) . Early exit has also been investigated by Dennis et al . ( 2019 ) . Tao et al . ( 2019 ) also use xt as input to an update gate . So do Seo et al . ( 2018 ) ; Jernite et al . ( 2017 ) ; Li et al . ( 2020 ) . However , they do not skip any input but perform partial state updates . A second class of architectures focuses on reducing the overal cost of the RNN . FastRNN is an RNN augmented with a residual connection with two extra scalar parameters and FastGRNN is an improved FastRNN : the residual connection is extended to a gate and RNN matrices are low rank , sparse and quantized ( Kusupati et al. , 2018 ) . Other architectures reduce the RNN length . Chan et al . ( 2016 ) train an encoder to reduce the input length . Yeung et al . ( 2016 ) ; Shan et al . ( 2018 ) ; Chen et al . ( 2018 ) propose various mechanisms to summarize subsequences of windows of inputs . 2.3 RECURRENT NEURAL NETWORK WITH HIERARCHICAL-DEPENDENT COMPLEXITY . A class of architectures focuses on hierarchy level concept to reduce the complexity . These methods are mainly used in the context of multi-layer RNNs where each layer is supposed to model a different level in the hierarchy ( e.g . for a corpus the levels could be documents , paragraphs , sentences , words , letters ) . These approaches are based on the fact that a hierarchical separation exists within a sequence of inputs , which might not always be the case . In ( Koutnik et al. , 2014 ) , the hidden state is partitioned into different modules , each module has its own clock period , meaning that they will be updated at different times . Skipping updates of part of the hidden state decreases the computational cost . In ( Koutnik et al. , 2014 ) , the update periods are chosen arbitrarily , for example using an exponential series . For stacked-RNNs , Chung et al . ( 2017 ) ; Chang et al . ( 2017 ) conditionally update each layer based on a feature level criterion , or by dilating a skip connection . Layers close to the inputs would model lower feature levels and be updated more frequently . Further layers would model higher level features . In ( Chung et al. , 2017 ) , a layer modeling sentences would be updated only when a word is processed entirely ( in a model fed character by character ) , from the layer modeling words . Before the end of a word is reached , the state of the former layer is copied across input steps . 2.4 RELATIONSHIP TO OUR WORK . ThrRNN is the closest model to SkipW . Both are flexible RNNs and skip some inputs . However , ThrRNN optimizes computational cost on average over sequences . This induces two variabilities : a ) inter-sequence variability : the model will not use the same number of updates for every sequence ; and b ) intra-sequence variability : the number of updates will not be uniform across time steps , updates may be concentrated in a certain part of the sequence . These two variabilities can cause the model to exceed its computational budget and , therefore , to either shut down or delay the output . SkipW does not have this problem as it strictly enforces a computational constraint over each window of inputs . Other strategies for flexible models are not straightforward to apply to RNN . They require specialized training algorithms . They have never been applied to models processing inputs of an RNN or to make an RNN flexible and it is not clear how they would need to be modified . Furthermore , these models adapt between independent inputs whereas , for sequences , adaptation is necessary between time steps . RNN architectures with low complexity are orthogonal to our approach . They do not offer flexibility . They could be combined with and benefit from our approach . However , SkipRNN ( which we are based on ) and related methods have one big advantage over others : by skipping inputs , they also skip any modification of an input , such as processing by an expensive CNN for images . As SkipW makes decision over a window of inputs , it has some superficial similarity to methods summarizing windows or hierarchical RNNs . However , SkipW a ) does not summarize windows and b ) does not even look at these inputs before deciding what to skip . 3 SKIP-WINDOWS ARCHITECTURE . Skip-Windows ( SkipW ) is a wrapper for a RNN cell S. It uses a conditional computation mechanism to skip some updates . Rather than at each input xt , update gates are computed at the beginning of windows of inputs , that is , every L time steps ( Figure 1 ) . In other words , before any new L-size window of inputs , a L-size vector ũW is computed . ũW [ i ] can be seen as the importance of input i in the window . Then , the architecture includes a selectK mechanism . This function takes as input the vector ũW and outputs the vector ũKW , setting L − K elements to a value that ensures the associated inputs are not processed ( 0 in Figure 2 ) . Therefore , it ensures that at most K out of every L inputs will be processed . In other words , it forces the RNN cell to skip ( L − K ) out of every L inputs . This ensures a strict upper bound on the computational cost of the model for a sequence and for each window , therefore alleviating the inter-sequence variability and intra-sequence variability issues . Similarly to other works , the binary state update L-size vector , uW , is then obtained by binarizing the remaining values as in equation 1 . For example , by setting all values below a threshold to a value that ensures the associated inputs are not processed ( 0 in Figure 2 ) . An example of the Skip-Window cell implementation is represented in Figure 2 . In this case , selectK is implemented as a topK function . This enforces the strict constraint on the number of updates . The topK operation keeps unchanged the K highest value in ũW , t , and sets the ( L −K ) others to 0 . The corresponding architecture can be characterized as follows : st = ut · S ( st−1 , xt ) + ( 1− ut ) · st−1 ( 5 ) ũW , t+1 = γ · σ ( Ww ( st−1 , t ) + bw ) + ( 1− γ ) · ũW , t ( 6 ) γ = { 1 if i == 0 0 otherwise ( 7 ) i = t mod L ( 8 ) ũKW , t = topK ( ũW , t ) ( 9 ) ut = fbinarize ( ũ K W , t [ i ] , thr ) = { 0 if ũKW , t [ i ] < thr 1 otherwise ( 10 ) where Ww is a weight matrix of size ( N + 1 ) × L , N is the number of hidden units as defined by the RNN cell S , bW is a L-vector bias , σ is the sigmoid function and mod is the modulo operation . Instead of a topK function for selectK , it is also possible to use a stochastic sampling mechanism that randomly selects ( without replacement ) K out of L elements of ũW where the probability of selecting each element of index i is proportional to ũW [ i ] . Some selectK alternatives are discussed and evaluated in Appendix H. Including the time step t in equation 6 is also optional and can be replaced by a value ensuring the state is not static if no update is made in a window . For example , the number of inputs since the last update or the number of windows already processed . Training the model The model is trained to minimize a two-part loss , similarly to Campos et al . ( 2018 ) . The first term measures the accuracy of the task , and the second one penalizes inputs used : Lbudget = λ T∑ t=1 ut , ( 11 ) where λ is the cost associated to the use of a single input . More experimental details are provided in Appendix B . Error gradients The model is differentiable except for the fbinarize and topK function . To train the model using standard backpropagation the straight-through estimator is used as done in Campos et al . ( 2018 ) for fbinarize . Other alternatives might involve reinforcement learning such as REINFORCE ( Williams , 1992 ) or , in the case of the topK function , the usage of a differentiable topK as proposed in Xie et al . ( 2020 ) . Early experiments using a differentiable topK ( Xie et al. , 2020 ) have shown worse results than the straight-through estimator . This suggests that constraining computation may be an interesting approximation for a topK operation . Adapting computational cost at inference During inference , adapting the tradeoff between model performance and computational cost can be done using two factors : the K in equation 9 and the thr in equation 10 . These two parameters can be modified together or one at a time . Increasing/lowering thr parameter in [ 0 , 1 ] encourages the model to process fewer/more inputs . Changing K in { 0 . . . L } forces the model to process at most K/L of the window . Choice of the window size hyper-parameter By the nature of the model , the task can influence the choice of L. It can be hand tuned or computed using typical hyper-parameter search methods such as grid search . Choosing small L allows the model to make update decisions for the near future only but offers less choice in operating points . Similarly a bigger L requires the model to predict its update decisions for a bigger time span but offers more flexibility . At the extreme when L = 1 , each window consists of a single input .

Summary: This work introduces Skip-Window (SkipW), an approach that allows RNNS to have improved computational efficiency at the cost of accuracy. SkipW adds a procedure to existing RNN cells that allows them to process fewer inputs while remaining in a strict computational budget. This work demonstrates the benefits of SkipW through experiments on multiple data sets.