Title: SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model
Venue: NeurIPS 2024
Accepted: yes
Keywords: Time series data

Computational complexity reduction

Latency reduction

Real-Time results

Inference at the edge

Causality
Abstract: Consumer electronics used to follow the miniaturization trend described by Moore’s Law. Despite increased processing power in Microcontroller Units (MCUs), MCUs used in the smallest appliances are still not capable of running even moderately big, state-of-the-art artificial neural networks (ANNs) especially in time-sensitive scenarios. In this work, we present a novel method called Scattered Online Inference (SOI) that aims to reduce the computational complexity of ANNs. SOI leverages the continuity and seasonality of time-series data and model predictions, enabling extrapolation for processing speed improvements, particularly in deeper layers. By applying compression, SOI generates more general inner partial states of ANN, allowing skipping full model recalculation at each inference.

SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model

Abstract

Consumer electronics used to follow the miniaturization trend described by Moore's Law. Despite increased processing power in Microcontroller Units (MCUs), MCUs used in the smallest appliances are still not capable of running even moderately big, state-of-the-art artificial neural networks (ANNs) especially in time-sensitive scenarios. In this work, we present a novel method called Scattered Online Inference (SOI) that aims to reduce the computational complexity of ANNs. SOI leverages the continuity and seasonality of time-series data and model predictions, enabling extrapolation for processing speed improvements, particularly in deeper layers. By applying compression, SOI generates more general inner partial states of ANN, allowing skipping full model recalculation at each inference.

1 Introduction

Moore's Law (Moore, 1998) predicts that the number of transistors on a microchip will double every 2 years. As the number of transistors increases, the computational capacity of systems-on-chip (SoCs) also grows. Accompanied by the further miniaturization of components, the increased complexity of SoCs allows for the production of more compact appliances with equal or higher capabilities.

Despite the continuous development of more advanced hardware, the smallest appliances still cannot fully benefit from increasingly popular neural-based solutions, as they are not able to run them efficiently. Examples of such appliances include wireless in-ear headphones, smart watches, and AR glasses. Furthermore, the size of state-of-the-art neural networks is growing at much faster rate than that described by Moore's Law (Xu et al., 2018). Adding to the challenge, according to researchers (Leiserson et al., 2020), Moore's Law is starting to decelerate due to the physical constraints of semiconductors, and the 'room at the bottom' is depleting.

Currently available neural network technologies enable machines to outperform humans in numerous applications in terms of measured performance (Nguyen et al., 2020; Bakhtin et al., 2022; Schrittwieser et al., 2020). However, despite this achievement, the same models fall significantly short when it comes to energy efficiency compared to humans. The human brain consumes a mere 20 Watts of power (Laughlin et al., 1998; Sengupta & Stemmler, 2014; Balasubramanian, 2021) and according to Xu et al. (2018) it is estimated to be over five orders of magnitude more energy efficient than modern neural networks.

This disparity in energy efficiency can be attributed to the common pursuit of the highest model quality in the literature, showcasing the full capabilities of the developed technology, often at the expense of efficiency. This behavior is justifiable due to the ease of comparing different solutions using well-defined metrics that are independent of the hardware and software. Conversely, estimating a model's energy efficiency is a more complex task, influenced by various factors including the running environment.

However, this trend within our community may prove restrictive, particularly for applications like realtime systems. These applications naturally demand optimal performance alongside high-efficiency solutions, rendering most current Deep Neural Networks (DNNs) impractical for such tasks. Furthermore, due to the substantial discrepancy between assumptions for high-efficiency on-device solutions and high-performing monolithic models, compressing these models may not be a viable means of applying state-of-the-art DNNs to real-time tasks.

The importance of neural system efficiency is also increasingly significant from ecological and economic standpoints (Lacoste et al., 2019; Patterson et al., 2021).

1.1 Related Works

The Short-Term Memory Convolution (STMC) (Stefa´ nski et al., 2023) was devised to enhance the efficiency of Convolutional Neural Networks (CNNs) inference by reducing computation requirements at each step via eliminating the need to recompute prior states. The authors achieved a notable 5.68fold reduction in inference time and a 30% decrease in memory consumption compared to the Incremental Inference method (Romaniuk et al., 2020). STMC enables the conversion of a standard CNN model, which typically requires an input of size at least as large as its receptive field, into a model capable of processing a single input frame at a time, akin to Long-Short Term Memory networks (LSTM). The SOI method is built upon the STMC foundation, offering distinct treatment of strided layers and yielding a compelling new inference pattern 1 .

Routing methods constitute a popular category of algorithms tailored to optimize the inference process of Recurrent Neural Networks (RNNs). In the field of Natural Language Processing (NLP), Yu et al. (2017) introduced an approach that involves traversing segments of the computational graph. This traversal is guided by decisions made by a reinforcement learning model following the processing of a fixed number of words. Another notable contribution by Campos et al. (2018) yielded an algorithm capable of selectively bypassing partial state updates within an RNN during inference, influenced by the input's length. In NLP terms, this concept can be compared to the mechanism of skipping words.

The research by Jernite et al. (2017) introduced a distinct method to regulate computation within a recurrent unit. This method relied on a scheduler unit that facilitated partial updates to the network's state, only when the computational budget limit was reached. Meanwhile, Seo et al. (2018) proposed an approach referred to as 'word skimming'. In this approach, the authors designed both small and large RNN models that could be interchangeably utilized for inference through the utilization of Gumbel softmax. The exploration of hybrid techniques combining jumping, skimming, and skipping was further advanced by Hansen et al. (2019), who published additional solutions in this direction.

The RNN routing methods have found successful applications in CNNs as well. Wang et al. (2018) introduced an approach that enables a model to learn a policy for skipping entire convolutional layers on a per-input basis. A similar concept, involving adaptive inference graphs conditioned on image inputs, was put forth by Veit & Belongie (2018). Additionally, several authors have contributed methods for early-exit during CNN inference (Bolukbasi et al., 2017; Teerapittayanon et al., 2016; Huang et al., 2017). In these methods, the network is trained to skip portions of the computational graph towards the final stages, based on the characteristics of the input.

Other commonly employed methods for model optimization include pruning (LeCun et al., 1989) and quantization (Gray & Neuhoff, 1998). Importantly, both of these methods are not mutually exclusive and can coexist alongside routing methods within a single neural network.

1.2 Novelty

In this study, we introduce a method for reducing the computational cost of a ANN model while incurring only a negligible decrease in the model's performance. Importantly, these reductions are achieved with minimal alterations to the architecture, making it suitable for various tasks where factors such as energy consumption or time are of paramount importance.

Our approach involves the conversion of a conventional ANN model, initially trained to process segments of time-series data with arbitrary lengths in an offline mode, into a model that processes

the data element by element, enabling real-time usage. Notably, our method builds upon the STMC technique (Stefa´ nski et al., 2023). STMC is designed to perform each distinct operation exactly once. SOI extends this concept by omitting some operations in a structured manner.

In this study, we introduce a novel method named Scattered Online Inference (SOI), which is based on the following key principles:

The reduction of computational complexity is achieved through the implementation of partial predictions of the network's future state.

SOI operates as a two-phase system. The initial phase involves compressing data within the time domain using data compression. The subsequent phase focuses on data reconstruction, employing the most suitable extrapolation scheme.

The method preserves the causal nature of the optimized network architecture.

SOI's applicability is confined to a single-frame online inference and it necessitates the incorporation of skip connections to update the network's output following each inference.

1.3 Limitations

While our method, demonstrates significant advantages in reducing computational complexity, it is essential to acknowledge several limitations inherent to our approach.

First, the performance reduction observed with SOI, although acceptable within the context of our work, may not be tolerable in applications requiring the highest accuracy levels. This reduction, while compensated by a substantial reduction in computational cost, suggests a trade-off between efficiency and model performance that may limit SOI's applicability in scenarios where performance is the absolute priority.

Second, the flexibility provided by SOI to balance computational cost and model performance introduces complexity in selecting the optimal configuration. The method allows users to adjust this trade-off based on application requirements, but this demands careful tuning and validation, which could be resource-intensive in training.

Third, the method's reliance on partial state predictions and data compression may introduce cumulative errors over time, particularly in longer sequences of time-series data. This is less critical in real-time, short-sequence applications but could degrade performance in tasks requiring continuous operation over extended periods without full model recalculations.

Lastly, SOI's applicability is primarily demonstrated on specific neural network architectures. Although we expect SOI to generalize to other architectures, its effectiveness in reducing computational complexity while maintaining acceptable performance may vary depending on the network design, the nature of the task, and the dataset. Further research and experimentation are necessary to explore its utility across a broader range of models and applications.

Time

Network

depth

A

B

Extrapolatedframe

Outputofconvolutionallayer

Removed frame

D

E

XX

Time

Network

x4x5xxx

Extrapolated frame

Convolution

Outputofconvolutional layer

Skip connection

Buffered frame

Copy

Predicted frames(Precomputed)

D

x4x5x6x7x8

2 Methods

When processing a time-series in online mode, the model goes through each incoming element 2 separately. In this paper we refer to such an event as inference .

An improvement in computation complexity is achieved by introducing the partially predictive compression layer adapted for online inference, as well as by avoiding the redundant computations done during previous inferences as in STMC study. Therefore, after each inference, the results which would be recalculated in subsequent runs are cached. We refer to such cacheable outputs as a partial state of the network.

2.1 Scattered Online Inference

Scattered Online Inference (SOI) is a method which modifies the inference pattern of a network to skip the recalculation of certain layers in a predetermined pattern. This is achieved through the use of compression and extrapolation. Both operations are exclusively applied along the time axis. In this study, as example, we employed strided convolutions as compression layers and a basic form of extrapolation, where the last known value is used to replace the missing ones 3 . To facilitate a better understanding of the SOI algorithm in CNNs, in Figure 1, we define three types of convolutional layers utilized in our method. For comparison purposes, Figure 1 also includes standard convolution and strided convolution.

Strided-Cloned Convolution (S-CC) performs a 2-step operation. Firstly, it applies a strided convolution to the input in order to compress the data. In the second step it fills the gaps created by striding using any form of extrapolation. In our experiments we extrapolate by simply copying a previous frame. The copied frame is then aligned with a future frame relatively to its input. In practice, we split the stride and extrapolation operations into different layers which results in optimization of computational complexity between those layers. Because of that we will refer to this as S-CC pair.

Shifted Convolution (SC) shifts data in time domain after the convolution thus creating additional predicted network states. This layer may be used for additional latency reduction.

Shifted Strided-Cloned Convolution (SS-CC) is a combination of S-CC pair and SC layer which is needed if we want to do both of them at the same point of the network. In our experiments we extrapolate output vector first and then apply data shifting to reduce size of introduced partial state prediction.

SOI can be divided into two types depending on how prediction is handled. These two types have significantly different inference patterns (Fig. 2).

Partially Predictive (PP) In this type of SOI, we do not introduce any additional predictive components in the network. This implies that after compression, the most recent frame stores information for the current output frame and the future output frame. This type of SOI uses S-CC pairs only. This configuration results in only one inference (the first one), which updates all network states, while all subsequent ones use relevant buffered partial network states. Although hybrid setups involving more full passes are viable, they fall outside the scope of this paper. It's important to note that this mode does not reduce peak computational complexity, but rather the average computational complexity.

Fully Predictive (FP) In this type of SOI, we introduce additional predictive components to the network. Compared to PP SOI, the most recent frame does not store any information about the current output frame, but rather about two future ones, hence the name 'Fully Predictive', as only the future output is calculated. This is a more challenging task than PP SOI, but it can significantly decrease the latency of the model. This mode utilizes both S-CC pairs and SC layers. It optimizes both peak computational complexity and latency because it allows some inferences to be predicted in full. The fully predicted inference, in contrast to the regular inference which requires newly collected input, operates only on already processed data and can calculate relevant network states while the system awaits the new data, reducing the amount of computation required when the new data arrives.

Both of these types of SOI can be combined. This occurs when we first introduce PP SOI compression and then after some number of layers, we introduce an additional shift in the time axis after which the model can be treated as FP SOI.

2.2 Mathematical Formulation

Let us assume that an input of the model is represented by a 1D time series vector X ∈ R N composed of N samples. Additionally let's say that a network is composed of 5 convolutional layers and output vector l Y for l -th layer is of the same shape as the input. Each convolutional layer has a 1D kernel h l ∈ R M l which can be represented by a Toeplitz matrix H l ∈ R N × ( N -M l +1) . We get:

After which we apply activation function σ and get the input for the next layer:

We use l X t to denote a segment of X ending in time t of a length matching the context it is used in (e.g. assuming h l denotes a convolutional kernel of layer l and provided with the context h l · l X t , l X t has M l elements to match the kernel size).

By using the STMC inference pattern we perform inference for each element of X separately and reuse past partial states of each layer to fully reconstruct the receptive field of the network which can be represented as follows:

where ( · | t · ) represents the concatenation of vectors in time axis.

If we use a convolutional layer with a stride of 2 as our second layer, then in the standard pattern, the size of the output vector of this layer is halved compared to its input. Consequently, every subsequent layer also has its input and output tensors reduced to half the size. We can restore the output to its original size by, for instance, applying transposed convolution. Let's assume that we apply transposed

convolution in the 4th layer of our network. In comparison to our initial plain network, the new strided network will have the same number of operations in both the first and last layers. The 2nd, 3rd, and 4th layers will each have half the computations as before. In the 2nd layer, this reduction is due to the application of stride. In the third layer, it is a result of the smaller input size. Similarly, in the 4th layer, if we disregard multiplications by zero.

When employing the STMC inference method, it's anticipated that each layer should process a single element and produce a single output. If we apply a similar stride and transposed convolution pattern without any additional modification, we'll face difficulties in reconstructing the output. Strided convolution would provide output values for even-numbered inferences (assuming a stride size of 2). However, during odd inferences, the 4th layer (transposed convolution) would require an upcoming even-numbered inference value, which would not yet be available. The authors of STMC propose a solution where every inference is treated as even-numbered, maintaining separate states for odd and even input positions. However, this pattern presents a challenge due to the exponential increase in the number of states for each added strided convolution.

SOI addresses this issue by removing the necessity of storing additional states, albeit at a cost to measured performance. For instance, in our network, we achieve this by abstaining from calculating the outputs of the 2nd, 3rd, and 4th layers during every even inference. To maintain causality, the transposed convolution output must be temporally shifted to produce either current and future frames or solely future frames, depending on the chosen SOI inference mode. Additionally, we advocate for the use of a skip connection between the input of the strided convolution and the output of the transposed convolution to update deeper layers of the network with information about the current data. This operation aims to minimize the influence of data forecasting on the optimized part of the network.

To formally describe partially predictive SOI, let's assume that the network contains layer l d with a stride of size 2 4 and layer l u which reverses the downsampling performed by l d . Typically, l d would be in the encoder part, while l u would be in the decoder.

The downsampling layer only returns a new element for even-numbered inferences. It's important to note that until the upsampling layer is reached, there is no need to perform any further computations when t is odd. Namely for l such that l d < l ≤ l u :

(4)

At layer l u we reconstruct the output by duplicating the convolution output.

Note that l u X T 2 s = l u X T 2 s +1 .

Wethen concatenate the output with data from the skip connection ( ( · | c · ) represents the concatenation of vectors in channel axis).

In the example above we traded inference over l u -l d layers at the cost of inference over additional channels from skip connection concatenation. The additional cost of skip connection does not take effect in architectures where skip connections naturally exist like U-Net which we use as one of the examples in this study.

For fully predictive variant we modify equation (5) and add a shift in time axis.

lX

t-2

t-1

Id

t-1

Currentframe

Bufferedframe

t-1

Copiedframe

t-2

t-1

t

Removedframe

t-2

t-1

SOI can be seen as a form of forecasting where instead of training the model to predict output for input yet unknown, we predict partial states of the network. This is because preserving causality in

the strided convolution forces us to predict a convolution results when the next second input is not yet available.

3 Experiments

3.1 Speech Separation

We selected speech separation as our first experimental task. The choice of the task is dictated by our current research interests and the potential benefits of fast online inference. In this task we focus on separating clean speech signal from noisy background. In literature this task is also referred to as speech denoising or noise suppression.

For this experiment we adopted the U-Net architecture as it is widely used for this specific task and inherently has skip connections which will allow for applying SOI inference pattern without substantial alterations. Our model is composed of 7 encoder and 7 decoder layers. The Deep Noise Suppression (DNS) Challenge - Interspeech 2020 dataset (Reddy et al., 2020), licensed under CC-BY 4.0, was used for both training and evaluation purposes.

Position of S-CC pair By introducing S-CC pair to the network we are enforcing data predictiveness of the network. The exact number of the predicted future partial states of the model depends on the position of S-CC pair and number of those pairs within the network. In addition it is worth noting that larger amount of predicted partial states of the network leads to higher reduction of computational complexity. In this experiment we test every position of S-CC pairs while applying up to two such modules to the model's architecture.

Position of SS-CC pair SS-CC pair introduces additional shift in time axis compered to S-CC pair. In this experiment we alter the position of SS-CC pair within the network and also separately alter the position of S-CC pair and time shift which might be consider as a hybrid of partially and fully predictive pattern.

Resampling Simple resampling of audio signal may be used to reduce the number of calculations done by neural networks but will yield significant increase in model latency. Nonetheless in this experiment we compare SOI to four different resampling methods: SoX which is based on method proposed by Soras (2004), using Kaiser window, polyphase and linear. With this methods we resampled our speech separation dataset from 16k to 8k at the input of the model and then from 8k to 16k at the output of the model. For every resampling we used our baseline model and compared it to three selected SOI models.

Pruning In this experiment we used unstructured global magnitude pruning (similar to Han et al. (2015)) to show how our method can be combined with pruning leading to better results than pruning on a standard model. For this experiment we chose to prune 'SOI 1' and 'SOI 2 | 6' variants of our baseline model. Each step we pruned 4096 weights from model and we report how models performed on each step.

3.2 Acoustic Scene Classification

Acoustic scene classification (ASC) is our second task of choice. The goal of the task is to estimate the location where the specific sample was recorded. This task is commonly considered as an auxiliary problem in various online scenarios such as selection of bank of denoising filters.

For all our tests in ASC task we used GhostNet architecture (Han et al., 2020). We tested 7 different model sizes for all 3 variants - Baseline, STMC and SOI. Each test was repeated 5 times. We used the TAU Urban Acoustic Scene 2020 Mobile dataset (Heittola et al., 2020) for both training and validation. Models were trained on a single Nvidia P40 GPU for 500 epochs using Adam optimizer with initial learning rate of 1e-3.

4 Results

4.1 Speech Separation

Partially Predictive Results of partially predictive SOI in speech separation task are shown in figure 4. We tested variants with a single S-CC layer ('S-CC') and two S-CC layers ('2xS-CC X '). In the latter case, the experiments are grouped by the position of the first S-CC layer ( X ). Value of SI-SNRi metric is highly dependant on the position of S-CC layers. Generally, the earlier the S-CC layer is introduced the lower SI-SNRi but higher complexity reduction. This phenomena may be explained by the difficulty of the partial state prediction task. All layers subsequent to the S-CC layer are required to predict twice the number of elements compared to the layers without having the S-CC as an prior layer. Overall, with SOI we can achieve linear dependency between computation cost and SI-SNRi up to 64% of complexity reduction (with linearity factor of 0.001 dB SI-SNRi per 1 MMAC/s (0.017 dB SI-SNRi per 1% of reduction)). A significant SI-SNRi drop is observed if S-CC layer is introduced too early. In table 1 we present selected results from the whole experiment.

Fully Predictive Results for speech separation using fully predictive SOI are presented in figure 5. Details for selected models can be found in table 2. Reduction of metrics observed for fully predictive variant tends to be larger than for a single S-CC layer in partially predictive SOI. Additional reduction of metrics stems from added shift in time axis. Added shift produces a pattern where some part of model can be updated between inferences, as it only depends on past data. Size of this model part depends on the position of an SC or SS-CC layer within network. We refer to the size of this model part as 'Precomputed' in table 2. The increase of precomputed partial state leads to reduction of SI-SNRi metric but allows to speed-up computation, because this part can be computed in advance. With FP SOI we achieved 50% reduction of computational cost at the expense of 11.3% of SI-SNRi but with 83.7% of network calculated using past data only.

MMAC/s

363.84

727.67

1091.51

1455.34

1819.18

亚

SI-SNRi [dB]

STMC

2xS-CC 2

6

Predictive1

2xS-CC 3

Predictive2

2xS-CC 4

S-CC

2xS-CC 5

2xS-CC6

5

0.2

0.4

0.6

0.8

1.0

Complexityreductionfactor

PrecomputedMMAC/s

363.84

727.67

1091.51

1455.34

1819.18

SI-SNRi(dB)

/

STMC

-

SOI3

Predictive1

SOI 4

Predictive2

IIOS

SO16

SOI2

-

SOI7

b

0.2

0.4

0.6

0.8

1.0

Precomputedfactor

Resampling The results presented in the table 3 show that SOI outperforms the listed resampling methods in terms of complexity reduction under the constrained of preserving the quality of the original model (STMC). The results for resampling-based complexity reduction indicate that model's quality depends on the used resampling algorithm and, for all the selected ones, the initial information loss strongly affects the model performance.

Complexity(MMAC/s)

363.84

727.67

1091.51

1455.34

1819.18

8

SI-SNRi (dB)

6

4

STMC

2

IOS

2xSOI

0

0.2

0.4

0.6

0.8

1.0

Peakcomplexityreductionfactor

Pruning The application of SOI along with the pruning of STMC model surpasses the effect of application of the pruning alone. The addition of SOI allowed for achieving a further reduction in computational complexity by around 300 MMAC/s for the same model's performance, which is about 16% of the original model. Interestingly the 'SOI 2|6' surpassed the 'SOI 1' model at around 6 dB SI-SNRi. The results of the experiment are shown in Fig. 6.

4.2 Acoustic Scene Classification

Results for the ASC task are collected in Table 4. For models I, III and VII we observed that our method led to an increase in accuracy, whereas other models showed a the decrease in metrics. The largest decrease in accuracy was around 2.20%, and the largest increase was 1.69%. These results indicate that SOI does not lead to a decrease in model quality for this particular task compared to the STMC model. This can be explained by the much slower change of output (acoustic scene label) compared to the previous task (speech mask). In our tests, the reduction in computational complexity of SOI models amounted to around 16% compared to the STMC model. This reduction decreased to 11% for the smallest model due to the addition of the skip connections. For this particular architecture, our method also reduced the number of parameters for the largest tested models by around 7%.

5 Conclusion

In this work, we presented a method for reducing the computational cost of a convolutional neural network by reusing network partial states from previous inferences, leading to a generalization of these states over longer time periods. We discussed the effect of partial state prediction that our method imposes on the neural model and demonstrated that it can be applied gradually to balance model quality metrics and computational cost.

Our experiments highlight the high potential for computational cost reduction of a CNN, especially for tasks where the output remains relatively constant, such as event detection or classification. We achieved a computational cost reduction of 50% without any drop in metrics in the ASC task and a 64.4% reduction in computational cost with a relatively small reduction of 9.8% in metrics for the speech separation task. We also showcased the ability of SOI to control the trade-off between model's quality and computational cost, allowing for resource- and requirement-aware tuning.

The presented method offers an alternative to the STMC solution for strided convolution. While SOI reduces network computational complexity at the expense of measured performance, STMC ensures that metrics are not reduced but at the cost of increased memory consumption at an exponential rate. SOI is akin to methods like network pruning, but unlike pruning, it does not require special sparse kernels for inference optimization. It is worth noting that these methods are not mutually exclusive, therefore, the STMC strided convolution handler, SOI, and pruning can coexist within a neural network to achieve the desired model performance.

References

Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science , 378(6624):1067-1074, 2022. doi: 10.1126/science.ade9097.

Vijay Balasubramanian. Brain power. Proceedings of the National Academy of Sciences , 118(32): e2107022118, 2021. doi: 10.1073/pnas.2107022118. URL https://www.pnas.org/doi/abs/ 10.1073/pnas.2107022118 .

Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning Volume 70 , ICML'17, pp. 527-536. JMLR.org, 2017.

Víctor Campos, Brendan Jou, Xavier Giró-i-Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: learning to skip state updates in recurrent neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018. URL https://openreview.net/forum?id=HkwVAXyCW .

Jiaming Gong, Wei Liu, Mengjie Pei, Chengchao Wu, and Liufei Guo. Resnet10: A lightweight residual network for remote sensing image classification. In 2022 14th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA) , pp. 975-978, 2022. doi: 10.1109/ICMTMA54903.2022.00197.

R.M. Gray and D.L. Neuhoff. Quantization. IEEE Transactions on Information Theory , 44(6): 2325-2383, 1998. doi: 10.1109/18.720541.

Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2020.

Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 , NIPS'15, pp. 1135-1143, Cambridge, MA, USA, 2015. MIT Press.

Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Lioma. Neural speed reading with structural-jump-LSTM. In International Conference on Learning Representations , 2019. URL https://openreview.net/forum?id=B1xf9jAqFQ .

Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 770-778, 2015. URL https://api.semanticscholar.org/CorpusID:206594692 .

Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. TAU Urban Acoustic Scenes 2020 Mobile, Development dataset. Zenodo, February 2020. doi: 10.5281/zenodo.3819968. URL https://doi.org/10.5281/zenodo.3819968 .

Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification. In International Conference on Learning Representations , 2017.

Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. In International Conference on Learning Representations , 2017. URL https://openreview.net/forum?id=S1LVSrcge .

Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. In Proceedings of the International Conference on Computer Vision (ICCV) , 2011.

Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 , 2019.

Simon B. Laughlin, Rob R. de Ruyter van Steveninck, and John C. Anderson. The metabolic cost of neural information. Nature Neuroscience , 1(1):36-41, May 1998. ISSN 1546-1726. doi: 10.1038/236. URL https://doi.org/10.1038/236 .

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky (ed.), Advances in Neural Information Processing Systems , volume 2. MorganKaufmann, 1989. URL https://proceedings.neurips.cc/paper/1989/file/ 6c9882bbac1c7093bd25041881277658-Paper.pdf .

Charles E. Leiserson, Neil C. Thompson, Joel S. Emer, Bradley C. Kuszmaul, Butler W. Lampson, Daniel Sanchez, and Tao B. Schardl. There's plenty of room at the top: What will drive computer performance after moore's law? Science , 368(6495):eaam9744, 2020. doi: 10.1126/science. aam9744.

Gordon E Moore. Cramming more components onto integrated circuits. Proceedings of the IEEE , 86 (1):82-85, 1998.

Thai Son Nguyen, Sebastian Stueker, and Alexander H. Waibel. Super-human performance in online low-latency recognition of conversational speech. In Interspeech , 2020.

David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2104.10350 , 2021.

Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The interspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. In Proc. Interspeech 2020 , 2020.

Michal Romaniuk, Piotr Masztalski, Karol Piaskowski, and Mateusz Matuszewski. Efficient LowLatency Speech Enhancement with Mobile Audio Streaming Networks. In Proc. Interspeech 2020 , pp. 3296-3300, 2020. doi: 10.21437/Interspeech.2020-2443.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, dec 2020. doi: 10.1038/s41586-020-03051-4. URL https://doi.org/10. 1038%2Fs41586-020-03051-4 .

Biswa Sengupta and Martin B. Stemmler. Power consumption during neuronal computation. Proceedings of the IEEE , 102(5):738-750, 2014. doi: 10.1109/JPROC.2014.2307755.

Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. Neural speed reading via skim-RNN. In International Conference on Learning Representations , 2018. URL https://openreview. net/forum?id=Sy-dQG-Rb .

Laurent Soras. The quest for the perfect resampler. 02 2004.

Grzegorz Stefa´ nski, Krzysztof Arendt, Paweł Daniluk, Bartłomiej Jasik, and Artur Szumaczuk. Shortterm memory convolutions. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=4DU_HCijfJp .

Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR) , pp. 2464-2469, 2016. doi: 10.1109/ICPR.2016.7900006.

Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In European Conference on Computer Vision (ECCV) , 2018.

Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII , volume 11217 of Lecture Notes in Computer Science , pp. 420-436. Springer, 2018. doi: 10.1007/978-3-030-01261-8\_25. URL https://doi.org/10.1007/978-3-030-01261-8_25 .

Xiaowei Xu, Yukun Ding, Sharon Xiaobo Hu, Michael Niemier, Jason Cong, Yu Hu, and Yiyu Shi. Scaling for edge inference of deep neural networks. Nature Electronics 2018 1:4 , 1:216-222, 4 2018. ISSN 2520-1131. doi: 10.1038/s41928-018-0059-3. URL https://www.nature.com/ articles/s41928-018-0059-3 .

Adams Wei Yu, Hongrae Lee, and Quoc Le. Learning to skim text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp. 1880-1890, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1172. URL https://aclanthology.org/P17-1172 .

A Reproducibility Notes

A.1 Speech Separation

For this experiment we adopted the U-Net architecture as it is widely used for this specific task and inherently has skip connections which will allow for applying SOI inference pattern without substantial alterations. Our model is composed of 7 encoder and 7 decoder layers, each comprising STMC/tSTMC, batch norm and ELU activation layers. Each model was trained for 100 epochs using Adam optimizer with initial learning rate of 1e-3. We trained each model on a single Nvidia P40 GPU 5 times and reported the average SI-SNRi. The mean training time of a single model amounted to about 14 hours. The Deep Noise Suppression (DNS) Challenge - Interspeech 2020 dataset (Reddy et al., 2020), licensed under CC-BY 4.0, was used for both training and evaluation purposes. For training, we used 16384 10s samples without any form of augmentation and for both validation and test sets we used 64 samples with similar setup to the training set.

A.2 Acoustic Scene Classification

For all our tests in ASC task we used GhostNet architecture (Han et al., 2020). Our baseline model is the original architecture with 'same' padding making it not applicable in online scenario. STMC model changes the padding to manually-applied padding in left-most (oldest) side of data and applies STMC inference pattern. SOI model adds upsampling after each processing block and skip connections between downsampling/upsampling layers.

Models were trained on a single Nvidia P40 GPU for 500 epochs using Adam optimizer with initial learning rate of 1e-3. We tested 7 different model sizes for all 3 variants - Baseline, STMC and SOI. Each test was repeated 5 times. The mean training time of a single model amounted to about 4 hours. We used the TAU Urban Acoustic Scene 2020 Mobile dataset (Heittola et al., 2020) for both training and validation.

B Strided Convolutions are Better for Longer Predictions

In this experiment, we investigated the impact of strided convolutions on predictive inference. Our test environment consisted of a U-Net model applied to a speech separation task. We examined two model variants: Predictive and Strided Predictive . The Predictive model serves as our baseline U-Net with an added time shift at the end. The Strided Predictive model, in addition to the time shift, incorporates strided convolutions in place of standard ones. For each model, we conducted tests with four different lengths of prediction, ranging from 1 frame to 4 frames. We performed five training runs for each model using the setup outlined in Section 3.1. The results of this experiment are displayed in Figure 7 and Table 5.

Predictive

-至-

Stridedpredictive

(dB）

SI-SNRi (

4

一

2

3

4

Prediction size (frame)

We conclude that strided convolutions shows higher potential for longer predictions. We attribute this effect to the fact, that using strides forces stronger generalization of outputs of strided convolutions because they are applied in multiple contexts.

C Inference Time and Peak Memory Footprint

To showcase SOI's influence on inference time and peak memory consumption, we extended our results from Table 1 to include these measurements with a single S-CC layer. The collected results are presented in Table 6.

Additionally, in Figure 8, we show how inference time and peak memory consumption depend on SI-SNRi and the complexity reduction factor.

A

B

ET

7.75

7.75

i(dB)

SI-SNRi (dB)

7.50

7.50

SI-SNRi (

7.25

7.25

7.00

++

STMC

S-CC

7.00

STMC

-

S-CC

6

8

10

15

20

25

Avg.inference time (ms)

Peak memory footprint (MB)

MMAC/s

D

MMAC/s

1091.51

1455.34

1819.18

1091.51

1455.34

1819.18

Peak memory footprint (MB)

Avg. inference time (ms)

10

25

8

20

6

STMC

--S-CC

15

STMC

S-CC

0.6

0.8

1.0

0.6

0.8

1.0

Complexity reduction factor

Complexity reduction factor

These results suggest that the inference time of the tested model depends linearly on SI-SNRi and the complexity reduction factor, while peak memory consumption has significant impact on SI-SNRi at the beginning. From 60% complexity reduction onward, the peak memory footprint decreases sharply.

D Interpolation

Our method may also benefit from the usage of interpolation methods in place of extrapolation. We tested a singular S-CC layer with three different types of interpolation: nearest-neighbor, bilinear, and bicubic. The results of this experiment can be observed in Figure 9 and Table 7. For comparison, we included our extrapolated duplication method in the results of this experiment.

In this experiment, we achieved the best results with bilinear or bicubic interpolation, although the results for bilinear interpolation showed much higher variance than any other method. It is important to remember that

7.8

7.6

SI-SNRi [dB]

7.4

STMC

7.2

Nearest-neighbor

Bilinear

Bicubic

7.0

Duplication

2

4

6

S-CC layer position

even though we achieve slightly better results using interpolation, the usage of interpolation comes at the cost of higher latency as we need to wait for an additional time frame.

E Different Extrapolation Methods

In the main paper we used element duplication as an extrapolation method but we also pointed out that any type of extrapolation may be used. Here we compare the results of our speech separation experiment using element duplication with another commonly found method - a transposed convolution. We tested both partially predictive and fully predictive modes using the setup described in Section 3.1.

Results of the experiment with PP SOI are presented in figure 10. In this case we tested a U-Net with two S-CC layers. Each plot represents a different position of the first S-CC pair and each point on X-axis represents a different position of the second S-CC pair. We also present results for hybrid models where duplication and transposed convolutions were used together - in the first and second S-CC pairs respectively.

In this experiment neither method demonstrated a significant advantage. It seems that duplication tends to perform better than transposed convolution if it is introduced deeper within the network and vice versa although difference is marginal.

—Duplication

Transposed Convolution

Hybrid

7.0

7.0

7.0

6.0

6.5

6.5

SI-SNRi (dB)

2

4

6

4

6

4

5

6

7

7.4

7.2

7.4

7.2

6.8

7.2

5

6

6

7

6

7

8

2nd S-CC layer position

Results achieved with FP SOI are shown in figure 11. Each plot represents a different position of S-CC pair and each point represents different position of SC layer. Achieved results confirm previous conclusions.

This experiment proves that element duplication is a viable method and thus should be chosen for its simplicity.

Duplication

Transposed Convolution

7.5

7.2

7.0

7.0

6.5

6.8

SI-SNRi (dB)

6.5

2

4

6

2

4

6

4

6

7.5

7.5

7.6

7.2

7.2

7.2

7.4

6.9

4

6

5

6

7

6

7

6

7

8

SC layer position

F Video Action Recognition

Other domains can benefit from SOI, as it can be applied to any time-series data. To illustrate this, we conducted experiments using SOI for an action recognition task with video data. We utilized the HMDB-51 dataset Kuehne et al. (2011), which contains 24 fps video data of human actions split into 51 classes. We trained a popular ResNet-10 architecture Gong et al. (2022) in three variants: regular, small (with halved number of channels) and tiny (with number of channels reduced fourfold), where we replaced 3D convolutional layers with 3D STMC layers. Here, we applied SOI by introducing a skip connection between the output of block 2 and the input of block 4, optimizing block 3.

To demonstrate that SOI can work not only with STMC, we also applied it to MoViNets, which use a method called 'Stream buffers'. We trained two variants of MoViNets, A0 and A1, in their streaming form. SOI was applied by optimizing blocks 4 and 5. Please note that SOI can be used not only with 3D convolutions but also with their popular 2D+1 variant.

All models were trained for 100 epochs with Adam optimizer and learning rate 5e-5. Each model was trained on Nvidia A100 GPU with batch size of 16. The mean training time amounted to about about 37h.

Achieved results suggest that SOI is suitable for video domain as well. ResNet-10 architecture proved to be highly compatible with SOI for this task as results achieved with SOI variant of this model outperformed the regular ones. We believe that this improvement was imposed by the increase in receptive field as SOI adds additional strided convolution. The achieved reduction of complexity for this family of models was between 10-17% depending on model size. For MoViNets, the decrease of around 3% in accuracy can be spotted, although the imposed complexity reduction was higher compared to ResNet and amounted to 23-30%.

G Acoustic Scene Classification with ResNet

To further evaluate the effectiveness of our method, we conducted experiments on the accoustic scene classification task using ResNet architecture He et al. (2015). We chose ResNet for two primary reasons: (1) its widespread use and recognition as a standard deep learning model, and (2) its relatively large size compared to other architectures tested, such as GhostNet, which allows us to better assess how SOI performs on larger models.

We tested 4 different ResNet models for all 3 variants - Baseline, STMC and SOI. Each test was repeated 5 times. We used the TAU Urban Acoustic Scene 2020 Mobile dataset (Heittola et al., 2020) for both training and validation.

The results are presented in Table 11.

The results, as shown in Table 11, demonstrate that the SOI-enhanced ResNets consistently outperformed the baseline models in terms of accuracy. Specifically, the SOI-based models achieved higher classification accuracy across all tested configurations. We belived this improvement can be attributed to two key factors:

The introduction of SOI allows the model to generalize over longer time frames by expanding its receptive field. This helps the model capture more temporal context, which is crucial for tasks like ASC.

SOI encourages the model to generalize its output states by predicting future partial states, which improves the model's ability to handle variations in the data.

Despite ResNet's depth and complexity, the method proved successful, not only reducing computational cost but also improving performance.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations of proposed method are discussed in section 1.3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Reproducibility notes are present in section A.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We cannot open-source our codebase due to its commercial purpose.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see section A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Results are accompanied by error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see section A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper presents a general method for reduction of computational complexity of neural networks.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Datasets and their accompanying licenses are listed in section A.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper presents a method called Scattered Online Inference (SOI) aimed at reducing the computational complexity of ANNs. By applying compression and extrapolation techniques, SOI caches partial states of CNNs, allowing it to skip full model recalculation at each inference. The proposed method is positioned as a solution for real-time systems, where energy efficiency and reduced latency are critical.

STRENGTHS:
1. The research problem is important. The focus on real-time systems and applications in consumer electronics, where computational resources are limited, adds practical value to the research.

2. The extensive experiments on audio tasks indicate the effectiveness of the proposed method.

WEAKNESSES:
The presentation of this work is not very good: 

First, the proposed method is heavily based on STMC, but STMC is not well explained in this manuscript. 

Second, there is no theoretical justification of how much the computational complexity will decrease, making it difficult for reviewers to assess the trade-off between accuracy and complexity.

Third, the Mathematical Formulation part is not well-described. i).Fig. 3 is just put there without any explaination and reference to it. ii). There are too many words but not clear formulae to explain the process. iii). The notation is a little bit strange.

QUESTIONS:
See weaknesses.

LIMITATIONS:
1. The proposed method only focuses on CNNs. Extending it to Transformer architectures, which are increasingly popular, presents significant challenges.

2. This manuscript does not discuss the method's limitation, although the authors claim that the discussion is in sec. 1.2.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method called Scattered Online Inference (SOI) aimed at reducing the computational cost of convolutional neural networks (CNNs) by reusing partial states from previous inferences. The method generalizes these states over longer time periods, balancing computational efficiency and model performance. SOI leverages techniques such as strided convolution, transposed convolution, and skip connections to achieve significant reductions in computational complexity without substantial loss of accuracy. The paper presents extensive experimental results demonstrating the effectiveness of SOI in two tasks: speech separation and acoustic scene classification. In the speech separation task, the method achieved a 64.4% reduction in computational cost with only a 9.8% decrease in SI-SNRi. In the acoustic scene classification task, a 50% reduction in computational cost was achieved without any drop in accuracy. The SOI method also showcases the ability to control the trade-off between model quality and computational cost, allowing for resource- and requirement-aware tuning.\n\nSTRENGTHS:\n### Strengths\n\nThe paper introduces the novel Scattered Online Inference (SOI) method, which reduces the computational complexity of convolutional neural networks (CNNs) by reusing partial states from previous inferences. This originality is evident in its creative combination of strided and transposed convolutions with new techniques for managing computational resources. The paper is methodologically sound, with thorough experimental evaluations showing significant computational cost reductions without substantial performance loss. The writing is clear and well-structured, making complex concepts accessible. The significance of this work lies in its applicability to resource-constrained environments and real-time applications, offering a valuable alternative to existing methods like STMC.\n\nWEAKNESSES:\n### Weaknesses\n\nWhile the paper presents a novel approach, there are several areas where it could be improved:\n\n1. **Early Layer Impact on SI-SNRi**: The paper notes a significant drop in SI-SNRi when the S-CC layer is introduced too early in the network. This suggests that the method may not be robust across different network configurations. The authors could improve this by exploring adaptive strategies for placing S-CC layers based on the specific characteristics of the task and dataset.\n\n2. **Complexity of Implementation**: The SOI method involves multiple steps and different types of convolution operations, which may make implementation complex and potentially error-prone. Providing a more detailed implementation guide or open-sourcing the code could help alleviate this concern and improve reproducibility.\n\n3. **Limited Scope of Evaluation**: The experiments focus primarily on speech separation and acoustic scene classification. While these are relevant tasks, the generalizability of SOI to other types of tasks and datasets is not thoroughly explored. Expanding the evaluation to include more diverse tasks and datasets could strengthen the paper’s claims about the broad applicability of SOI.\n\n4. **Quantitative Analysis of Trade-offs**: The paper mentions the trade-off between model quality and computational cost but lacks a detailed quantitative analysis of this trade-off across different configurations. Including more comprehensive metrics and visualizations of these trade-offs could provide deeper insights and help practitioners make more informed decisions when applying SOI.\n\n5. **Parameter Sensitivity**: The paper does not discuss the sensitivity of SOI’s performance to various hyperparameters, such as the number and position of S-CC and SS-CC layers. A sensitivity analysis could reveal potential limitations and guide users in tuning these parameters for optimal performance.\n\nAddressing these weaknesses could significantly enhance the impact and usability of the proposed SOI method.\n\nQUESTIONS:\nPlease refer to weakness.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
The authors propose a novel method that reduces the computational cost of regular ANN models to achieve better online inference efficiency. The core technique named Scattered Online Inference (SOI) is able to reduce computational cost with partial predictions.

STRENGTHS:
- The proposed method is designed to avoid redundant computation in convolution layers.
- Brings little learning quality drop.
- Extensive experiments.

WEAKNESSES:
- The organization of this paper makes it very difficult for readers to appreciate the true merit of this paper.
- It is unclear which part is reusing STMC and which part is uniquely proposed by SOI.
- No backgrounds are provided for STMC.
- To show that the proposed algorithm achieves higher efficiency, peak memory usage and latency should be provided.

QUESTIONS:
- What are the removed frames? How to decide the removal?
- A better explanation of Figure 2 C&D could be provided in conjunction with section 2.2. Which frames are predicted in C&D?

LIMITATIONS:
Limitations are discussed in the paper. The proposed technique is mainly limited to time-series data.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a novel method called Scattered Online Inference (SOI) aimed at reducing the computational cost of convolutional neural networks (CNNs). This method leverages the reuse of network partial states from previous inferences, thereby generalizing these states over extended periods. SOI enables extrapolation, particularly enhancing processing speed in deeper layers. By applying compression techniques, SOI generates more generalized inner partial states within the CNN, allowing the system to skip full model recalculations for each inference.\n\nTheir contributions include offering a distinct treatment of strided layers, resulting in a new and efficient inference pattern. This approach modifies the inference pattern of the network to skip the recalculation of certain layers according to a predetermined scheme. Their method significantly reduces the computational cost of CNN model with only a negligible decrease in performance. These optimizations are achieved with minimal changes to the architecture, making the method suitable for tasks where reducing energy consumption or processing time is crucial.\n\nSTRENGTHS:\nFrom my perspective, authors have touched each dimesnion of originality, quality, clarity, and significance.\n\n1.) Their proposed SOI approach is inspired and built upon Short-Term Memory Convolution approach.\n\n2.) In order to support their approach, they have performed extensive experiements with different architectures (U-Net, GhostNet and their variants) on two benchmark datasets: Interspeech 2020 dataset and TAU Urban Acoustic Scene 2020 Mobile dataset and also compared their reuslts with state-of-the-art model.\n\n3.) They achieved a computational cost reduction of 50% without any drop in metrics in the acoustic scene classification task and a 64.4% reduction in computational cost for the speech separation task.\n\n4.) Through extensive experiments, they showed the ability of SOI to control the trade-off between model’s quality and computational cost, allowing for resource- and requirement-aware tuning.\n\n5.) They have shared all the information regarding experiments and each hyperparameter used that will be useful to reproduce the results.\n\nWEAKNESSES:\nThe limitation of paper is discussed in one line in subsection 1.2 that it incurs a neglible decrease in model's performance. From my perspective, it should have been more explanatory. For speech separation task, there was 9.8% of reduction in  metrics. I think, this reduction is not negligible. I believe it would have been better if authors had clarified or mentioned more supporting statements regarding the results.\n\nQUESTIONS:\nI also would like to see different versions of the ResNet model because they too have skip connections and how much computational cost gets reduced using SOI?\n\nLIMITATIONS:\nnot clearly but yes, up to certain degree", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}