SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model
Authors: Grzegorz Stefański
Paweł Daniluk
Artur Szumaczuk
Jakub Tkaczuk
Venue: NeurIPS 2024 poster (2024)
Keywords: Time series data
Computational complexity reduction
Latency reduction
Real-Time results
Inference at the edge
Causality
Primary area: speech_and_audio

Abstract: Consumer electronics used to follow the miniaturization trend described by Moore’s Law. Despite increased processing power in Microcontroller Units (MCUs), MCUs used in the smallest appliances are still not capable of running even moderately big, state-of-the-art artificial neural networks (ANNs) especially in time-sensitive scenarios. In this work, we present a novel method called Scattered Online Inference (SOI) that aims to reduce the computational complexity of ANNs. SOI leverages the continuity and seasonality of time-series data and model predictions, enabling extrapolation for processing speed improvements, particularly in deeper layers. By applying compression, SOI generates more general inner partial states of ANN, allowing skipping full model recalculation at each inference.

SOI: Scaling Down Computational Complexity by Estimating Partial States of the Model
Grzegorz Stefański', Pawel Daniluk², Artur Szumaczuk', Jakub Tkaczuk2
'Samsung Al Center Warsaw 'Samsung R&D Institute Poland {g stesffans,k, p daniluk, a. szumaczuu,, j tkaczuk) Asazzuugg.aammmgg..coo com
Abstract
Consumer electronics used follow the miniaturization trend described by Moore's Law. Despite increased processing power in Microcontroller Units (MCUs), MCUs used in the smallest appliances are still not capable of running even moderately big, state-of-the-arr artificial neural networks (ANNs) especially in time-sensitive sce- narios. In this work, we present a novel method called Scattered Online Inference (SOI) that aims to reduce the computational complexity of ANNs. SOI leverages the continuity and seasonality of time-series data and model predictions, enabling extrapolation for processing speed improvements, particularly in deeper layers. By applying compression, SOI generates more general inner partial states of ANN, allowing skipping full model recalculation at each inference.
Introduction
Moore's Law (Moore, 1998) predicts that the number of transistors on microchip will double every years. As the number of transistors increases, computational capacity of systems-on-chip (SoCs) also grows. Accompanied by the further miniaturization of components, the increased complexity of SoCs allows for the production more compact appliances with equal or higher capabilities.
Despite the continuous development of more advanced hardware, the smallest appliances still cannot fully benefit from increasingly popular neural-based solutions, they are not able to run them efficiently. Examples of such appliances include wireless in-ear headphones, smart watches, and AR glasses. Furthermore, the size of state-of-the-art art neural networks is growing much faster rate than that described by Moore's 's Law (Xu et al., 2018). Adding to the challenge, according to researchers (Leiserson al., 2020), Moore's Law is starting to decelerate due to the physical constraints of semiconductors, and the "room at the bottom" is depleting.
Currently available neural network technologies enable machines to outperform humans in numerous applications in terms of measured performance (Nguyen et al., 2020; Bakhtin et al., 2022; Schrit- twieser et al., 2020). However, despite this achievement, the same models fall significantly short when it comes to energy efficiency compared to humans. The human brain consumes a mere 20 Watts of power (Laughlin et al., 1998; Sengupta & Stemmler, 2014: Balasubramaniaan, 2021) and according to Xu et al. (2018) it is estimated to be over five orders of magnitude more energy efficient than modern neural networks.
This disparity in energy efficiency can be attributed to the common pursuit of the highest model quality in the literature, showcasing the full capabilities of the developed technology, often the expense of efficiency. This behavior is justifiable due to the ease of comparing different solutions using well-defined metrics that are independent of the hardware and software. Conversely, estimating a model's energy efficiency is a more complex task, influenced by various factors including the running environment.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, this trend within our community may prove restrictive, particularly for applications like real- time systems. These applications naturally demand optimal performance alongside high-efficiency solutions, rendering most current Deep Neural Networks (DNNs) impractical for such tasks. Further- more, due to the substantial discrepancy between assumptions for high-efficiency on-device solutions and high-perfooee terh-rrnforiing monolithic models, compressing these models may not be a viable means of applying state-of-the-ar DNNs to real-time tasks.
The importance of neural system efficiency is also increasingly significant from ecological and economic standpoints (Lacoste al., 2019; Patterson et al., 2021).
1.1 Related Works
The Short-Term Memory Convolution (STMC) (Stefański et al., 2023) was devised to enhance the efficiency of Convolutional Neural Networks (CNNs) inference by reducing computation requirements at each step via eliminating the need to recompute prior states. The authors achieved notable 5.68- fold reduction in inference time and a 30% decrease in memory consumption compared to the Incremental Inference method (Romaniuk et al., 2020). STMC enables the conversion of a standard CNN model, which typically requires an input of size at least as large as its receptive field, into a model capable of processing a single input frame at a time, akin to Long-Shop Short Term Memory networks (LSTM). The SOI method is built upon the STMC foundation, offering distinct treatment of strided layers and yielding a compelling new inference pattern'.
Routing methods constitute popular category of algorithms tailored to optimize the inference process of Recurrent Neural Networks (RNNss. In the field of Natural Language Processing (NLP), Yu et al. (2017) introduced an approach that involves traversing segments of the computational graph. This traversal is guided by decisions made by reinforcement learning model following the processing of a fixed number of words. Another notable contribution by Campos et al. (2018) yielded an algorithm capable of selectively bypassing partial state updates within an RNN during inference, influenced by the input's length. In NLP terms, this concept can be compared to the mechanism of skipping words.
The research by Jernite et al. (2017) introduced distinct method to regulate computation within a recurrent unit. This method relied on a scheduler unit that facilitated partial updates to the network's s state, only when the computational budget limit was reached. Meanwhile, Seo et al. (2018) proposed an approach referred to as "word skimming". In this approach, the authors designed both small and large RNN models that could be interchangeably utilized for inference through the utilization of Gumbel softmax. The exploration of hybrid techniques combining jumping, skimming, and skipping was further advanced by Hansen et al. (2019), who published additional solutions in this direction.
The RNN routing methods have found successful applications in CNNs as well. Wang et al. (2018) introduced an approach that enables model to learn policy for skipping entire convolutional layers on per-input basis. A similar concept, involving adaptive inference graphs conditioned on image inputs, was put forth by Veit & Belongie (2018). Additionally, several authors have contributed methods for early-exit during CNN inference (Bolukbasi et al., 2017: Teerapitayanon et al., 2016; Huang et al., 2017). In these methods, the network trained skip portions of the computational graph towards the final stages, based on the characteristics of the input.
Other commonly employed methods for model optimization include pruning (LeCun et al., 1989) and quantization (Gray& & Neuhoff, 1998). Importantly, both of these methods are not mutually exclusive and can coexist alongside routing methods within a single neural network.
1.2 Novelty
In this study, we introduce a method for reducing the computational cost of a ANN model while incurring only negligible decrease in the model's performance. Importantly, these reductions are achieved with minimal alterations to the architecture, making it suitable for various tasks where factors such as energy consumption time are of paramount importance.
Our approach involves the conversion of a conventional ANN model, initially trained to process segments of time-series data with arbitrary lengths in an offline mode, into model that processes
By the term of an inference pattern we mean a full computational graph of single inference or of repeating sequence of inferences if they influence the computational graphs of the following model executions in case of SOI STMC.
2

---

the data element by element, enabling real-time usage. Notably, our method builds upon the STMC technique (Stefañski al., 2023). STMC designed perform each distinct operation exactly once. SOI extends this concept by omitting some operations in a structured manner.
In this study, introduce novel method named Scattered Online Inference (SOI), which is based on the following key principles:
The reduction of computational complexity is achieved through the implementation of partial predictions of the network's future state.
SOI operates two-phase system. The initial phase involves compressing data within the time domain using data compression. The subsequent phase focuses on data reconstruction, employing the most suitable extrapolation scheme.
The method preserves the causal nature of the optimized network architecture.
SOT's applicability is confined single-frame online inference and it necessitates the incorporation of skip connections to update the network's output following each inference.
1.3 Limitations
While our method, demonstrates significant advantages in reducing computational complexity, is essential to acknowledge several limitations inherent to our approach.
First, the performance reduction observed with SOI, although acceptable within the context of work, may not be tolerable in applications requiring the highest accuracy levels. This reduction, while compensated by substantial reduction in computational cost, suggests trade-off between efficiency and model performance that may limit SOI's applicability in scenarios where performance is the absolute priority.
Second, the flexibility provided by SOI to balance computational cost and model performance introduces complexity in selecting the optimal configuration. The method allows users adjust this trade-off based on application requirements, but this demands careful tuning and validation, which could be ioource-intensive in training.
Third, the method's reliance on partial state predictions and data compression may introduce cumu- lative errors over time, particularly in longer sequences of time-series data. This is less critical in real-time, scout-scquence applications but could degrade performance in tasks requiring continuous operation over extended periods without full model recalculations.
Lastly, SOI's applicability is primarily demonstrated on specific neural network architectures. Al- though we expect SOI to generalize to other architectures, its effectiveness in reducing computational complexity while maintaining acceptable performance may vary depending on the network design, the nature of the task, and the dataset. Further research and experimentation are necessary to explore its utility across broader range of models and applications.
Time
Network
B
Extrepolated
Output convolutional layer
Removed frame
C
D
Figure 1: SOI for convolutional operations. For visualization purposes we show data as frames in time domain. A) Standard convolution. Strided convolution. C) Strided-Clones Convolution. D) Shifted convolution. E) Shifted Strided-Cloned Convolution.

---

Time
Extrapolated frame
Convolutio
Output of convetutiona layer
Skip connection
Bulfered forns
Oopy
Predicted Frame (Precomiuted))
D
Figure 2: Inference patterns of each type of SOI based on U-Net architecture. A) Unmodified causal U-Net. B) Partially predictive (PP) SOL. C) Even inference of PP. D) Odd inference of PP. E) Fully predictive (FP) SOI. F) Even inference of FP. G) Odd inference of FP.
Methods
When processing time-series in online mode, the model goes through each incoming element? separately. In this paper we refer to such an event as inference.
An improvement in computation complexity is achieved by introducing the partially predictive compression layer adapted for online inference, as well as by avoiding the redundant computations done during previous inferences in STMC study. Therefore, after each inference, the results which would recalculated in subsequent runs are cached. We refer to such cacheable outputs as a partial state of the network.
2.1 Scattered Online Inference
Scattered Online Inference (SOT) a method which modifies the inference pattern of a network to skip the recalculation of certain layers in predetermined pattern. This is achieved through the use of compression and extrapolation. Both operations are exclusively applied along the time axis. In this study, example, extrapolation, where the last known value is used to replace the missing ones?. To facilitate better understanding of the SOI algorithm in CNNs, in Figure 1, define three types of convolutional layers utilized our method. For comparison purposes, Figure also includes standard convolution and strided convolution.
employed strided convolutions as compression layers and basic form of
Strided-Clone Co Convolution (S-CC) performs a 2-step operation. Firstly, applies a strided convolu- tion the input in order to compress the data. In the second step it fills the gaps created by striding using any form of extrapolation. In our experiments we extrapolate by simply copying previous frame. The copied frame is then aligned with future frame relatively to its input. In practice, split the stride and extrapolation operations into different layers which results in optimization of computational complexity between those layers. Because of that we will refer to this as S-CC pair.
We define an "element" chunk data arbitrary size, including smallest possible unit a singular data point in time series. The size the incoming data chunk depends on the specific system being used.
More sophisticated methods tested as well and presented in supplementary materials.

---

Shifted Convolution (SC) shifts data in time domain after the convolution thus creating additional predicted network states, This layer may be used for additional latency reduction.
Shifted Strided-Clone -looned Convolution (SS-CC) is a combination of S-CC pair and SC layer which is needed if want to do both of them at the same point of the network. In experiments extrapolate output vector first and then apply data shifting to reduce size of introduced partial state prediction.
SOI can be divided into two types depending on how prediction is handled. These two types have significantly different inference patterns (Fig. 2).
Partially Predictive (PP) In this type of SOI, we do not introduce any additional predictive components in the network. This implies that after compression, the most recent frame stores information for the current output frame and the future output frame. This type of SOI uses S-CC pairs only. This configuration results in only one inference (the first one), which updates all network states, while all subsequent ones use relevant buffered partial network states. Although hybrid setups involving more full passes are viable, they fall outside the scope of this paper. It's important to note that this mode does not reduce peak computational complexity, but rather the average computational complexity.
Fully Predictive (FP) In this type of SOI, we introduce additional predictive components to the network. Compared PP SOI, the most recent frame does not store any information about the current output frame. but rather about two future ones, hence the name "Fully Predictive", as only the future output is calculated. This a more challenging task than PP SOI, but it can significantly decrease the latency of the model. This mode utilizes both S-CC pairs and SC layers. It optimizes both peak computational complexity and latency because it allows some inferences be predicted in full. The fully predicted inference, in contrast to the regular inference which requires newly collected input, operates only on already processed data and can calculate relevant network states while the system awaits the new data, reducing the amount of computation required when the new data arrives.
Both of these types of SOI can be combined. This occurs when first introduce PP SOI compression and then after some number of layers, we introduce an additional shift in the time axis after which the model can be treated as FP SOI.
2.2 Mathematical Formulation
Let us assume that an input of the model is represented by ID time series vector X E R.W composed of N samples. Additionally let's say that network is composed of 5 convolutional layers and output vector 'Y for 1-th layer is of the same shape the input. Each convolutional layer has 1D kernel h e M which can be represented by Toeplitz matrix Hi N-N M,+1), We get:
'Y= Hi H 'x"
(1)
After which apply activation function and get the input for the next layer:
4+ = ('Y)
(2)
We use 2 X, to denote a segment of X ending in time / of length matching the context it is used in (e.g. assuming fu denotes a convolutional kernel of layer and provided with the context hy 'x,, I X, has Mi elements to match the kernel size).
By using the STMC inference pattern perform inference for each element of X separately and reuse past partial states of each layer to fully reconstruct the receptive field of the network which can be represented as follows:
L+1 x = (0:((+12))) 1 (00 'x7))
where (. (2) represents the concatenation of vectors in time axis.
If we use convolutional layer with stride of 2 as our second layer, then in the standard pattern, the size of the output vector of this layer is halved compared to its input. Consequently, every subsequent layer also has its input and output tensors reduced to half the size. We can restore the output to its original size by, for instance, applying transposed convolution. Let's assume that apply transposed
5

---

convolution in the 4th layer of our network. In comparison to our initial plain network, the new strided network will have the same number of operations in both the first and last layers. The 2nd, 3rd, and 4th layers will each have half the computations as before. In the 2nd layer, this reduction is due to the application of stride. In the third layer, it is result of the smaller input size, Similarly, in the 4th layer, if we disregard multiplications by zero.
When employing the STMC inference method, it's anticipated that each layer should process single element and produce single output. If we apply a similar stride and transposed convolution pattern without any additional modification, we'll Il face difficulties reconstructing the output. Strided convolution would provide output values for even-numen nen-nnumeredd inferences (assuming a stride size of 2). However, during odd inferences, the 4th layer (transposed convolution) would require an upcoming even-nune numbered inference value, which would not yet be available. The authors of STMC propose solution where every inference is treated as wen-numbered, maintaining separate states for odd and even input positions. However, this pattern presents a challenge due to the exponential increase in the number states for each added strided convolution.
SOI addresses this issue by removing the necessity of storing additional states, albeit at a cost to measured performance. For instance, in our network, we achieve this by abstaining from calculating the outputs of the 2nd, 3rd, and 4th layers during every even inference. To maintain causality, the transposed convolution output must be temporally shifted to produce either current and future frames or solely future frames, depending on the chosen SOI inference mode. Additionally, we advocate for the use of skip connection between the input of the strided convolution and the output of the transposed convolution to update deeper layers of the network with information about the current data. This operation aims to minimize the influence of data forecasting on the optimized part of the network.
To formally describe partially predictive SOI, let's assume that the network contains layer la with stride of size 2' and layer l, which reverses the downsampling performed by la. Typically, la would be in the encoder part, while L, would be in the decoder.
The downsampling layer only returns a new element for even-numbered inferences. It's important note that until the upsampling layer is reached, there is no need to per- form any further computations when t is odd. Namely for 1 such that la < I s I,:
if is even ( x,-1 / ahe .'x)))
41 41x,=
X,-I,
1-11
if is odd
At layer lu we reconstruct the output by duplicating the convolu- tion output.
Current frame
1. (1o (1+ X/-1 ahu, 1 1xF)7)
(5)
Buffered frame
Note that tu X2 X2
We then concatenate the output with data from the skip connection ((- le represents the concatenation of vectors in channel axis).
"
Copied frame
tu+1 X, (1++1 [1++1 X
la xxx,
X
Removed frame
In the example above traded inference over L, 1, layers the cost of inference over additional channels from skip connection concatenation. The additional cost of skip connection does not take effect in architectures where skip connections naturally exist like U-Net which we use as one of the examples this study.
For fully predictive variant we modify equation (5) and add a shift in time axis.
Figure 3: SOI PP inference pat- temn
1++1 x -1~
x (
[
(hi,. 1 ,xx_,)))
(7)
SOI can be seen as a form of forecasting where instead of training the model to predict output for input yet unknown, we predict partial states of the network. This is because preserving causality in
'The choice of 2 is for notational simplicity. The same derivation be applied for arbitrary stride size.

---

the strided convolution forces us to predict convolution results when the next second input is not yet available.
Experiments
3.1 Speech Separation
We selected speech separation as our first experimental task. 'The choice of the task dictated by our current research interests and the potential benefits of fast online inference. In this task focus separating clean speech signal from noisy background. In literature this task is also referred to as speech denoising noise suppression.
For this experiment we adopted the U-Net architecture as widely used for this specific task and inherently has skip connections which will allow for applying SOI inference pattern without substantial alterations. Our model is composed 7 encoder and 7 decoder layers. The Deep Noise Suppression (DNS) Challenge Interspeech 2020 dataset (Reddy et al., 2020), licensed under CC-BY 4.0, was used for both training and evaluation purposes.
Position of S-CC pair By introducing S-CC pair to the network we enforcing data predictiveness of the network. The exact number the predicted future partial states of the model depends on the position of S-CC pair and number of those pairs within the network. In addition is worth noting that larger amount of predicted partial states of the network leads to higher reduction of computational complexity. In this experiment we test every position of S-CC pairs while applying up two such modules to the model's architecture.
Position SS-CC pair SS-CC pair introduces additional shift in time axis compered S-CC pair. In this experiment we alter the position of SS-CC pair within the network and also separately alter the position of S-CC pair and time shift which might be consider hybrid of partially and fully predictive pattern.
Resampling Simple resampling of audio signal may used to reduce the number of calculations done by neural networks but will yield significant increase in model latency. Nonetheless in this experiment we compare SOI to four different resampling methods: SoX which based on method proposed by Soras (2004), using Kaiser window, polyphase and linear. With this methods resampled our speech separation dataset from 16k to 8k at the input of the model and then from 8k to 16k at the output of the model. For every resampling we used our baseline model and compared it to three selected SOI models.
Pruning In this experiment used unstructured global magnitude pruning (similar Han et al. (2015)) to show how our method can be combined with pruning leading to better results than pruning on standard model. For this experiment chose prune "SOI I" and "SOI 2/6" variants of our baseline model. Each step we pruned 4096 weights from model and we report how models performed on each step.
3.2 Acoustic Scene Classification
Acoustic scene classification (ASC) is our second task of choice. The goal of the task to estimate the location where the specific sample was recorded. This task is commonly considered as auxiliary problem in various online scenarios such as selection of bank denoising filters.
For all our tests in ASC task we used GhostNet architecture (Han et al., 2020). We tested 7 different model sizes for all 3 variants Baseline, STMC and SOI. Each test was repeated times. used the TAU Urban Acoustic Scene 2020 Mobile dataset (Heittola et al., 2020) for both training and validation. Models were trained on a single Nvidia P40 GPU for 500 epochs using Adam optimizer with initial learning rate of Ie-3.
7

---

Results
4.1 Speech Separation
Results of partially predictive SOI in speech separation task shown in figure 4. We tested variants with single S-CC layer ("S-CC") and two S-CC layers ("2xS-CC x"). In the latter case, the experiments are grouped by the position of the first S-CC layer (X). Value of SI-SNRi metric is highly dependant on the position of S-CC layers. Generally, the earlier the S-CC layer is introduced the lower SI-SNRi but higher complexity reduction. This phenomena may be explained by the difficulty of the partial state prediction task. All layers subsequent to the S-CC layer are required predict twice the number of elements compared to the layers without having the S-CC as an prior layer. Overall, with SOI we can achieve linear dependency between computation cost and SI-SNRi up to 64% of complexity reduction (with linearity factor of 0.001 dB SI-SNRi per MMAC/s (0.017 dB SI-SNRi per 1% of reduction)). A significant SI-SNRi drop is observed if S-CC layer is introduced too early. table we present selected results from the whole experiment.
Partially Predictive
Fully Predictive Results for speech separation using fully predictive SOI are presented in figure 5. Details for selected models can be found in table 2. Reduction of metrics observed for fully predictive variant tends be larger than for single S-CC layer in partially predictive SOI. Additional reduction of metrics stems from added shift in time axis. Added shift produces pattern where some part of model can be updated between inferences, as only depends on past data. Size of this model part depends on the position of an SC or SS-CC layer within network. We refer to the size of this model part as "Precomputed" table 2. The increase of precomputed partial state leads to reduction of SI-SNRi metric but allows to speed-up computation, because this part can be computed in advance. With FP SOI we achieved 50% reduction of computational cost at the expense of 11.3% of SI-SNRi but with 83.7% 7% of network calculated using past data only.
MMACS
Table 1: Selected results from experiments with partially predictive SOI for speech sepa- ration.
363.84
727.67
1091 51
1455.34
1819.18
Model
SESNKI (dB)
SLSNKRI retain (%6
Complesity retain (15)
Complexity (MMAC/S)
SL- S3-SNRR (dB)
STMC
7.69
100.0
100.0
1819.2
Prodictive
7.41
95.3
100.0
1819.2
Predictive
6.51 cM
STMC
Preliclive 2
Predeiiee
2.5-C04
2SSCCC
S400
5-00.
5-00
7.23
7:47
7.55
26
97.2
98.2
51.4
64.8
8388
1524.3
1178.7
935.2
SCC
6
2xS-CC 1
6:27
81.6
29.1
528.8
2x8. OC 1
6.54
90.2
35.6
648.5
0.2
0.4
0.6
0.8
1.0
20S-CC sm
2xS-CC
33.8
43.8
796.4
Complexity reduction factor
20S-CC41 7.1428
92.9
47.1
837.3
Figure 4: Results of speech separation experi- ment with PP SOI.
2xS-OC 7.30
94.9
56.7
1031.2
7.40 201
Procomputed
Table 2: Selected results from experiments with fully predictive SOI for speech separation.
363.84
727.67
1091_51
1455.34
1819.18
Model
SE-NNRi (JB)
SI-SNRi
Complexity retain (5)
Completity (MMAC1)
Procumputed (%)
STMC
7.69 1t
100.0
100.0
1819.2
Predictive
7.4122
96.3
100.0
1819.2
51 SNRRR
Predictive 2
6.51
$4.7
100.0
1819.2
100.0
STMC
-I- 5013
SS-00
6.64
85.3
51.4
935.2
97.2
Pedictive
-8 5014
S8-CC5
7,24
94.1
64.8
1178.7
70.4
Pedictive
--- 5015
SS-0C
7.52
97.8
$3.8
1524.3
32.4
S01
-T- 5017
S-CC ]|3
6.82
B8.7
50.0
909.6
83.7
S-CC 166
$7.4
0.2
0.8
S-CC
6.93 20
90.1
51.4
935.2
70.
S-CC
7.10:20
92.3
58.1
1057.5
$7.4
Precomputed
S-CC 4/6
7.30:20
94.9
61.5
1118.4
$7.4
Figure 5: Results of speech separation experiment with FP SOI.
7.23
6-00
7.39
96.1
71.3
1256.9
32.4
8

---

Resampling The results presented in the table 3 show that SOI outperforms the listed resampling methods in terms of complexity reduction under the constrained of preserving the quality of the original model (STMC). The results for resampling-hased complexity reduction indicate that model's quality depends on the used resampling algorithm and, for all the selected ones, the initial information loss strongly affects the model performance.
Complexity (MMACH)
Table Comparison between resampling and SOI
363,65 84
727.67
1091.51
1455,34
18
SNRi (dH)
Complexity (MMADS)
Method
SE-SNRI (05)
STMC:
7.69
1819.2
Linear
3.49
Polyphase
STMC
Kaiser
5.83
909.6
SoX
5.77
900.6
2x501
S-CC
7.47
1178.7
0.2
0.4
0.6
0.8
1.0
C
7.23
935.2
Park complexyy reducction factor
S-CC 1/3
6.27
5288
Figure 6: Pruning of STMC, SOI and 2xSOI mod-
els. Unpruned models are indicated by markers.
Pruning The application of SOI along with the pruning of STMC model surpasses the effect of application of the pruning alone. The addition of SOI allowed for achieving further reduction in computational complexity by around 300 MMAC/s for the same model's performance, which about 16% of the original model. Interestingly the "SOI 216" surpassed the "SOI I" model at around 6 dB SI-SNRi. The results of the experiment are shown in Fig. 6.
4.2 Acoustic Scene Classification
Table 4: Results of ASC experiment.
Results for the ASC task are collected in Ta- ble 4. For models I, III and VII observed that our method led to an increase in accuracy, whereas other models showed the decrease in metrics. The largest decrease in accuracy was around 2.20% and the largest increase was 1.69%. These results indicate that SOI does not lead to a decrease in model quality for this par- ticular task compared to the STMC model. This can be explained by the much slower change of output (acoustic scene label) compared to the previous task (speech mask). our tests, the reduction in computational complexity of SOI models amounted to around 16% compared to the STMC model. This reduction decreased to 1 11% for the smallest model due to the addition of the skip connections. For this particular ar- chitecture, our method also reduced the number of parameters for the largest tested models by around 7%.
Top-1 Accuracy
Complesity (MMACK)
Number of Parametera
Model
Method
Baseline STMC
423.07
1470
1470
SDI
55,90
0.37
1833
Baseline SIMC
64.18
959.67
3352
64.18
0.94
3352
SDI
Baseline SIMC
66.15
1624.11
5814
SOI
68.14
1.37
6653
Baseline STMC
70.57
2405.09
8696
IV
70.57
2,35
8696
SOI
70.32
1.97
8835
Baseline STMC
76.91
6769.70
25450
76.91 -434
SOI
24633
cue
VI
SIMC
$1.66
12.78
50391
SOI
80.3312
10.75
47668
Baseline STMC
$3.07 10
21395526
83432
VII
83.07
20.87
83432
SOI
4%%
77753
Conclusion
In this work, we presented method for reducing the computational cost of a convolutional neural network by reusing network partial states from previous inferences, leading to generalization of these states over longer time periods. We discussed the effect of partial state prediction that our method imposes on the neural model and demonstrated that can be applied gradually to balance model quality metries and computational cost.

---

Our experiments highlight the high potential for computational cost reduction of CNN. especially for tasks where the output remains relatively constant, such as event detection or classification. We achieved computational cost reduction of 50% without any drop in metrics in the ASC task and 64.4% reduction in computational cost with a relatively small reduction of 9.8% in metries for the speech separation task. We also showcased the ability of SOI to control the trade-off between model's quality and computational cost, allowing for resource- and requirement-awarr tuning.
The presented method offers an alternative to the STMC solution for strided convolution. While SOI reduces network computational complexity the expense of measured performance, STMC ensures that metrics are not reduced but at the cost of increased memory consumption at exponential rate. SOI is akin methods like network pruning. but unlike pruning, it does not require special sparse kernels for inference optimization. It is worth noting that these methods are not mutually exclusive, therefore, the STMC strided convolution handler, SOI, and pruning can coexist within a neural network to achieve the desired model performance.
References
Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitis, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, and Markus Zijlstra. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science, 378(6624):0067-1074 2022. doi: 10.1 1226/science.ade909
Vijay Balasubramanian. Brain power. Proceedings of the National Academy of Sciences. 118(32): e2107022118, 2021. doi: 10. 1073/pnas.2107022111 URL https: //t::://wwwe //www pnas. porg/dii/abss 10. 1073/paasp 2107022118.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, pp. 527-536. JMLR.org, 2017.
Víctor Campos, Brendan Jou, Xavier Giró-i-Nieto, Jordi Torres, and Shih-Fu Chang. Skip RNN: learn- ing to skip state updates in recurrent neural networks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpp:://ppnrreeee
Jiaming Gong, Wei Liu, Mengjie Pei, Chengchao Wu, and Liufei Guo. Resnet10: A lightweight residual network for remote sensing image classification. In 2022 14th International Conference on Measuring Technology and Mechatronics Automation (ICMTMA), pp. 975-978, 2022. doi: 10.1 D0.1009ICCTMMAA55533222000000.
R.M. Gray and D.L. Neuhoff. Quantization. IEEE Transactions on Information Theory, 44(6): 2325 2383, 1998. doi: 10.1109/18.720541.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1135-1143, Cambridge, MA, USA, 2015. MIT Press.
Christian Hansen, Casper Hansen, Stephen Alstrup, Jakob Grue Simonsen, and Christina Lioma. Neural speed reading with structural-jupp-LLSTMM... In International Conference Learning Representations, 2019. URL https:///peerrviies www.fouumttmteteeeemooo wdw.ornunnddiooooAaaoa
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015. URL https: / /appi. semanticscholar cacillarrrrrgggge /
10

---

Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. TAU Urban Acoustic Scenes 2020 Mobile, Development dataset. Zenodo, February 2020. doi: 10.5281/zenodo.381996.. URL nttps://doi. org/10. 10. 2281/zenodo.38100 3819968.
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Q. Weinberger. Multi-scale dense networks for resource efficient image classification. International Conference on Learning Representations, 2017.
Yacine Jemiiee, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. In International Conference on Learning Representations, 2017. URL
H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. HMDB: large video database for human motion recognition. Proceedings of the International Conference on Computer Vision (ICCV), 2011.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 2019.
Simon B. Laughlin, Rob R. de Ruyter van Steveninck, and John C. Anderson. The metabolic cost of neural information. Nature Neuroscience, 1(1):36-41, May 1998. ISSN 1546-1726. doi: 10.1038/236. URL https://doi. www/oo......oooooooooo..../ 1038..000020000
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. D. Touret- zky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan- Kaufmann, 1989. URL https: http://///cococo.s......e neurips. cttrriisscccppeeee httpp/ppeee9///iiiiia//ee (G082208a11703300221111111111111 Paper pdf.
Charles E. Leiserson, Neil C. Thompson, Joel S. Emer, Bradley C. Kuszmaul, Butler W. Lampson, Daniel Sanchez, and Tao B. Schardl. There's plenty of room at the top: What will drive computer performance after moore's law? Science, 368(44)))caamm774 2020. doi: 10.1126/science aam9744.
Gordon E Moore. Cramming more components onto integrated circuits. Proceedings of the IEEE, 86 (1):82-85, 1998.
Thai Son Nguyen, Sebastian Stueker, and Alexander H. Waibel. Super-human performance in online low-latency recognition of conversational speech. In Interspeech, 2020.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint arXiv:2 104. 10350, 2021.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. Proc. Interspeech 2020, 2020.
Michal Romaniuk, Piotr Masztalski, Karol Piaskowski, and Mateusz Matuszewski. Efficient Low- Latency Speech Enhancement with Mobile Audio Streaming Networks. In Proc. Interspeech 2020, pp. -300, 2020. doi: 10.21437//mterspeechh 2020-2443.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model. Nature, URL https://doi. htttp:://do..or/1000
188(7839):604-609 dec 2020. doi: 10.1038/s41186- 1118860020-00 020-03051-4,
Biswa Sengupta and Martin B. Stemmler. Power consumption during neuronal computation. Pro- ceedings of the IEEE, 102(5):738-7555 2014. doi: 10.1 10.1100//R00.222443377..
Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. Neural speed reading via skim-RNN. In International Conference on Learning Representations, 2018. URL. httpss//opperrevir: net/forme htt///formmiddide dQG-Rb. Rb.

---

Laurent Soras. The quest for the perfect resampler. 02 2004.
Grzegorz Stefański, Krzysztof Arendt, Pawel Daniluk, Bartlomiej Jasik, and Artur Szumaczuk. Short- term memory convolutions. In The Eleventh International Conference on Learning Representations, 2023. URL http:::///// htttps://openreviert net/foruml iim=:dddc=-_D_____
Surat Teerapittayanon, Bradley McDanel, and H.T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464 2469, 2016. doi: 10.1 1..1099/CPR.2211.7900000
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In European Conference on Computer Vision (ECCV), 2018.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E. Gonzalez. Skipnet: Learning dynamic routing convolutional networks. Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIII, volume 11217 of Lecture Notes in URL Computer Science, pp. 420-436. Springer, 2018. doi: 10. 10.1007/978-3-00-0022011 https://doi. btta:://doi.oor/100/ 10///10.110 1007 0/1///9783-300012----1--
Xiaowei Xu, Yukun Ding. Sharon Xiaobo Hu, Michael Niemier, Jason Cong, Yu Hu, and Yiyu Shi. Scaling for edge inference of deep neural networks. Nature Electronics 2018 1:4, 1:216-222, 2018. ISSN 2520-1 111. doi: 10. 10.100/8/41122 articles/s41928-011 -00-009-----0
nature. URL hattps://wwar / com/
Adams Wei Yu. Hongrae Lee, and Quoc Le. Learning to skim text. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1880-1890, Vancouver, Canada, July 2017. Association for Computational Linguisties. doi: 10.18653/1/P17-1172. URL https: http://aclaanooog..oooooo org/P17-1172.
12

---

A Reproducibility Notes
A.1 Speech Separation
For this experiment we adopted the U-Net architecture as it is widely used for this specific task and inherently has skip connections which will allow for applying SOI inference pattern without substantial alterations. Our model is composed of 7 encoder and 7 decoder layers, each comprising STMCASTMC, batch norm and ELU activation layers. Each model was trained for 100 epochs using Adam optimizer with initial learning rate of le-3. We trained each model on a single Nvidia P40 GPU 5 times and reported the average SI-SNRi. The mean training time of single model amounted to about 14 hours. The Deep Noise Suppression (DNS) Challenge- Interspeech 2020 dataset (Reddy al., 2020), licensed under CC-BY 4.0, used for both training and evaluation purposes. For training, we used 16384 10s samples without any form of augmentation and for both validation and test sets we used 64 samples with similar setup to the training set.
A.2 Acoustic Scene Classification
For all our tests in ASC task we used GhostNet architecture (Han et al., 2020). Our baseline model is the original architecture with "same" padding making it not applicable in online scenario. STMC model changes the padding to manually-applied padding left-most (oldest) side of data and applies STMC inference pattern. SOI model adds upsampling after each processing block and skip connections between downsampiingupsampiigg layers.
Models were trained on single Nvidia P40 GPU for 500 epochs using Adam optimizer with initial learning rate of le-3. We tested different model sizes for all 3 variants Baseline, STMC and SOL Each test was repeated 5 times. The mean training time of a single model amounted to about hours. We used the TAU Urban Acoustic Scene 2020 Mobile dataset (Heittola et al., 2020) for both training and validation.
B Strided Convolutions are Better for Longer Predictions
In this experiment, investigated the impact of strided convolutions on predictive inference. Our test environment consisted of U-Net model applied to a speech separation task. We examined two model variants: Predictive and Strided Predictive. The Predictive model serves as our baseline U-Net with an added time shift the end. The Strided Predictive model, in addition to the time shift, incorporates strided convolutions in place of standard ones. For each model, we conducted tests with four different lengths of prediction, ranging from frame frames. We performed five training runs for each model using the setup outlined in Section 3.1. The results of this experiment are displayed in Figure 7 and Table 5.
Predictive
Stided predictive
SI-SNRi (dB)
3
4
Prediction size (frame)
Figure Comparison between standard convolutions and strided convolution in pre- dictive inference.
We conclude that strided convolutions shows higher potential for longer predictions. We attribute this effect to the fact, that using strides forces stronger generalization of outputs of strided convolutions because they are applied in multiple contexts.
Table 5: Results of experiment the influence of strided convolution predictive inference.
Length of prediction Predictive
SI-SNRi (dB)
Strided predictive
7.41 ..09
7.24 +0.44
6,51 4003
6.70 40.07
4.61 0..05
5.47 -0.07
3.59 +0.08
4.00 +0.11
13

---

C Inference Time and Peak Memory Footprint
To showcase SOI's influence inference time and peak memory consumption, extended results from Table to include these measurements with a single S-CC layer. The collected results presented in Table 6.
Table 6: Results from experiments with partially predictive SOI for speech separation with added average inference time and peak memory footprint.
Model
SI-SNRi
SI-SNRi
Complexity
Complexity (MMAC/s)
Avg. inference time (ms)
Peak memory footprint (MB)
(dB)
retain (%)
retain (%)
STMC
7.69
100.0
100.0
1819.2
9.93 = 0.13
27.2
-888
Predictive
7.41
-U00
96.3
100.0
1819.2
9.93 = 0.13
27.2
Predictive 2
6.51
84.7
100.0
1819.2
9.93 + 0.13
27.2
.008
S-CC
7.15 2013
93.0
50.1
911.4
5.28 + 0.07
14.6
S-CC
7.23 -000
94.0
51.4
935.2
5.63 = 0.11
18.7
3-CC
7.28 2005
94.7
58.1
1057.5
6.27 = 0.10
24.0
S-CC4
7.43 +0.85
96.7
61.5
1118.3
6.67 = 0.13
25.1
S-CC5
7.47 2015
97.2
64.8
1178.7
6.98 + 0.14
25.6
6-CC6
7.56 400
98.3
71.3
1296.9
7.50 = 0.08
26.1
S-CC7
7.55
-U. 2005
98.2
83.8
1524.3
8.43 0.12
26.6
Additionally, in Figure 8, show how inference time and peak memory consumption depend SI-SNRi and the complexity reduction factor.
A
B
7.75
7.75
SI-SNRI (dB)
4
SI-SNRI (dB)
7.50
7.50
7.25
7.25
+ STMC
S.CC
7.00
STMC
- S-CC
10
20
Avg. inference time (ms)
Peak memory footprint (MB)
C
MMAC/s
D
MMAC/s
1091.51
1455.34
1819.18
1091.51
1455.34
1819.18
Peak memory foolprint (MB)
Avg. inference time (ms)
10
STMC
S-CC
0.6
0.8
1.0
0.6
0.8
1.0
Complexity reduction factor
Complexity reduction factor
Figure 8: Average inference time and peak memory footprint
14

---

These results suggest that the inference time of the tested model depends linearly on SI-SNRi and the complexity reduction factor, while peak memory consumption has significant impact on SI-SNRi at the beginning. From 60% complexity reduction onward, the peak memory footprint decreases sharply.
D Interpolation
Our method may also benefit from the usage of interpolation methods in place of extrap- olation. We tested a singular S-CC layer with three different types of interpolation: nearest-neighbor, bilinear, and bicubic. The results of this experiment can be observed in Figure 9 and Table 7. For comparison, we in- cluded our extrapolated duplication method
7.5
SE (dB
74
STMC
7.2
neightr
Billnear
Bicubic
Duplication
in the results of this experiment.
2
In this experiment, sults with bilinear or bicubic interpolation, although the results for bilinear interpolation showed much higher variance than any other method. It is important even though we achieve slightly better results using interpolation, the usage of interpolation comes at the cost of higher latency as we need to wait for additional time frame.
achieved the best re-
S-CC layer pasition
Figure 9: Comparison between extrapolated frame duplication and interpolation methods for PP SOI.
remember that
Table 7: Results of experiment on interpolation methods with PP SOI.
SI-SNRi (dB)
Nearest- neighbor
Model
Duplication
Bilinear
Bicubic
S-CC1
7.15 -0
7.13 +0.10
7.22 40.11
7.18 -005
S-CC2
7.23 0.11
7.17 10.06
7.42 0.24
7.3 0.10
S-CC3
7.28 +007
7.32 +0.10
7.48 +0.14
7.49 -0.17
S-CC4
7.43 -0.14
7.35 -116
7.48 0.18
7.48
0004
-004
S-CC5
7.47 -007
7.51 +0.12
7.63 +0,10
7.54 4306
6-CC6
7.56
0.06
-005
7.52 0.05
40.05
7.62
40.33
0.12
7.67
-0.88
.009
S-CC7
7.55 +005
7.51 1007
7.66 0.16
7.61 .008
E Different Extrapolation Methods
In the main paper used element duplication that any type of extrapolation may be used. Here we compare the results of our speech separation experiment using element duplication with another commonly found method transposed convo- lution. We tested both partially predictive and fully predictive modes using the setup described in Section 3.1.
extrapolation method but also pointed out
Results of the experiment with PP SOI are presented in figure 10. this case we tested U-Net with two S-CC layers. Each plot represents a different position of the first S-CC pair and each point X-axis represents a different position the second S-CC pair. We also present results for hybrid models where duplication and transposed convolutions were used together in the first and second S-CC pairs respectively.
In this experiment neither method demonstrated a significant advantage. It seems that duplication tends to perform better than transposed convolution if introduced deeper within the network and vice versa although difference is marginal.
15

---

Diplication
Transposed Conclution
Hybed
70
7.0
SU-SNRI (dB)
7.4
14
6.8
2nd S-CC layer position
Figure 10: Comparison between frame duplication and transposed convolution for PP SOI.
Results achieved with FP SOI are shown in figure 11. Each plot represents different position of S-CC pair and each point represents different position of SC layer. Achieved results confirm previous conclusions.
This experiment proves that element duplication a viable method and thus should be chosen for its simplicity.
Table 8: Results of experiment on different extrapolation methods with PP SOL.
SI-SNRi (dB)
Model
Transposed convolution
Duplication
Hybrid
S-CC 1/2
5,43 0.17
5.78 0.12
5,68 0.06
S-CC 1/3
6.27 +0.18
6.30 +0.09
6.31 +0.29
S-CC 1/4
6.48 -016
6.51 -0.00
6.54 .011
S-CC 1|5
6.62 +0.09
6.68 +0.11
6.73 +0.03
S-CC 1/6
6.94 0.01
6.92 0.14
6.77 009
40.10
S-CC 1/7
7.01 +0,09
7.03 +0.07
6.92 +0.07
S-CC
6,25 +0.04
6.29 +0.09
6,28 +0.04
S-CC2/4
6.43 0.07
6.44 6.44-.0.1
6.45 -013
S-CC2|5
6.67 +0.05
6.67 +0.09
6.71 +0.13
S-CC2/6
6.99 0.16
6.87 0.05
6,85 0.06
S-CC217
7.09 +0,07
7.01 19.15
7.02 +0.05
3-CC3J4
6.57 +0.13
6.58 +0.07
6,57 +0.12
S-CC3J5
6.84
+0.26
6.74+0.10
6.78 .015
-0.55
3/6C
7.02 +0.17
6.99 +0.17
6,92 +0.09
S-CC3/7
7.10 0008
7.05 0.00
7.08 0.06
S-CC
6.84 40.09
6.84 40.15
6.83 0.8
S-CC
7.14
-0188
40.03
7.05 +8.11
7.11
-.06
40.06
S-CC
7.18 -0.12
7.15 .009
7.18 2.09
S-CC
7. 7.14 +0.13
7.15 +0.07
7.14
S-CC
7.30
40.08
7.25
+0.07
7.27 .009
40.04
J008
.0066
S-CC 6|7
7.34 0.16
7.29 0.05
7.30 0.10
16

---

Duplication
Transposed Comolation
72
7.0
6.5
6.5
St SUSNN (dB)
7.5
7.6
72
72
7.4
6.9
SC layer position
Figure 11: Comparison between frame duplication and transposed convolution for FP SOI.
Table 9: Results of experiment on different extrapolation methods with FP SOI.
SI-SNRi (dB)
Model
Transposed convolution
Duplication
SS-CC
6.50 .0.0
6.77 .0.05
S-CC 1/2
6.63
50.05
4.00
6.66
+0.07
0.04
S-CC 1/3
6.82 2004
6.80 -0.04
S-CC 1/4
6.93 +0007
6.96 -0.8
S-CC 1/5
6.98 2006
7.00 40.12
S-CC 1/6
7.06 +0.00
7.14 +005
S-CC 1
7.21 -0.15
7.14 -0.14
40.31
SS-CC2
6.64 40.3
6.73 -0.11
S-CC2/3
6.80 +0.08
6.83 +0.08
2/CC
6.95 2010
6.91
+0.14
-0.00
2|CC.2/5
6.93 +0.13
6.98 +0.8
2-CC22/6
7.10 0.05
7.14 00.12
2-CC
7.21 +0007
7.18 +0.0
SS-CC
6.81 +0.17
6,86 +0.00
3-CC3/4
6.97 2006
7.01 -0.08
S-CC3|55
7.06 +0.12
7.02 +0.08
3-CC3/6
7.10 3.09
7.12 00.15
3-CC3/7
7.21 #0UR
7.23 #0.12
SS-CC4
7.00 +004
6,98 +008
S-CC45
7.10 AU02
7.12 40.10
S-CC446
7,30 +0.1
7.16 +0.12
4-CC
7.34 0004
7.26 -0.10
SS-CC
7.24 0.17
7.15 00.00
5/6
7.23
s00D
2004
7.25
+008
-0.05
S-CC.5/7
7.34 AL17
7.35 J1.04
SS-CC
7.31 +0.14
7.35 +0.6
6/CC
7.39 .0.66
40.12
7.28 .1.13
+0.10
SS-CC7
7.52 0.08
7.47 00.12
17

---

Video Action Recognition
Other domains can benefit from SOI, as it can be applied to any time-series data. To illustrate this, we conducted experiments using SOI for an action recognition task with video data. We utilized the HMDB-51 dataset Kuehne et al. (2011), which contains 24 fps video data of human actions split into 51 classes. We trained popular ResNet- 10 architecture Gong et al. (2022) in three variants: regular, small (with halved number of channels) and tiny (with number of channels reduced fourfold), where we replaced 3D convolutional layers with 3D STMC layers. Here, we applied SOI by introducing a skip connection between the output of block 2 and the input of block 4, optimizing block 3.
Table 10: Results of video action recognition experiment.
Regular
SOI
Model
Top-1 Acc
Complexity (GMAC/s)
Top-1 Acc (%)
Complexity
(%)
(GMAC/s)
ResNet-10
32.63
48.54
33.34
40.69
ResNet-10 small
31.24
15.05
31.41
13.09
ResNet-10 tiny
30.46
5.23
30.90
4.73
Mo ViNet AO
34.40
33.15
31.88
24.26
Mo ViNet A1
35.96
69.77
32.73
53.92
To demonstrate that SOI can work not only with STMC, we also applied it to MoViNets, which use method called "Stream buffers". We trained two variants of MoViNets, AO and A1, in their streaming form. SOI was applied by optimizing blocks and 5. Please note that SOI can used not only with 3D convolutions but also with their popular 2D+1 variant.
All models were trained for 100 epochs with Adam optimizer and learning rate 5e-5. Each model was trained on Nvidia A100 GPU with batch size of 16. The mean training time amounted to about about 37h.
Achieved results suggest that SOI is suitable for video domain as well. ResNet-10 architecture proved be highly compatible with SOI for this task results achieved with SOI variant of this model outperformed the regular ones. We believe that this improvement was imposed by the increase in receptive field SOI adds additional strided convolution. The achieved reduction complexity for this family of models was between 10-17% depending on model size. For Mo ViNes, the decrease of around 3% accuracy can be spotted, although the imposed complexity reduction was higher compared to ResNet and amounted to 23-30%.
G Acoustic Scene Classification with ResNet
To further evaluate the effectiveness of our method, we conducted experiments on the accoustic scene classification task using ResNet architecture He et al. (2015). We chose ResNet for two standard deep learning model, and (2) its
primary reasons: (1) its widespread use and recognition relatively large size compared to other architectures tested, such as GhostNet, which allows us to better assess how SOI performs on larger models.
We tested 4 different ResNet models for all variants Baseline, STMC and SOL. Each test was repeated 5 times. We used the TAU Urban Acoustic Scene 2020 Mobile dataset (Heittola al., 2020) for both training and validation.
The results presented in Table 11.
18

---

Table 11: Results of ASC experiment with ResNet.
Top-1
Complexity (GMAC/s)
Number of Parameters
Model
Method
Accuracy (%)
Baseline
85.13 10,33
1132
143.65
18
STMC
85.13 -1.32
15.56
11.7M
SOI
91.55 +8.72
12.35
Baseline
86.03
+1.36
686.96
3.08
34
STMC
86.03 +1.30
32.65
21.8M
SOI
92.01
+0.69
26.46
.0.9
Baseline
89.66 0.54
794.34
50
STMC
89.66 10.60
33.10
25.6M
SOI
91.43 -1.89
27.99
Baseline
94. 74 0.89
2168.81
101
STMC
94. 74 +0,40
112.84
44.5M
SOI 96.22 .0.5
95.83
The results, shown in Table 11, demonstrate that the SOI-enhanced ResNets consistently outper- formed the baseline models in terms of accuracy. Specifically, the SOI-based models achieved higher classification accuracy across all tested configurations. We belived this improvement can be attributed to two key factors:
The introduction of SOI allows the model to generalize over longer time frames by expanding its receptive field. This helps the model capture temporal context, which is crucial for tasks like ASC.
SOI encourages the model to generalize its output states by predicting future partial states, which improves the model's ability to handle variations in the data.
Despite ResNet's depth and complexity, the method proved successful, not only reducing computa- tional cost but also improving performance.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations proposed method are discussed in section 1.3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
Paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Reproducibility notes are present in section A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification:
We cannot open-source our codebase due to its commercial purpose.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
ooniiicy)))))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips.coo. wwtppplic/gguiiieicccceoooocccce gttee/uwiee///eee CCodeSubmiieeeeece
hotooi/igguddeesodeeeubbiiiiiiii wwtpommmssiiooolllliiiooioo (eionnllccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see section A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Results are accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see section A.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpps:////uuupaaee http//nuurri.cccccc cttpp//cecpppppttttattaiaaeeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: The paper presents a general method for reduction of computational complexity of neural networks.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Datasets and their accompanying licenses are listed in section A.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "FugwGVGxHm", "reviewer": "Reviewer_rrin", "summary": "The authors introduce a novel method called Scattered Online Inference (SOI) aimed at reducing the computational cost of convolutional neural networks (CNNs). This method leverages the reuse of network partial states from previous inferences, thereby generalizing these states over extended periods. SOI enables extrapolation, particularly enhancing processing speed in deeper layers. By applying compression techniques, SOI generates more generalized inner partial states within the CNN, allowing the system to skip full model recalculations for each inference.\n\nTheir contributions include offering a distinct treatment of strided layers, resulting in a new and efficient inference pattern. This approach modifies the inference pattern of the network to skip the recalculation of certain layers according to a predetermined scheme. Their method significantly reduces the computational cost of CNN model with only a negligible decrease in performance. These optimizations are achieved with minimal changes to the architecture, making the method suitable for tasks where reducing energy consumption or processing time is crucial.", "questions": "I also would like to see different versions of the ResNet model because they too have skip connections and how much computational cost gets reduced using SOI?", "limitations": "not clearly but yes, up to certain degree", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "From my perspective, authors have touched each dimesnion of originality, quality, clarity, and significance.\n\n1.) Their proposed SOI approach is inspired and built upon Short-Term Memory Convolution approach.\n\n2.) In order to support their approach, they have performed extensive experiements with different architectures (U-Net, GhostNet and their variants) on two benchmark datasets: Interspeech 2020 dataset and TAU Urban Acoustic Scene 2020 Mobile dataset and also compared their reuslts with state-of-the-art model.\n\n3.) They achieved a computational cost reduction of 50% without any drop in metrics in the acoustic scene classification task and a 64.4% reduction in computational cost for the speech separation task.\n\n4.) Through extensive experiments, they showed the ability of SOI to control the trade-off between model’s quality and computational cost, allowing for resource- and requirement-aware tuning.\n\n5.) They have shared all the information regarding experiments and each hyperparameter used that will be useful to reproduce the results.", "weaknesses": "The limitation of paper is discussed in one line in subsection 1.2 that it incurs a neglible decrease in model's performance. From my perspective, it should have been more explanatory. For speech separation task, there was 9.8% of reduction in  metrics. I think, this reduction is not negligible. I believe it would have been better if authors had clarified or mentioned more supporting statements regarding the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review! We appreciate your detailed feedback and the opportunity to address your comments and concerns. We are pleased to see that you recognize the originality, quality, clarity, and significance of our work. Your positive feedback on our approach, extensive experiments, and detailed presentation of results is highly appreciated.\n\n> I also would like to see different versions of the ResNet model because they too have skip connections and how much computational cost gets reduced using SOI?\n\nIn our research, we have applied SOI to a 3D version of ResNet-10 and MoviNets for the video action recognition task. To see the results of our experiment, please refer to Appendix E. Nevertheless, we believe this is a very fitting suggestion. ResNet is a widely known and often-used architecture, and we recognize the value of extending our research to cover ResNet models as well. Unfortunately, we could not conduct additional experiments on such a short notice. We plan to test SOI on various ResNet architectures and include the preliminary results in the revised manuscript.\n\n> The limitation of paper is discussed in one line in subsection 1.2 that it incurs a neglible decrease in model's performance. From my perspective, it should have been more explanatory. For speech separation task, there was 9.8% of reduction in metrics. I think, this reduction is not negligible. I believe it would have been better if authors had clarified or mentioned more supporting statements regarding the results.\n\nWe understand your concerns regarding the 9.8% reduction in metrics for the speech separation task. In the context of our work, we find such reduction to be negligible due to the following reasons:\n\n- Our primary focus is on real-time systems where computational resources are limited and latency is a critical factor. In these scenarios, achieving a significant reduction in computational cost while maintaining performance close to the original model is a considerable achievement. Therefore, the reduction in quality must always be considered in relation to the reduction in computational cost. The decrease in performance is often an acceptable trade-off for the substantial gains in efficiency and reduced latency. In our case, a 9.8% drop in quality is an acceptable price for a 64.4% reduction in computational cost.\n\n- When comparing our results to the other state-of-the-art methods, the observed 9.8% performance reduction can be considered relatively small (e.g., see Figure 6 for our results with pruning). Many popular techniques that aim to reduce computational complexity or improve efficiency often degrade the performance to a much higher extent. Our method, on the other hand, manages to minimize the reduction in quality while providing substantial computational benefits.\n\n- In practical applications, the difference in performance (9.8% reduction) might not significantly impact the overall user experience, especially when balanced against the improved efficiency and reduced resource consumption. For many real-time applications, the ability to deploy a model with lower computational requirements can be more critical than a slight reduction in performance metrics.\n\n- The SOI method allows for flexible tuning between computational cost and model performance. This means that, depending on the specific requirements and constraints of the application, users can adjust the trade-off to either prioritize efficiency or performance. The 9.8% reduction observed in our experiments represents a particular balance point that can be adjusted if needed.\n\nWe will revise the subsection 1.2 to include this detailed explanation and to provide a clearer rationale for why we consider the decrease in performance to be negligible in the context of the broader benefits offered by our method.\n\nThank you once again for your detailed review and constructive suggestions. We are committed to improving our paper based on your feedback and look forward to presenting a stronger and more comprehensive manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0X24kAkl4Q", "reviewer": "Reviewer_Lrh6", "summary": "The authors propose a novel method that reduces the computational cost of regular ANN models to achieve better online inference efficiency. The core technique named Scattered Online Inference (SOI) is able to reduce computational cost with partial predictions.", "questions": "- What are the removed frames? How to decide the removal?\n- A better explanation of Figure 2 C&D could be provided in conjunction with section 2.2. Which frames are predicted in C&D?", "limitations": "Limitations are discussed in the paper. The proposed technique is mainly limited to time-series data.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method is designed to avoid redundant computation in convolution layers.\n- Brings little learning quality drop.\n- Extensive experiments.", "weaknesses": "- The organization of this paper makes it very difficult for readers to appreciate the true merit of this paper.\n- It is unclear which part is reusing STMC and which part is uniquely proposed by SOI.\n- No backgrounds are provided for STMC.\n- To show that the proposed algorithm achieves higher efficiency, peak memory usage and latency should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback on our paper. We appreciate your constructive comments, which will help us improve the quality and clarity of our work. We are glad to hear that you found the method's design, minimal impact on learning quality, and extensive experiments to be strengths of our paper.\n\n> What are the removed frames? How to decide the removal?\n\nThe removed frames are those cropped after a layer during training to ensure that the network maintains a causal manner. This ensures proper alignment of the frames during training and allows achieving the desired inference pattern during model inference. The cropping is essential to prevent an increase in latency. To retain causality within the network, it is crucial to remove all rightmost extrapolated frames, as illustrated in Figure 1C. The number of removed frames can be adjusted to introduce more predictive factors into the network, subsequently reducing latency, peak computational complexity, and peak memory usage.\n\n> A better explanation of Figure 2 C&D could be provided in conjunction with section 2.2. Which frames are predicted in C&D?\n\nThese subfigures illustrate the inference patterns of the Partially Predictive (PP) SOI. In Figure 2C, the even inference of PP SOI updates all network states (both blue and orange), whereas in Figure 2D, the odd inference utilizes buffered partial states of the second layer (previously blue) from the previous inference.\n\nThe frames predicted are those that have been extrapolated (white ones) based on the previously observed frames. The extrapolated frames are a generalized output of the layer for a larger number of input frames (longer time). Therefore, these frames represent a prediction of this general state for frames that the model has not yet seen. This approach allows the network to maintain continuity and causality without having to recalculate the entire sequence.\n\n> To show that the proposed algorithm achieves higher efficiency, peak memory usage and latency should be provided.\n\nThank you for your suggestion regarding the benefits of providing peak memory usage and latency to demonstrate the higher efficiency of our proposed algorithm.\n\nIn response, we have included the peak memory usage and inference time (as a measure of latency) in the supplementary PDF added during the rebuttal. These results are presented in Table 9 and Figure 12, continuing the enumeration from the original paper. These results were achieved using the Intel Xeon Gold 6246R CPU with a clock speed of 3.40 GHz. We will incorporate these results into the revised manuscript to provide a more comprehensive evaluation of our algorithm's efficiency.\n\nWe hope these additional results strengthen our claims and provide the necessary evidence to support the effectiveness of our method.\n\n> It is unclear which part is reusing STMC and which part is uniquely proposed by SOI.\n\nAlthough we were inspired by STMC, SOI is quite different. STMC is a method that removes redundant calculations in the online inference of CNNs. With SOI, we aimed to reduce computational complexity by not processing data through the whole model on each inference but still having the ability to update the output of a model on each incoming input frame. SOI offers different handling of strided convolution. Although STMC can handle strided convolution to decrease computational complexity, the authors stated in their paper that the usage of stride requires buffering multiple states, which unfortunately increases memory requirements exponentially (due to the increase in the number of shift registers).\n\n> No backgrounds are provided for STMC.\n\nWhile we did not include an extensive background on STMC, we referenced the STMC paper in our manuscript. We could not provide an additional explanation of the STMC method due to the length requirements for a paper. We understand that the reviewers / readers may not have the time to fully engage with the paper and all relevant references. We will work on fitting a brief overview of STMC to address this concern in the revised manuscript.\n\n> The organization of this paper makes it very difficult for readers to appreciate the true merit of this paper.\n\nThe structure of our paper follows a standard format: introduction, method description, experiments description, results, and conclusion. We made every effort to ensure a natural flow of ideas and make sure that each section builds on the previous one.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y4paQbC2XB", "reviewer": "Reviewer_L5QG", "summary": "This paper introduces a novel method called Scattered Online Inference (SOI) aimed at reducing the computational cost of convolutional neural networks (CNNs) by reusing partial states from previous inferences. The method generalizes these states over longer time periods, balancing computational efficiency and model performance. SOI leverages techniques such as strided convolution, transposed convolution, and skip connections to achieve significant reductions in computational complexity without substantial loss of accuracy. The paper presents extensive experimental results demonstrating the effectiveness of SOI in two tasks: speech separation and acoustic scene classification. In the speech separation task, the method achieved a 64.4% reduction in computational cost with only a 9.8% decrease in SI-SNRi. In the acoustic scene classification task, a 50% reduction in computational cost was achieved without any drop in accuracy. The SOI method also showcases the ability to control the trade-off between model quality and computational cost, allowing for resource- and requirement-aware tuning.", "questions": "Please refer to weakness.", "limitations": "No.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "### Strengths\n\nThe paper introduces the novel Scattered Online Inference (SOI) method, which reduces the computational complexity of convolutional neural networks (CNNs) by reusing partial states from previous inferences. This originality is evident in its creative combination of strided and transposed convolutions with new techniques for managing computational resources. The paper is methodologically sound, with thorough experimental evaluations showing significant computational cost reductions without substantial performance loss. The writing is clear and well-structured, making complex concepts accessible. The significance of this work lies in its applicability to resource-constrained environments and real-time applications, offering a valuable alternative to existing methods like STMC.", "weaknesses": "### Weaknesses\n\nWhile the paper presents a novel approach, there are several areas where it could be improved:\n\n1. **Early Layer Impact on SI-SNRi**: The paper notes a significant drop in SI-SNRi when the S-CC layer is introduced too early in the network. This suggests that the method may not be robust across different network configurations. The authors could improve this by exploring adaptive strategies for placing S-CC layers based on the specific characteristics of the task and dataset.\n\n2. **Complexity of Implementation**: The SOI method involves multiple steps and different types of convolution operations, which may make implementation complex and potentially error-prone. Providing a more detailed implementation guide or open-sourcing the code could help alleviate this concern and improve reproducibility.\n\n3. **Limited Scope of Evaluation**: The experiments focus primarily on speech separation and acoustic scene classification. While these are relevant tasks, the generalizability of SOI to other types of tasks and datasets is not thoroughly explored. Expanding the evaluation to include more diverse tasks and datasets could strengthen the paper’s claims about the broad applicability of SOI.\n\n4. **Quantitative Analysis of Trade-offs**: The paper mentions the trade-off between model quality and computational cost but lacks a detailed quantitative analysis of this trade-off across different configurations. Including more comprehensive metrics and visualizations of these trade-offs could provide deeper insights and help practitioners make more informed decisions when applying SOI.\n\n5. **Parameter Sensitivity**: The paper does not discuss the sensitivity of SOI’s performance to various hyperparameters, such as the number and position of S-CC and SS-CC layers. A sensitivity analysis could reveal potential limitations and guide users in tuning these parameters for optimal performance.\n\nAddressing these weaknesses could significantly enhance the impact and usability of the proposed SOI method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review of our paper. We appreciate your feedback and the opportunity to clarify and improve our work. We are pleased that you recognize the novelty and significance of the SOI method, as well as the thoroughness of our experimental evaluation and the clarity of our writing. Your acknowledgment of our method's potential impact on resource-constrained environments and real-time applications is greatly appreciated.\n\n**Early Layer Impact on SI-SNRi**: The drop in SI-SNRi when the S-CC layer is introduced early in the network can be attributed to the following reasons:\n\n- In our tests, the output of the S-CC layer is extrapolated by duplicating the output of this layer. After concatenation with the current data (as described in Equation 6), the network needs time to correctly merge these states. In the U-Net model, this concatenation happens in the decoder, meaning the sooner we introduce SOI, the less time the model has to recover.\n- Early layers in convolutional neural networks are responsible for capturing fundamental features such as edges and textures. Introducing the S-CC layer too early can disrupt this essential feature extraction process, leading to a less informative representation being passed to subsequent layers. This disruption is more impactful because foundational features are crucial for the overall performance of the network.\n- Later layers in the network generally capture more complex and higher-level features that benefit more from the memorization provided by the S-CC layer. Introducing S-CC at these stages allows the network to maintain a balance between computational efficiency and the preservation of important contextual information.\nRegarding the robustness of the method across different network configurations, it is important to note that the observed performance drop does not imply that the method is fundamentally flawed but rather highlights the importance of carefully selecting the placement of the S-CC layer. We acknowledge that the placement of the S-CC layer is critical and have provided guidelines and empirical evidence in the manuscript to help practitioners make informed decisions about where to introduce the S-CC layer to achieve optimal performance. We would also like to note that the SOI's flexibility and adaptability allow for fine-tuning based on the specific characteristics of the task and the network. \n\nWe will include these points in the revised manuscript to provide clearer guidance on the effective use of the S-CC layer in different network configurations. We will also investigate adaptive strategies for placing the S-CC layer in our future work - thank you for the excellent suggestion! As you also noted, the implementation of SOI is quite complex already, so we fear that adding it right now might further complicate the paper.\n\n**Complexity and Implementation**: We acknowledge the complexity of the SOI implementation and recognize that it might be challenging. We have attempted to counteract this by providing a detailed depiction of the newly introduced layers in Figure 1 and illustrating how inference will look in the simple predefined pattern of odd and even inference in Figure 2. We realize that several implementation details were omitted, so we will add an extensive implementation guide in the appendix of the revised manuscript. Thank you for pointing out this issue.\n\n**Limited Scope of Evaluation**: In Appendix E, we provided an evaluation of our method on a video action recognition task using ResNets and MoviNets. We did not include this experiment in the main text because our lab focuses on audio data. Speech separation is actually one of the most challenging tasks for our method, as it involves dealing with fast-changing input signals.\n\nThe SOI method is designed to be integrated into a variety of convolutional neural network (CNN) architectures. Our experiments with different models (U-Net, GhostNet, and their variants) have demonstrated its effectiveness. We believe that other popular architectures, such as ResNet, VGG, and Inception, can also benefit from the SOI approach. The SOI method includes parameters that can be tuned based on the specific requirements of different tasks and architectures.\n\nWe are conducting ongoing research to validate the generalizability of SOI across more tasks and architectures, including non-CNNs. Future publications will include comprehensive results and analyses from these experiments, providing additional evidence of the method's broad applicability. By publishing our results now, we hope to encourage the research community to apply SOI to their specific use cases and share their findings. This collaborative approach will help identify any limitations and further refine the method.\n\n**Quantitative Analysis of Trade-offs**: When composing the paper, we had to consider our limited resources and make editorial choices to present the method effectively. We decided to focus on a detailed description of the algorithm itself, allowing ML practitioners to replicate and extend our study. While we recognize the benefits of conducting and presenting a detailed analysis of the trade-offs, we decided to concentrate our efforts on a well-described algorithm and a summary of our results. The paper's description of the SOI algorithm and our brief summarization of the results will constitute a well-defined baseline for ML practitioners to start from.\n\n**Parameter Sensitivity**: In the paper, we briefly present our findings on the dependency of SOI's performance on the number and placement of the novel layers in the section describing the results of our algorithm for the speech separation task. We provided some general conclusions on this issue. We agree that a thorough sensitivity analysis could provide important insights into hyperparameter optimization routines, and we encourage ML practitioners to experiment with these parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D4JXcYecRL", "reviewer": "Reviewer_MDRT", "summary": "This paper presents a method called Scattered Online Inference (SOI) aimed at reducing the computational complexity of ANNs. By applying compression and extrapolation techniques, SOI caches partial states of CNNs, allowing it to skip full model recalculation at each inference. The proposed method is positioned as a solution for real-time systems, where energy efficiency and reduced latency are critical.", "questions": "See weaknesses.", "limitations": "1. The proposed method only focuses on CNNs. Extending it to Transformer architectures, which are increasingly popular, presents significant challenges.\n\n2. This manuscript does not discuss the method's limitation, although the authors claim that the discussion is in sec. 1.2.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. The research problem is important. The focus on real-time systems and applications in consumer electronics, where computational resources are limited, adds practical value to the research.\n\n2. The extensive experiments on audio tasks indicate the effectiveness of the proposed method.", "weaknesses": "The presentation of this work is not very good: \n\nFirst, the proposed method is heavily based on STMC, but STMC is not well explained in this manuscript. \n\nSecond, there is no theoretical justification of how much the computational complexity will decrease, making it difficult for reviewers to assess the trade-off between accuracy and complexity.\n\nThird, the Mathematical Formulation part is not well-described. i).Fig. 3 is just put there without any explaination and reference to it. ii). There are too many words but not clear formulae to explain the process. iii). The notation is a little bit strange.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper. We appreciate the time you have taken to provide your insights and the opportunity to improve our work. We are pleased that you recognize the importance of the research problem, the practical value of focusing on real-time systems and consumer electronics, and the effectiveness of our extensive experiments on audio tasks.\n\n> First, the proposed method is heavily based on STMC, but STMC is not well explained in this manuscript.\n\nWhile we did not include an extensive background on STMC, we referenced the STMC paper in our manuscript. We could not provide an additional explanation of the STMC method due to the length requirements for the paper. We understand that the reviewers / readers may not have the time to fully engage with the paper and all relevant references. We will work on fitting a brief overview of STMC to address this concern in the revised manuscript.\n\n> Second, there is no theoretical justification of how much the computational complexity will decrease, making it difficult for reviewers to assess the trade-off between accuracy and complexity.\n\nOur experimental results clearly demonstrate significant computational savings, which serve as a practical validation of our method. While theoretical analysis is valuable, the provided empirical evidence should suffice to illustrate the effectiveness of SOI. To further support our experimental claim, we have included the peak memory usage and inference time (as a measure of latency) in the supplementary PDF added during the rebuttal. These results are presented in Table 9 and Figure 12, continuing the enumeration from the original paper. These results were achieved using the Intel Xeon Gold 6246R CPU with a clock speed of 3.40 GHz. We will incorporate these results into the revised manuscript to provide a more comprehensive evaluation of our algorithm's efficiency. Additionally, we will make the best effort to include theoretical insights in the revised paper to strengthen our case.\n\n> Third, the Mathematical Formulation part is not well-described. i).Fig. 3 is just put there without any explanation and reference to it. ii). There are too many words but not clear formulae to explain the process. iii). The notation is a little bit strange.\n\nWe appreciate this feedback and will add a missing reference to Figure 3 in the text. This figure represents Equations 4-6 in graphical form and was added to help readers better understand our method. We will also extend the caption of this figure to better explain its significance and how it fits into the overall methodology.\n\nThe mathematical formulation is precise and follows standard conventions in the field. The notations are carefully chosen to align with the specific needs of our methodology. However, we will review this section for any ambiguities and provide a clearer explanation.\n\n> The proposed method only focuses on CNNs. Extending it to Transformer architectures, which are increasingly popular, presents significant challenges.\n\nThe paper clearly states its focus on CNNs, which is a deliberate choice given their relevance in the targeted applications (real-time processing). Extending to Transformer architectures is beyond the current scope and should not detract from the contributions made within the context of CNNs. Please note that we are conducting ongoing research to validate the generalizability of SOI across more tasks and architectures, including non-CNNs. Future publications will include comprehensive results and analyses from these experiments, providing additional evidence of the method's broad applicability. By publishing our results now, we hope to encourage the research community to apply SOI to their specific use cases and share their findings. This collaborative approach will help identify any limitations and further refine the method.\n\n> This manuscript does not discuss the method's limitation, although the authors claim that the discussion is in sec. 1.2.\n\nSection 1.2 does discuss the limitations briefly in the first paragraph and through lines 96-98. We acknowledge that this brief discussion could be expanded. We will elaborate on it in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In the attached PDF, we provided peak memory footprint and average inference time measurements for a single S-CC layer in our U-Net for the speech separation task. These results were achieved using the Intel Xeon Gold 6246R CPU with a clock speed of 3.40 GHz. The TFLite Model Benchmark Tool with C++ binary was adopted as the measurement tool. We will incorporate these results into the revised manuscript to provide a more comprehensive evaluation of our algorithm's efficiency.

We hope these additional results strengthen our claims and provide the necessary evidence to support the effectiveness of our method.

Meta-review: This work proposes a novel method that reduces the computational cost of regular ANN models to achieve better online inference efficiency. The presentation and contributions are clear and significant. All concerns of reviewers are fixed well.
Decision: Accept (poster)