Short-Term Memory Convolutions
Authors: Grzegorz Stefański
Krzysztof Arendt
Paweł Daniluk
Bartłomiej Jasik
Artur Szumaczuk
Venue: ICLR 2023 poster (2023)

Abstract: The real-time processing of time series signals is a critical issue for many real-life applications. The idea of real-time processing is especially important in audio domain as the human perception of sound is sensitive to any kind of disturbance in perceived signals, especially the lag between auditory and visual modalities. The rise of deep learning (DL) models complicated the landscape of signal processing. Although they often have superior quality compared to standard DSP methods, this advantage is diminished by higher latency. In this work we propose novel method for minimization of inference time latency and memory consumption, called Short-Term Memory Convolution (STMC) and its transposed counterpart. The main advantage of STMC is the low latency comparable to long short-term memory (LSTM) networks. Furthermore, the training of STMC-based models is faster and more stable as the method is based solely on convolutional neural networks (CNNs). In this study we demonstrate an application of this solution to a U-Net model for a speech separation task and GhostNet model in acoustic scene classification (ASC) task. In case of speech separation we achieved a 5-fold reduction in inference time and a 2-fold reduction in latency without affecting the output quality. The inference time for ASC task was up to 4 times faster while preserving the original accuracy.

Published as a conference paper at ICLR 2023
SHORT-TERM MEMORY CONVOLUTIONS
Grzegorz Stefański, Krzysztof Arendt, Pawel Daniluk, Bartlomiej Jasik, Artur Szumaczuk Samsung R&D Institute Poland {g lg.stefanski, k. ararend, p. p.danilu,, b. b.jasik, a.szumaczuk www.:euuuuuusamuuugg..mom
ABSTRACT
The real-time processing of time series signals is a critical issue for many real- life applications. The idea of real-time processing is especially important in audio domain as the human perception of sound is sensitive to any kind of disturbance in perceived signals, especially the lag between auditory and visual modalities. The rise of deep learning (DL) models complicated the landscape of signal pro- cessing. Although they often have superior quality compared to standard DSP methods, this advantage is diminished by higher latency. In this work we pro- pose novel method for minimization of inference time latency and memory con- sumption, called Short-Term Memory Convolution (STMC) and its transposed counterpart. The main advantage of STMC the low latency comparable to long short-term memory (LSTM) networks. Furthermore, the training of STMC-based models faster and more stable as the method is based solely on convolutional neural networks (CNNs). In this study demonstrate application of this solu- tion to a U-Net model for speech separation task and GhostNet model in acoustic scene classification (ASC) task. In case of speech separation we achieved 5-fold reduction in inference time and a 2-fold reduction in latency without affecting the output quality. The inference time for ASC task was up times faster while preserving the original accuracy.
INTRODUCTION
Convolutional neural networks (CNNs) arose to one of the dominant types of models in deep learn- ing (DL). The most prominent example is the computer vision, e.g. image classification (Rawat & Wang, 2017), object detection (Zhao al., 2019), image segmentation (Minace et al., 2021).
CNN models proved to be effective certain signal processing tasks, especially where the long- term context is not required such speech enhancement (Sun et al., 2021), sound source separation (Stoller et al., 2018), or sound event detection (Lim et al., 2017). Some authors showed that convo- lutional models can achieve similar performance that recurrent neural networks (RNN) a fraction of model parameters (Takahashi & Mitsufuji. 2017). It is also argued that CNNs are easier to par- allelize than recurrent neural networks (RNNs) (Gui et al., 2019; Liu et al., 2022; Rybalkin al., 2021: Kong et al., 2021).
However, unlike RNNs, which can process incoming data one sample at the time, CNNs require a chunk of data to work correctly. The minimum chunk size is equal to the size of the receptive field which depends on the kernel sizes, strides, dilation, and number of convolutional layers. Additionally, overlaps may be required to reduce the undesired edge effects of padding. Hence, standard CNN models are characterized by higher latency than RNNs.
Algorithmic latency which is related to model requirements and limitations such as the minimal chunk size (e.g. size of single FFT frame), model loek-a etc. is inherent the algorithm. can be viewed delay between output and input signals under assumption that all computations are instantancous. Computation time the second component of latency. In case of CNNs it is not linearly dependent the chunk size, due the fact that the whole receptive field has to be processed regardless of the desired output size.
In the case of audio-visual signals, humans able spot the lag between audio and visual stimulus above 10 ms (Mepherson et al., 2016). However, the maximum latency accepted in conversations can be up 40 ms (Staelens al., 2012; Jaekl al., 2015; Ipser al., 2017). For best human-device

---

Published as a conference paper at ICLR 2023
interactions in many audio applications the buffer size is set to match the maximum acceptable latency.
1.1 RELATED WORKS
Many researchers presented solutions addressing the problem of latency minimization in signal pro- cessing models. Wilson et al. (2018) studied a model consisting of bidirectional LSTM (BLSTM), fully connected (FC) and convolutional layers. Firstly, they proposed to use unidirectional LSTM instead of BLSTM and found that it reduces latency by factor of 2 while having little effect on the performance. Secondly, they proposed to alter the receptive field of each convolutional layer to be causal rather than centered on the currently processed data point. In addition, it was proposed to shift the input features with respect to the output, effectively providing a certain amount of future context, which the authors referred to look-ahead. The authors showed that predicting future spectrogram masks comes at a significant cost in accuracy, with a reduction of 6.6 dB signal to distortion ratio (SDR) with 100 ms shift between input/output, compared to zero-look-ahee causal model. It was argued, that this effect occurs because the model is not able to respond immediately to changing noise and speech characteristics.
Romaniuk et al. (2020) further modified the above-mentioned model by removing LSTM and FC layers and replacing the convolutional layers with depth-wise convolutions followed by point-wise convolutions, among other changes in the model. These changes allowed to achieve 10-fold reduc- tion in the fused multiply-accmmulate operations per second (FMS/s). However, the computational complexity reduction was accompanied by a 10% reduction in signal to noise ratio (SNR) perfor- mance. The authors also introduced partial caching of input STFT frames, which they referred to as incremental inference. When each new frame arrives, it is padded with the recently cached input frames to match the receptive field of the model. Subsequently, the model processes the composite input and yields corresponding output frame.
Kondratyuk et al. (2021) introduced a new family of CNN for online classification of videos (MoViNets). The authors developed a more comprehensive approach to data caching, namely layer-wise caching instead of input-only caching. MoViNets process videos in small consecutive subclips, requiring constant memory. It is achieved through so-called stream buffers, which cache feature maps at subclip boundaries. Using the stream buffers allows to reduce the peak memory consumption up an order of magnitude in large 3D CNNs. The less significant impact of 2-fold memory reduction was noted in smaller networks. Since the method is aimed at online inference, stream buffers are best used in conjunction with causal networks. The authors enforced causality by moving right (future) padding to the left side (past). It was reported, that stream buffers lead to approximately 1% reduction in model accuracy and slight increase in computational complexity, which however might be implementation dependent.
1.2 NOVELTY
In this work we propose a novel approach to data caching in convolutional layers called Short- Term Memory Convolution (STMC) which allows processing of the arbitrary chunks without any computational overhead (i.e. after model initialization computational cost of a chunk processing linearly depends on its size), thus reducing computation time. The method is model- and task- agnostic as its only prerequisite is the use of stacked convolutional layers.
We also systematically address the problem of algorithmic latency (look-ahead namely) by dis- cussing causality of the transposed convolutional layers and proposing necessary adjustments to guarantee causality of the auto-encoder-like CNN models.
The STMC layers are based on the following principles:
Input data is processed in chunks of arbitrary size in online mode.
Each chunk is propagated through all convolutional layers, and the output of each layer is cached with shift registers (contrary input-only caching as in Romaniuk et al. (2020)),
providing so-called past context.
The past context is never recalculated by any convolutional layer. When processing a time series all calculations are performed exactly once, regardless of the processed chunk size.
2

---

Published as a conference paper at ICLR 2023
The conversion of convolutional layers into STMC layers is performed after the model is trained, so the conversion does not affect the training time, nor does change the model output.
STMC can be understood a generalization of the stream buffering technique Kondratyuk et al. (2021), although it was developed independently with a goal to reduce latency and peak memory consumption in CNNs with standard, strided, and transposed convolutions as well as pooling layers (as opposed to standard convolutions only in MoViNets). STMC approaches the caching issue differently than MoViNets. It uses shift registers, which cache feature maps of all layers and shift them in the time axis according to the input size. The shift registers enable the processing of data chunks of arbitrary size, where the chunk size may vary in time. Furthermore, shift registers can buffer multiple states of the network, which allows for handling strides and pooling layers. Finally, we propose transposed STMC, making it suitable for a broader family of networks, including U-nets.
METHODS
This section presents the Short-Term Memory Convolutions and their transposed counterpart.
2.1 SHORT-TERM MEMORY CONVOLUTIONS
The STMC and transposed STMC (ISTMC) layers introduce additional cache memory to store past outcomes. Fig. 2 shows the conceptual diagram of our method including the interaction between introduced memory and actual convolution. The cache memory provides values calculated in the past to subsequent calls of the STMC layers in order to calculate the final output without any redundant computations. The goal is to process all data only once.
The design of the STMC shift registers based the observa- tion that CNN layers of an on- line model process input data par- tially overlapping with data from the previous step. The only differ- ence is that the new data is slightly shifted and only a small portion of the input is actually new. As shown in Fig IA, assuming 2 con- volutional kernels of size 2, at time step & only the outputs concern- ing the new 6th frame were not al- ready computed in time step 4-- In order fully reconstruct the out- put without any redundant com- putations, the convolution results
Convolube
Somariae
repl
Rufferd homee
+ Puiding
Colot
Figure 2: A) Short-Term Memory Convolution. B) Transposed Short- Memory Convolution.
A
VIM
veut
2" Conetttton
2TSTTM
DAM
Rapesting farnes ctfffonr it layens
Canmertone
Figure 1: Effect of STMC on the network. A) Convolution redundancy effect in time domain. B) U-Net network with STMCASTMC.
3

---

Published as a conference paper at ICLR 2023
(feature maps) must be saved at each level of the network (Fig. 1B). The number of saved frames in particular STMC block is equal to the size of its receptive field. The input of the next convolutional layer is then padded with previously saved state. This method retains all the past context from the network in an efficient way. The efficiency stems from the fact that all cached data is already pre-processed by the respective layer.
For the purpose of demonstration let us assume that a ID time series data X, E RN composed of N samples is an input of the convolutional layer at time The convolutional layer has kernel h1 R R which can be also represented by Toeplitz matrix H1 N M1+1) assuming valid
padding. We can calculate the output signal Y, (Vt-N :, N y/ as follows:
Y = [H, x?)
(1)
where is activation function applied element-wise. Y is then processed by a second layer (h2 E I M and H2 M)) M1 M2 respectively):
2 Y?²)
(2)
Execution of the above operations for each subsequent time step has a computational complexity of Mi) + O(N M2). To reduce the computational demand, one can apply the incremental inference scheme as described in (Romaniuk al., 2020). Input data would be cropped to match the size of the receptive field, which in our example is M1 + M2 1. In case of K layers of size bounded by M, computational complexity of incremental inference would be O(K²M²).
STMC provides a further improvement, because it caches not only the inputs, but all intermediate outputs of all convolutional layers. These outputs are then used in a subsequent call of the model by concatenating them with the newest data. This technique ensures that all convolutions process only the newest data with kernels placed at the most recent position. Only the newest frames y/- (h X.) need to be calculated:
Y= (Y ( x)))
(3)
where X, E IR-M are the newest input frames and the | represent the operation done by a shift register. For arbitrary sequence P and element a we have:
P = (p1.P2....PP.. Pr))
(4)
(P(a) (pz. p3. - Pa, 0))
(5)
Thus Ze is computed as:
-: (2-10oo10 - (Y1-1l0(11 - xx])
(6)
Complexity of this solution is O(M) + O(M2) or O(KM) for multiple layers under assumptions
given above.
It is worth noting that the effectiveness of STMC increases with the depth of the network. Buffer Y,- can be implemented as simple shift register of size M1 - (Fig. 2). The method may be more complicated in practice depending on the architecture of the CNN. For example, strided convolutions require buffering of multiple states (using multiple shift registers). However, this particular case, STMC would greatly improve the model's processing speed by mitigating stride- induced increased receptive field as the strided convolutions inflate the receptive field by the factor of stride. In comparison, incremental inference would be significantly affected by strided convolutions, because a larger receptive field requires more computations in the upper layers.
STMC can be applied to transposed convolutions as well. One approach is to use STMC with stan- dard convolutions with zero-padding to facilitate upsampling. However, a more optimized approach exists, which we will refer to as transposed STMC (ISTMC). The Fig. 3 shows the diagram of our im- plementation of transposed convolution which can take advantage of caching and presents the partial outputs for each operation. It is worth noting that all pre-convolutional operators are applied only in frequency axis. Firstly, the seattering is applied to inflate the input by introducing zero-valued re- gions inside the input. Each region size depends on the stride of previous transposed convolution. Secondly, the padding is added to ensure desired output shape and then the convolution is applied.
Figure 3: Transposed Short-Term Memory Convolution.
4

---

Published as a conference paper at ICLR 2023
2.2 LIMITATIONS
STMC is task-agnostic method and the presented caching and inference schemes can be applied to any task regardless of the domain. What more STMC can be considered model-agnostrr as well, because it can be applied to stacked convolutional layers anywhere in the model irrespectively of remaining modules.
As mentioned, the fist limitation of STMC is that it can be applied only to convolutional and pooling layers. In contrast RNNs and LSTMs which take only the most recent data as inputs, convolutional models requires longer data segments inputs. STMC simply caches the overlapping parts of the input segments with all intermediate results within the network between subsequent model calls.
The second limitation is that the exact implementation of STMC depends on the model's architec- ture. This work presents how to implement STMC to U-nets and GhostNet models. While it is possible to implement STMC to layers with strides and dilations, would considerably increase the complexity of a network. E.g. strided convolutions require multiple shift registers per layer.
Finally, STMC should be used with causal convolutions, which typically have lower latency than standard convolutions due to the lack of future context. In addition, causal convolutions are effec- tively "anti-causal" in the transposed convolutions, because the deconvolution output depends solely on the current and future input values (contrary to a causal case where output depends only on the past and the current input). This effect can be countered by shifting the model inputs with respect to model outputs, which works for stacked convolutional layers, but some additional modifications may be required in the case of more sophisticated architectures.
3 EXPERIMENTS
In this section we describe the experiments that evaluate the proposed method in terms of inference time and memory consumption. We present model architectures and experimental task alongside our dataset of choice and training setup. We provide the rationale behind the selected solutions.
3.1
SPEECH SEPARATION
3.1.1 EXPERIMENTAL TASK
The speech separation is the main task have selected to present the gains of STMCASTMC scheme. We adopt a general definition of this problem as the task of extracting broadly-understood speech signal from background interference. Because we mainly focused on getting the clean speech signal the task can be understood speech denoising or noise suppression. For clarity, we will refer the stated task as a speech separation.
Evaluation on desktop The TFLite Model Benchmark Tool with C++ binary was adopted as an evaluation tool for inference time and peak memory consumption. Each model was tested in 1,000 independent runs on 8 cores of the Intel(() Xeon(R) Gold 6246R CPU with 3.40 GHz clock, for which the average inference times and average peak memory consumption were calculated. Addi- tionally, the average SI-SNRi metric was calculated to ensure the proposed optimization methods do reduce the output's quality.
Latency L was calculated using the following formula:
L L= Le + + t. + tsTFr + 4SIIT
(7)
Components l, and l comprise algorithmic latency: L, is the length of the current audio chunk, ly the length of the look-ahead signal of the model - both are given in units of time. Components f. fsIFT and ESTFT relate to the computational cost: f; is the inference time, other two are time spent on calculating short-time Fourier transform and its inverse, respectively. Note that the length future audio l is zero in the case of causal models. STFT and iSTFT are computed for one frame only at each step. Our contribution demonstrates improvement in f, thanks to STMC as well as in ly due to introduced causality. For each model Latency estimated using lowest possible length of input with which is possible to achieve real-time processing, i.c. to have real-timo factor (RTF) equal or smaller then 1.
5

---

Published as a conference paper at ICLR 2023
Evaluation on mobile device To confirm the portability of the proposed method other devices, models were also evaluated on a mobile device using the same setup as on the desktop. We used Samsung Fold 3 SM-F9 UM-F925U equipped with Qualcomm Snapdragon 888. The models were analyzed using the TFLite Model Benchmark Tool, but this time 4 cores were used.
STMC with different model sizes To provide some intuition on how STMC may perform with models of different sizes we applied STMC to the same U-Net architecture with a scaled number of kernels. We tested 16 models, which sizes range from 76 KB to 32 806 KB. To smooth out measured characteristics, each model size was trained 5 times and we report the average values for all our results. TFLite conversion for each model was done similarly to our main experiment.
Influence of look-ahead As not every application requires such small latency, STMC may still be introduced to reduce computational complexity. We provide the experimental results where we measured SI-SNRi of models using different lengths of look-ahead. We trained 5 models per size of look-ahead. Our tested look-aheat ranges from 0 to 7 frames with a step of frame, where is our causal network and 7 is our non-causal network from main experiment.
3.1.2 MODELS AND TRAINING PROCESS
The study is based on the U-net architecture as it consists of convolutions and transposed convo- lutions alongside skip connections. Because of that, it renders a greater challenge for STMC due to the "anti-causality" of the decoder, as described in Section A. What is more, it is widely use for solving audio domain tasks and it is proven to achieve very good results. In total, seven U-nets compared: U-Net, U-Net INC. STMe. Causal U-Net, Causal U- U-N Causal U-Net STMC. and Causal U-Me The models differ causality and in STMC applied to the encoder and decoder. In addition, both U-Net and Causal U-Net process the whole data in single inference, so they are effectively off-line models. Other models work in an online mode processing data in chunks.
U-Net As our benchmark model, we incorporated network based on the U-Net architecture (Ron- neberger et al., 2015). The original architecture was simplified by removing max pooling operations, so the encoding is done solely by convolutions. We adapted the model for the audio separation task and achieved satisfying results with a model of depth 7. Each convolutional layer comprises a number of kernels of length 3 in the time domain, which leads to the receptive field of 29 for an assembled network. The kernel size in the frequency domain is set to 5. As the benchmark network consists of convolutional layers which are locally centered (Favero et al., 2021) frame, the receptive field of 14 for both past and the future context can be derived. The STFT input with real and imaginary parts as separate channels is utilized. resulting in complex mask for the input signal as the output of the network. A STFT window of size 1024 and hop of 256 are used.
current time
Causal U-Net To show the influence of causality we also trained Causal U-Net models. In all the considered causal models we used kemels of size 2 in the time domain to reduce the computation cost and the receptive field, as well as to compress the model size. The preliminary tests showed that the output quality of the casual model is not affected by the change. Such an approach allows to preserve only vital calculations and removes additional padding operations. The receptive field of the network is effectively reduced by half. To achieve causality, the output of the model was shifted by 7 STFT frames. To align the data inside of the network, all skip connections also needed to be shifted by the amount dependant on their depth inside the model.
Incremental inference We applied the incremental inference method (Romaniuk et al., 2020) to U-Net and Causal U-Net by cropping the original input size to the size of receptive field, yielding 29 STFT frame imput for U-Net and 15 STFT frames for the Causal U-Net. We will refer to this network as U-Net and Causal U-NetINC-
Short-Term Memory Convolutions Similarly to the incremental inference, STMC was applied to both U-Net and Causal U-Net networks. Firstly, we applied STMC inside the encoder and inside the decoder with zero-padding and output cropping (i.e. without (STMC). We will denote these networks as U-Net and Causal U-Netsime- To show the difference in inference time of STMC with cropping and with tSTMC, we also separately trained causal model consisting of both STMC and (STMC. This network is denoted by Causal U-Net,sTMe- The input size in the temporal axis for this method was set to and for U-Net and Causal U-Net, respectively.

---

Published as a conference paper at ICLR 2023
Dataset and training conditions The Deep Noise Suppression (DNS) Challenge Interspeech 2020 dataset (Reddy et al., 2020), licensed under CC-BY 4.0, was adopted for both training and validation of networks
it be easily processed for speech separation task. During training the models have seen around 100M randomly sliced 3-second audio samples. All models (U-Net, Causal U-Net and Causal U-Net,sTMc) were trained on Nvidia A100 with batch size of 128 and Adam optimizer with learning rate of le-3, Each model was trained for 100 epochs which took about 90 hours per model. The further details about the training scheme and used losses were omitted because they do not affect neither the utility nor the effectiveness of the proposed method.
All models were converted into TFLite representation with default converter settings and without quantization.
3.2 ACOUSTIC SCENE CLASSIFICATION
3.2.1 EXPERIMENTAL TASK
To show that our method has also an impact in small CNN models we applied it to a GhostNet (Han et al., 2020) architecture trained for an acoustic scene classification (ASC) task. We tested the inference time and peak memory consumption on CPU (Intel(R) Xeon(R) Gold 6246R CPU with 3.40 GHz clock), GPU (Nvidia A100), and on CPU with XNNPACK optimization. For comparison, we also tested LSTM and VibileViT (Mehta Rastegari, 2022) networks.
3.2.2 MODELS AND TRAINING PROCESS
GhostNet Our classification network consist of 3 ghost bottlenecks and ends with convolutional layer and average pooling. The model fed with a magnitude spectrogram of size the time axis and 256 in the frequency axis.
LSTM We tested the RNN architecture with LSTT blocks with and units and fully connected layers with 16 and units respectively. The model accepts an input of time bin of magnitude spectrogram with 256 frequency bins.
Mobile We also implemented and trained the Vibile network. In contrast to the original model described in (Mchta & Rastegari, 2022) we set the number of input channels to I and the number of output classes to 1.
Dataset and training conditions The TAU Urban Acoustic Scene 2020 Mobile dataset (Heit- tola et al., 2020), was adopted for both training and validation of networks as dataset for ASC task. We augmented the data by applying peak normalization, cropping and conversion to magni- tude spectrograms in specfied order. We followed DCASE 2020 challenge task 1A instruction train/test split. The further details about the training scheme and used losses were omitted because they do not affect neither the utility nor the effectiveness of the proposed method.
All models were converted into TFLite representation with default converter settings and without quantization.
4 RESULTS
4.1
SPEECH SEPARATION
Table 1: Results of evaluation on desktop using TFLite Model Benchmark Tool. 'C.' stands for causal. *Difference in metrics due to separate learning of models.
Results on desktop The results for all mod- els are presented side-by-side in Table 1. Com- paring to standard convolutions, the incremen- tal inference (INC) method achieves 8.00 8.00-fold and a 29.74-fold latency reduction in the case of non-causal and causal setting, respectively. Our method (STMC) outperforms INC with further 1.32-fold latency reduction for non-casual and 2.52-fold reduction for causal variants. STMC achieved also approximately 5.68-fold reduction
Name
SI M-SRRI (4B)
Inference time (036)
Latency (mss
Peak memory (MB)
Size
(KB)
U-Net
3,00
1831.02 +: 17.62
4-403.25
170.36
145
INC
X.00
151.92
+ 12.25
550.54
48.97
14 025
U--Net
7.58
962.61
25.91
138.18
C. INC
7.58
55.64
4: 2.47
133.98
31.57
9.367
C. STMC
7.58
10.37
1: 0.69
52.07
27.50
9.376
$1.50

---

Published as a conference paper at ICLR 2023
in inference time, compared to INC. The impact of STMC on the non-causal U-Net is lower as most of the computational cost of U-NetgTMC is due to the need for waiting for the fu- ture context (convolutional kernels centered with respect to the output). The implementation of transposed STMC in the decoder of the Causal U-Net model gives only a slight im- provement in latency (0.57 ms), but it reduces the peak mem- ory consumption by approximately 20%. The model size is similar for standard, INC, and STMC variants. The causal models are approximately 33% smaller than the non-causal ones, which is caused by the smaller kernels as described in Section 3.1.2.
Table 2: Results of evaluation on mobile using TFLite Model Bench- mark Tool. 'C'' stands for causal.
Name
Inference time (ms)
Latency (mm)
Peak memocy (MB)
U-Net
262.44 + 26.81
294.64
164.93
INC
58.51
2:3.4
323,72
44. 14
STMC
19.26
288.96
U.Neeet
240.95
2633.15
C. INC
14.02 00000071 0.71
50.74
26.53
C. STMC
3.15
003-::0.00 0.10
28.85
21.54
ISTMC
2.77
0.26
28.47
16.11
Results on mobile device The results of mobile evaluation are collected in Table 2. They further confirm the conclusions drawn from desktop results. Since the models used in this test the same as in the case of desktop evaluation, we did not include the model size and SI-SNRi metrics in the table.
The inference time and latency achieved on the mobile device are significantly lower than on the desktop, despite the same evaluation method used on both platforms. The inference time reduction factor between Causal U- and Causal U-NetasTMc is 5.06. The usage of tSTMC on this device has relatively higher impact than on the desktop, as this case it reduces the inference time by 12.06% (compared to 5.50%). Similarly to the desktop, the lowest peak memory consumption was achieved by Causal U-NMC
STMC with different model sizes The sults for this set of experiments are displayed in fig. 4 by set of plots. As was high- lighted before, both INC and STMC models have the same SI-SNRi metrics and the depen- dency of this metric on model size is showed in fig. 4A. Absolute peak memory consump- tion improvement from STMC scales linearly with model size (fig. 4B), although the rel- ative improvement compared to INC linearly
A
B
improsment (M00)
DANSS (80)
5.0
2.5
0.0
10
20
Model (MBB)
D
Model size (MB)
200
INC STMC
Intorees Time
decreases from 16% to 11%. STMC has
significant influence inference time in all
tested models which can be seen in fig. 4C-D.
20
30
For both inference patterns, similarly ab- solute peak memory improvement, inference time seems scale linearly with model size (fig. 4C), Speed-up of STMC over INC for this architecture is constant and amounts to around 81% %, but drops for models of size 1.32
Madal diou (MI)
Midiel abo (MT))
Figure 4: Influence of model size on STMC. A) SI-SNRi. B) Improvement over peak memory con- sumption. C) Inference time. D) STMC speed-up over INC.
MB and smaller to around 74%.
Influence of look-ahead As can be seen in fig. 5 the look-ahead has positive impact on metrics. It should also be noted that this experiment confirms that the 6% decrease of metrics is attributed exclusively to the lack of future content as in this test we actu- ally interpolated between our causal and non-causal networks from main experiment. Determined characteristic is not linear and the first frames seems to be more significant than the furthest to the right future frames ones as first frame increases SI-SNRi by 0.177 dB and the last one by 0.019 dB. In this plot we included error bars as the difference between the same models is more significant than in every other experiment we presented in this work. Within error bars for specific look-ahead falls all SI-SNRi results validation dataset we achieved for all 5 models of particular look-ahead we trained.
SNSS (cB)
8:11
Lock ahead [fuuess
Figure 5: Influence of look- ahead on SI-SNRi.
8

---

Published as a conference paper at ICLR 2023
4.2 ACOUSTIC SCENE CLASSIFICATION
Applying STMC to the Ghost- Net architecture has no impact on accuracy metrics. In addition we observed 4.84-fold reduction of CPU inference time, 34% im- provement of inference time for GPU and 2.24-fold reduction for XNNPACK. In case of peak mem- ory consumption we observed a 22% reduction for CPU and 30% reduction for XNNPACK with 4 KB increase in model size. We do not report peak memory con- sumption for GPU inference be-
Table Results of ASC experiment.
Netwock
Top Acc (8)
Inferenze time (mc)
Peak memory (MB)
Number of pommmeters
Iaput size
GhostNet
4,706 6.167
9:76
GhostNee.
65,83
0.912 0.504
296
94
1,086 5.612
11.49
GhosdNe me
0.973 1.866
7:59
GhossNec. te
65.83
0,602 0.32]
800
x
GhostNet the
0,484 2,447
8.10
LSIM
0,095 0.010
7.50
L.ST
60.84
0,099
0,0744
95
137
1x
LSIM WARGCE
0.119 0,548
Mobile XXS..
MoNilVT XXS
69.58
419.203
$2.948
11,456
38.11
300
256
cause the tool we used does not provide this information. By comparing STMC GhostNet to the LSTM we can observe that inference time of LSTM is up to 10x faster but comes at the cost of 12x higher number of parameters. It is important to note that the LSTM's supremacy of inference time over STMC decreases 4x faster inference for XNNPACK. Additionally, our tests showed that CNNs were overall more unstable in case of inference time than LSTMs. Accuracy of LSTM model was at 60.84% which is lower than STMC GhostNet with accuracy of 65.83%. The Mo- biTeViT, at 1.3M parameters, scored the highest observed accuracy of 69.58% alongside the highest inference time of 52 ms. This model got the best metrics for the given ASC task but high inference time discredits its use for online processing. The best performing model on TAU dataset which was submitted in the DCASE 2020 challenge task IA had an accuracy of of 74.4%, thus all models in our study can be deemed viable.
5 CONCLUSION
In this paper, we presented the method for inference time, latency, and memory consumption reduc- tion. It relies on buffering of layer outputs in each iteration. The overall goal is to process all data only once and re-use past results of all convolutional layers. The method was tested on an speech separation task on two devices; desktop computer and mobile phone and ASC task with Ghost- Net on x86 CPU and A100 GPU. In total for audio separation, 7 variants of a U-Net architecture were compared, differing in causality and buffering types. On both x86 and ARM, approximately a 5-fold reduction of inference time was achieved compared to the incremental inference, which is considered the state-of-the-an method for online processing using convolutional models. The STMC method has also 44-61% % lower latency and 30-39% lower peak memory consumption, depending on the test platform. All the causal models used in the test achieved approximately 6% reduction in SI-SNRi compared to their non-causal counterparts. The cause of the score reduction is attributed to the lack of future context and not the used caching method.
The STMC method proved to significantly reduce the number of operations inside CNNs, which may be crucial for many DL applications, e.g. deploying models on embedded devices. The method can be applied to most of the existing CNN architectures and can be combined with several different methods used for time series processing, e.g. dilated causal convolutions (van den Oord et al., 2016) or temporal convolutions (Lea et al., 2017). Using STMC and their transposed counterpart - ISTMC does not require any retraining of the model. It is, however, strongly advised to use causal models in order to reach near real-time processing.
The described approach is also relevant for applications beyond the audio separation task presented in this work. Since the STMC method leads to the reduction of inference time, latency, and peak memory consumption, it can be used for systems where the size of data is too large for the available memory computational power, e.g.: medical imaging (Blumberg et al., 2018), radio astronomy imaging (Corda et al., 2020), FPGA (Ma et al., 2018), video analysis (Lea et al., 2017), checksum computation (Filippas et al., 2022). We also believe that the described method may be useful for low latency systems on chip (SoC) for online processing applications (Park et al., 2022).

---

Published as a conference paper at ICLR 2023
REFERENCES
Stefano B. Blumberg, Ryutaro Tanno, lasonas Kokkinos, and Daniel C. Alexander. Deeper image Training low-memory neural networks for 3d images. In Medical Image Com- puting and Computer Assisted Intervention - MICCAI 2018, pp. 118-125. Springer International Publishing, 2018. ISBN 978-3-030-00928-11
quality transfer:
Stefano Corda, Bram Veenboer, Ahsan Javed Awan, Akash Kumar, Roel Jordans, and Henk Cor- poraal. Near memory acceleration on high resolution radio astronomy imaging. In 2020 9th Mediterranean Conference on Embedded Computing (MECO), pp. 1-6, 2020. doi: 10.1109/ MECC049872.2020.9114444
Alessandro Favero, Francesco Cagnetta, and Matthieu Wyart. Locality defeats the curse of dimen- sionality in convolutional feacher-student scenarios. In Advances in Neural Information Process- ing Systems, 2021. URL htttps://ppeereeiieeeee http://ppppreeeeeeeeeeeeeeeeeeee
Dionysios Filippas, Nikolaos Margomenos, Nikolaos Mitianoudis, Chrysostomos Nicopoulos, and Giorgos Dimitrakopoulos. Low-cost online convolution checksum checker. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 30(2):201-212, 2022. doi: 10. 10.1109.V
3119511.
Tao Gui, Ruotian Ma, Qi Zhang, Lujun Zhao, Yu-Gang Jiang, and Xuanjing Huang. Con-based chinese ner with lexicon rethinking. In Proceedings of the Twenty-Eightl International Joint Conference on Artificial Intelligence, IJCAI-19, pp. 4982-4988. International Joint Conferences on Artificial Intelligence Organization, July 2019. doi: 10.24963/ijcai.. 201c//i2011//92.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. TAU Urban Acoustic Scenes 2020 Mo- bile, Development dataset, February 2020. URL https: www...org/1.... 0tt.110.222112eedd/eee 3819968.
Alberta Ipser, Vlera Agolli, Anisa Bajraktari, Fatimah Al-Alawi, Nurfitriani Djuafara, and Elliot D. Freeman. Sight and sound persistently out of synch: stable individual differences in audiovisual synchronisation revealed by implicit measures of lip-voice integration. Scientific Reports, 7(1):
46413, April 2017. ISSN 2045-2322. doi: 10. 10.1038/srep44413.
Philip Jaekl, Jakob Seidlitz, Laurence R. Harris, and Duje Tadin. Audiovisual delay novel cue to visual distance. PLOS ONE, 10(10)11-12, October 2015. doi: 10. /..33a/joural.ppot wow.:um0....o...011...01 0141 125.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang. Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16020- 16030, June 2021. URL bectps://openaceeo html Haal//oonda Video_ Ladeo__Recogniiio__ccess 2021 parpppoa_eccc._eeo_pppaeeeeeee.. html.
thecvf..eccec com/ content /twpntteeec/PP2221111111 CVVR2021/
Jun Kong, Leixin Zhang, Min Jiang, and Tianshan Liu. Incorporating multi-level and attention mechanism for chinese clinical named entity recognition. Journal of Biomedical Informatics, 116: 2021. ISSN 1532-0464. doi: 10. 10.1006//.bb..221110777
Colin Lea, Michael D. Flynn, René Vidal, Austin Reiter, and Gregory D. Hager. Temporal convo- lutional networks for action segmentation and detection. In Proceedings 30thh IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. pp. 1003-1012. Institute of Electrical and Electronics Engineers Inc., November 2017. doi: 10. 2tt.:/CCP..2017.11..
Hyungui Lim, Jeongsoo Park, and Yoonchang Han. Rare sound event detection using 1D convolu- tional recurrent neural networks. Technical report, DCASE 2017 Challenge, September 2017.
Yingying Liu, Peipei Li, and Xuegang Hu. Combining context-relevant features with multi-stage attention network for short text classification. Computer Speech & Language, 71:101268, 2022. ISSN 0885-2308. doi: 10. 10161006.s...l.. 1021..0166...
10

---

Published as a conference paper at ICLR 2023
Yufei Ma, Yu Cao, Sarma Vrudhula, and Jac-sun Seo. Optimizing the convolution operation to accelerate deep neural networks on fpga. IEEE Transactions on Very Large Scale Integration (VLSI) Systems, 26(7):1354-1366 2018. doi: 10.1009TVLLS2201..8816000
Andrew Mcpherson, Robert Jack, and Giulio Moro. Action-sound latency: Are our tools fast enough? In The International Conference on New Interfaces for Musical Expression, April 2016.
Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight,
and mobile- friendly vision transformer. International Conference on Learning Representations, 2022. URL. https: www.revveeeeeveeeeeeeee ew. net. //oo/oormm" -eeiiii=-ihssho
Shervin Minace, Yuri Y. Boykov, Fatih Porikli, Antonio Plaza, Nasser Kehtarnavaz, and Demetri Terzopoulos. Image segmentation using deep learning: A survey, IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10. 1211099TAAMI.220133595990
Jun-Seok Park, Changsoo Park, Suknam Kwon, Hyeong-Seok Kim, Taeho Jeon, Yesung Kang, Heonsoo Lee, Dongwoo Lee, James Kim, YoungJong Lee, Sangkyu Park, Jun-Woo Jang, SangHyuck Ha, MinSeong Kim, Jihoon Bang. Suk Hwan Lim, and Inyup Kang. A multi-mode 8k-mac hw-utilization aware neural processing unit with unified multi-precistie datapath in 4nm flagship mobile soc. In 2022 IEEE International Solid-State Circuits Conference (ISSCC), volume 65, pp. 246-248, 2022. doi: 10.
Waseem Rawat and Zenghui Wang. Deep convolutional neural networks for image classification: A comprehensive review. Neural Computation, (00):22522244999 2017. doi: 10.1162/neco.a. 00990.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. In Proc. Interspeech 2020, 2020.
Michal Romaniuk, Piotr Masztalski, Karol Piaskowski, and Mateusz Matuszewski. Efficient Low- Latency Speech Enhancement with Mobile Audio Streaming Networks. In Proc. Interspeech 2020, pp. 3296-3300, 2020. doi:
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Vladimir Rybalkin, Jonas Ney, Menbere Kina Tekleyohannes, and Norbert Wehn. When massive gpu parallelism ain't enough: A novel hardware architecture of 2d-Istm neural network. ACM Trans. Reconfigurable Technol. Syst., 15(1), November 2021. ISSN 1936-7406. doi: 10.1145/ 3469661.
Nicolas Staelens, Jonas De Meulenaere, Lizzy Bleumers, Glenn Van Wallendael, Jan De Cock, Koen Geeraert, Nick Vercammen, Wendy Van den Broeck, Brecht Vermeulen, Rik Van de Walle, and Pict Demeester. Assessing the importance of audio/video synchronization for simultancous translation of video sequences. Multimedia Systems, 18(6):4554457 November 2012. ISSN 1432-1882. doi: 10.10077600533-002--266-4
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: multi-scale neural network for end-to-end audio source separation. In The 19th International Society for Music Informa- tion Retrieval Conference, 2018. URL http: http::://mmmm20000 ircam. f:/doc/pdfs/205 Paper. pdf.
Yuhang Sun, Linju Yang. Huifeng Zhu, and Jie Hao. Funnel Deep Complex U-Net for Phase-Aware Speech Enhancement. In Proc. Interspeech 2021, pp. 161-165, 2021. doi: 10.214377Interspeechh. 2021-10.
Naoya Takahashi and Yuki Mitsufuji. Multi-scal multi-band densenets for audio source separation. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pp. 21-25, 2017. doi: 10.1 209/WASPAA.2017.880

---

Published as a conference paper at ICLR 2023
Aäron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A Generative Model for Raw Audio. In Proc. 9th ISCA Workshop Speech Synthesis Workshop (SSW 9), pp. 125, 2016. URL https::t//// http///www.vgg//eeeee/ee / 909. 034 1.3499.pde
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural net- works. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML'16. pp. 1747-1756. JMLR.org, 2016. URL http: / //proceedings.. ml..mmlr www.m.......apemmmmmmmmmmmmmmm.m v48/ oordi 6 pdff
Kevin Wilson, Michael Chinen, Jeremy Thorpe, Brian Patton, John Hershey, Rif A. Saurous, Jan Skoglund, and Richard F. Lyon. Exploring tradeoffss in models for low-latency speech enhance- ment. In Proceedings of the 16th International Workshop on Acoustic Signal Enhancement, 2018.
Zhong- Qiu Zhao, Peng Zheng, Shou-Tao Xu, and Xindong Wu. Object detection with deep learning: A review. IEEE Transactions Neural Networks and Learning Systems, 30(11):3222-3222 2019. doi: 200.0//NNNNS2211.2878810
12

---

Published as a conference paper at ICLR 2023
A EFFICIENT CAUSALITY
Non-causal convolutions can be used in online inference by either waiting enough time for their receptive field be filled with "future" data, which increases latency, by replacing the future data with zeros or some other values, e.g. symmetrically reffecting the data. The latter approach, however, degrades output quality at the most recent frames, which coincidentally the online inference. Therefore, causality is typically required for robust online inference using convolutional networks. Causality is also desired for STMC, because STMC layers cache the past context as described in Section 2.1.
the most important for
Technically, the convolution operation does not take any assumptions on causality. In many applica- tions, especially in computer vision, kemnlls are centered with respect to the current output element of interest. For example, let C be a convolutional layer with dimensional kernel and valid padding. Let A and B be sample input and output tensors with dimensions of 5 x 1 and 3 respectively. We then have:
22
k3
0
d22
b2
CAT
k1
k2
(33
by
B
(8)
0
Ka
Q4
b4
When the convolution is used this way, both the past and future context is needed.
One of the methods for enforeing causality is by using masked convolutions, in which the kernels are multiplied by non-symmetric masks (Van Den Oord et al., 2016) before being convolved with the input. However, masked convolutions are more resource consuming than standard convolutions due to additional multiplications. Therefore, in signal processing it more desired to enforce causality by shifting the input with respect to the output (van den Oord et al., 2016). In other words, the elements of tensor B can be re-indexed to reflect temporal dependency in way that b, does not depend on any a, for j i:
ka
lts
Q2
bs
CAT
=
K1
K3
(3
by
B
(9)
0
0
k1
kg
K3
CAA
b5
b; k 410:-2 + kga; 1
kza;
(10)
Our case study also requires the model to reconstruct task-specified version of an input signal from its convolved representation. The simplest and effective approach obtain such an inverse of the convolution is to perform transpose-like operation on its Toeplitz matrix, as depicted in equation (11). Note that convolution and transposed convolution matrices are in transpose-relation with respect to shape and structure.
not share kernel parameters, but their
interesting phenomenon occurs when "causal" convolution transposed, because we then get:
k
0
T
k
0
bb
d
C²B
k
ba
0g
TT
(11)
K
bs
0g
0
Kg.
Los_
df = lib. + Kybe+1 + k4b+42
(12)
which is effectively "anti-causal", i.e. the deconvolution output depends solely current and future input values (contrary to a causal case where output depends only on past and current input). This effect can be countered by shifting model inputs with respect to model outputs:
a
0
bbg
on
C
Kg
0
0
Di
0g
TT
(13)
M
bs
0g
KL
Los
d4
Rig,
big
k/b
(14)
13

---

Published as a conference paper at ICLR 2023
This approach works for stacked convolutional layers, but in the case of more sophisticated archi- tectures some modifications may be required. For example, U-Net models have skip connections between convolutions and transposed convolutions. The data passed through each skip connection must also be properly shifted with respect to the outputs of transposed convolutions. The shift value depends the network architecture and may be different each level.
14


Reviews:
[{"review_id": "VsdWkxZp8hk", "reviewer": "Reviewer_xgZF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors identify an implicit characteristic of convolutional layer operations and exploit that to show benefits in operational speed/efficiency for CNN transposed CNN layers and pooling layers in deep nets. The idea itself is neat and clear. And the paper is presented well. The authors evaluate the proposed on non-causal and causal formulations of U-Net. ", "summary_of_the_review": "I believe this is a reasonable contribution. The authors acknowledge the limitations of the work which is appreciated. \nNevertheless, the evaluations are minima and some of the questions posed above are to be addressed to get clarity on the choices made in the proposal.  ", "strength_and_weaknesses": "Overall the paper is a reasonable contribution. Although the direct technical novelty is minimal there is quite an empirical value to the observations made in the STMC/tSTMC setups. The paper is well written and easy to understand. \n\nQuestions/Weaknesses? \n1) The authors have already mentioned that the proposed models' efficacy is dependent on the depth of the network and losing gains as depth increases. On this note can the authors confirm two things: (a) For networks with strided convolutions we cannot expect to see such gains since the buffer size increases drastically (any idea/estimate how bad this can be?) and (b) the latency calculations included times for stft/istft along with other, but given the simplest case of memory-1 buffer shown here in this paper these two times are fixed across all the models (rows in table 1)? Or is it not, am I missing something basic here! \n2) Does the TFlite model do any optimizations that may have inadvertant influence on the estimates shown in Table 2? i.e., trying to better disentangle the influence of the proposal vs. implicit optimization of the packages? \n3) What was the rationale for using U-Net based models for benchmarking as opposed to classical CNNs for other types of CV/AV tasks?  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Smrc3Tzsxr", "reviewer": "Reviewer_EphW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to store convolution outputs that have been already seen for the past data in a shift register and thus tries to avoid recomputation of the convolution output from those samples and works with the newly arrived data for online inference in convolutional neural nets. The final goal is to reduce the overall inference time. In the experiments, deep noise suppression (DNS) data is used in order to demonstrate the speed of the proposed model in a U-net architecture. Experiments compare causal and non-causal convolutions, their short-term memory (STMC) implementation and combination of STMC with transposed STMC (tSTMC). ", "summary_of_the_review": "The model introduces the use a shift-register to cache the intermediate convolution outputs at intermediate layers to reduce the inference time of convolutional networks. The method is applied only at the inference time and not during training. Experimental results show significant speed up in inference time with a reduced memory requirement. However, the best performing version of the model also degrades the signal quality. There is a trade-off between time vs. quality. The paper can get confusing at times (e.g., what is the task?). Analysis at the appendix could have been added to the main discussion.  ", "strength_and_weaknesses": "Strengths:\n- Experimental results show that the proposed model reduces the inference time and memory requirements.  \n- Some limitations of the proposed model have been clearly mentioned. \n\nWeaknesses: \n- The application domain is not clear. Even though the paper mentions audio separation several times, it uses deep noise suppression challenge data which is mainly used for noise suppression. In addition, independent of the application, the paper does not mention the loss function and what the outputs from the model are. \n\n- The transposed version of the STMC model is not well-described. For example, in Fig. 2B, from where do we get the [0,1] into the buffer in the rightmost part of the figure? \n\n- Figures are not described in detail in the main text. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8bzXNlCvn87", "reviewer": "Reviewer_TkYW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the Short-Term Memory Convolution (STMC) layers, which is designed to be a faster version of the convolution layers (in terms of computation time) and also achieve lower latency for real-time application. The paper first presents the approach, based on shift registers. The proposed layer is then evaluated on an audio separation task and compared with baseline models. It is shown to yield similar performance than the baseline with significant gain in inference time and latency.", "summary_of_the_review": "This paper Introduce a novel layer which looks useful for audio real-time applications. But in the current form, the paper is not clear enough on the methods and the experiments are not bad but seems limited. Hence I'll recommend rejection, but I would be happy to revise if the authors can adress my concerns.", "strength_and_weaknesses": "Strengths:\n- The paper is well structured and easy to read\n- The proposed approach seems novel AFAICT and well situated w.r.t. previous works.\n- The experiments shows the benefit of the proposed layer for real-time applications\n\nWeaknesses:\n- The description of the proposed layer is not very clear (see next question)\n- The experiments are a bit limited.\n- The task is not necessarily the most representative for audio and speech application, more experiments on other tasks such as ASR or VAD would have been welcome.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster