Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors identify an implicit characteristic of convolutional layer operations and exploit that to show benefits in operational speed/efficiency for CNN transposed CNN layers and pooling layers in deep nets. The idea itself is neat and clear. And the paper is presented well. The authors evaluate the proposed on non-causal and causal formulations of U-Net. 
strength_and_weaknesses: Overall the paper is a reasonable contribution. Although the direct technical novelty is minimal there is quite an empirical value to the observations made in the STMC/tSTMC setups. The paper is well written and easy to understand. 

Questions/Weaknesses? 
1) The authors have already mentioned that the proposed models' efficacy is dependent on the depth of the network and losing gains as depth increases. On this note can the authors confirm two things: (a) For networks with strided convolutions we cannot expect to see such gains since the buffer size increases drastically (any idea/estimate how bad this can be?) and (b) the latency calculations included times for stft/istft along with other, but given the simplest case of memory-1 buffer shown here in this paper these two times are fixed across all the models (rows in table 1)? Or is it not, am I missing something basic here! 
2) Does the TFlite model do any optimizations that may have inadvertant influence on the estimates shown in Table 2? i.e., trying to better disentangle the influence of the proposal vs. implicit optimization of the packages? 
3) What was the rationale for using U-Net based models for benchmarking as opposed to classical CNNs for other types of CV/AV tasks?  
clarity,_quality,_novelty_and_reproducibility: Clerity and quality are good. All results are reproducible. 
summary_of_the_review: I believe this is a reasonable contribution. The authors acknowledge the limitations of the work which is appreciated. 
Nevertheless, the evaluations are minima and some of the questions posed above are to be addressed to get clarity on the choices made in the proposal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the Short-Term Memory Convolution (STMC) layers, which is designed to be a faster version of the convolution layers (in terms of computation time) and also achieve lower latency for real-time application. The paper first presents the approach, based on shift registers. The proposed layer is then evaluated on an audio separation task and compared with baseline models. It is shown to yield similar performance than the baseline with significant gain in inference time and latency.
strength_and_weaknesses: Strengths:
- The paper is well structured and easy to read
- The proposed approach seems novel AFAICT and well situated w.r.t. previous works.
- The experiments shows the benefit of the proposed layer for real-time applications

Weaknesses:
- The description of the proposed layer is not very clear (see next question)
- The experiments are a bit limited.
- The task is not necessarily the most representative for audio and speech application, more experiments on other tasks such as ASR or VAD would have been welcome.
clarity,_quality,_novelty_and_reproducibility: I have some concern about the clarity of the paper:
- Section 2.1 is quite confusing for me, I don't get how the first and second equation of page 4 are equals, could the authors provide more mathematical details? also please number the equations.
- I think it would be help to also provide the mathematical description of standard convolution layer.

About the experiments:
- The experiments are quite limited, I want to see the results of MoViNets and of similar related works to be able to judge the benefit of the proposed approach.
- Simple baselines with very low latency are also missing. I think it's not hard to design a LSTM or CNN with 10ms latency (just one frame), it would be helpful to see the trade-off in performance there.
- I'm a bit confused by how the inference time is computed, is it computed over the whole testset? Please clarify. It would be useful to also report the Real Time Factor or equivalent (the time it takes to compute an input of a given duration) as this is more important in real-time application.
- Why is the processing time included in the latency? In my experience it doesn't affect the latency of real-time systems as long as the processing time is shorter than real-time (i.e. shorter than 10ms to process 10ms of input) for instance.
- Do the authors have an intuition about why the mobile is 5x faster than the desktop? It looks counter-intuitive as the desktop has more processing power.
- The GhostNet experiments in the appendix are quite interesting, would it be possible to add them to the main text? I think they are strengthening the paper.

The proposed approach seems novel, and the findings are significant for real-time audio applications. 

In terms of reproducibility, the proposed method would be difficult to reproduce because of the vague mathematical description and the absence of shared code.

summary_of_the_review: This paper Introduce a novel layer which looks useful for audio real-time applications. But in the current form, the paper is not clear enough on the methods and the experiments are not bad but seems limited. Hence I'll recommend rejection, but I would be happy to revise if the authors can adress my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to store convolution outputs that have been already seen for the past data in a shift register and thus tries to avoid recomputation of the convolution output from those samples and works with the newly arrived data for online inference in convolutional neural nets. The final goal is to reduce the overall inference time. In the experiments, deep noise suppression (DNS) data is used in order to demonstrate the speed of the proposed model in a U-net architecture. Experiments compare causal and non-causal convolutions, their short-term memory (STMC) implementation and combination of STMC with transposed STMC (tSTMC). 
strength_and_weaknesses: Strengths:
- Experimental results show that the proposed model reduces the inference time and memory requirements.  
- Some limitations of the proposed model have been clearly mentioned. 

Weaknesses: 
- The application domain is not clear. Even though the paper mentions audio separation several times, it uses deep noise suppression challenge data which is mainly used for noise suppression. In addition, independent of the application, the paper does not mention the loss function and what the outputs from the model are. 

- The transposed version of the STMC model is not well-described. For example, in Fig. 2B, from where do we get the [0,1] into the buffer in the rightmost part of the figure? 

- Figures are not described in detail in the main text. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Some parts are not clear or confusing. For example, the application domain is not clear (denoising or speech separation), there is a mention of quantization in the appendix but not in the main text. In Table 3, are there any missing numbers or what do the centered numbers mean?  

Quality: Fair. Experiments, especially the causal convolution based ones show significant reduction in inference time but it comes with a 6% degradation of SI-SNRi. Is there a way to avoid this trade-off? 

Novelty: The model proposes to use a shift-register to cache the intermediate convolution outputs at intermediate layers to reduce the re-computation time.

Reproducibility: Some experimental details are provided, it should be reproducible at a high level but some details are missing. For example, whether there is model quantization, how the inference time processing is performed exactly: What is meant by ``Input data is processed in chunks of arbitrary size in an online mode.’’? 

After the rebuttal phase, the paper has improved and got clearer. Hence I am increasing my initial score to 6.

Other comments/questions: 
- What is the task? Is it denoising or speech separation? From the dataset (DNS), it sounds like the former but in several paragraphs it is told to be separation.   
- In Appendix A, the equation for $b_i$ does not match with the matrix shown above it. Should the filter be reversed in the matrix $[k_3, k_2, k_1, 0, 0]$? 
- Why are the inference time patterns different between Fig 4D and 4E?
- In Appendix B, there is a mention of quantization but the main paper does not mention anything related to this. Does the paper apply model quantization in the experiments of the main text? 
- In the last sentence of the conclusion, it is mentioned that the SI-SNRi degradation in the causal STMC version is attributed to the lack of future context. Could the training vs. inference time mismatch be another reason? 

summary_of_the_review: The model introduces the use a shift-register to cache the intermediate convolution outputs at intermediate layers to reduce the inference time of convolutional networks. The method is applied only at the inference time and not during training. Experimental results show significant speed up in inference time with a reduced memory requirement. However, the best performing version of the model also degrades the signal quality. There is a trade-off between time vs. quality. The paper can get confusing at times (e.g., what is the task?). Analysis at the appendix could have been added to the main discussion.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: I Summary:

- I.1 Investigated Problem:
The paper investigated the problem of latency minimization, memory consumption, and inference time reduction for real-time processing of time series signals as it is a critical issue for many real-world applications

- I.2 Proposed Solution:
The authors present Short-Term Memory Convolution (STMC) and its transposed (tSTMC) counterpart layers. The proposed solution relies on the buffering of the layer's outputs. The proposed layers introduce an additional cache memory to store past observed outcomes. The cache memory provides values calculated in the past to subsequent calls of the STMC layers in order to calculate the final output without any redundant computations. When processing a time series all calculations are performed exactly once

- I.3 Validity Proof of the Proposed Solution:
Through an application of the solution to a U-Net model for an audio separation task. Several variants of a U-Net architecture were compared, differing in causality and buffering types. 
 A 5-fold reduction of inference time was achieved compared to incremental inference, which is considered the state-of-the-art method for online processing using convolutional models. All the causal models used in the test achieved approximately a 6% reduction in SI-SNRi compared to their non-causal counterparts. The cause of the score reduction is attributed to the lack of future context and not the used caching method.

II Strengths:

- II.1 From a structural point of view:
 . The paper is well-structured and easy to understand. 
- II.2 From an analytical point of view:
. Fair experiments have been conducted to demonstrate the potential of the proposed method;
  .The Intellectual honesty demonstrated by the authors is appreciated by the reviewers as some limitations of the proposed model have been clearly mentioned;
 . Significant empirical evidence is provided to support the case of the proposed solution in terms of reduction of inference time and memory consumption;
- II.3 From a perspective of soundness (development, unity, and coherence) and completeness (correctness): 
 . Active interaction of the authors during the rebuttal period and their openness to concerns and questions raised by the reviewers allowed them to clarify certain points.

III Addressing what can be thought of as weaknesses

Before the rebuttal, some of the reviewers raised concerns about:
	1- the clarity of the proposed method:
		The authors updated the submission by editing the section related to the latency description.
	2- the nature of the task related to the conducted experiments:
		Whether it is about source separation or denoting. Authors explained during the rebuttal that the task of speech separation is equivalent to speech enhancement/denoising as secondary sources of signals can be considered as noise to the original source

IV. Potential of the paper:

- IV.1 From a Potential perspective (Potential of the paper to the community): The proposed solution has the potential to be of benefit to the whole community, especially the one interested in real-time processing of time series. The extension and investigation of other models can also be of great benefit to the machine learning community in general.

justification_for_why_not_higher_score: - Evaluation can be considered non-conventional. 
- Code is not provided. If provided it can add more value to the submission and its long-term impact.
justification_for_why_not_lower_score: Solid work with clear practical benefits. The presented work deserves to be accepted as a poster at the venue.
Decision: Accept: poster