Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a very interesting idea (if I understand it correctly), which is to implement conditioning on e.g. a speaker embedding vector via a parametrization of the layer-to-layer activation functions of an overall deep neural net architecture. The motivation is to achieve the same effect as the well-known concatenation approach, but without requiring as many new parameters.

I found the formalism hard to follow, impeding my grasp of the fundamental proposal. However, eventually I got the idea. The results are promising, showing good preservation of e.g. WER quality on the unconditioned scenario, with good WER on the specialized scenario, and with fewer parameters than the concatenation method.

I think the name "learned activations" somewhat undersells the idea -- wouldn't it be better named as "learned activation _functions_"? That immediately would clarify the fundamental concept -- and highlight it's originality.
main_review: See my comments above in the Summary for strengths and weaknesses. In an nutshell, I think the paper suffers from a lack of clarity that undersells what is otherwise a nice idea.

I suggest introducing the specifics of the core of the idea earlier on in the paper. There's a sense of anticipation that is a bit frustrating for me as a reader.

Some more specific comments follow.

Re: sentence ending in, "... and a family of `a` basic activations {Ai : R → R} a i=1 of which a particular realization could be the set of the most commonly used activations in deep learning (e.g. relu, sigmoid, tanh, and so on)": check the grammar, I believe this is not a complete sentence given what comes before the sub-clause I just quoted.

What is "softmax_rowwise"?

Re: First 3 equations in Section 3.1: What is the relationship between LA_elementwize(c | z_j) and LA(h_j | z_j)? (Numbering the equations would be helpful). What is c?

I did not quite follow the formalism. I am unable to clearly relate the equations in 3.1 with Figure 1c.

A number of references are given for the concatenation approach, which is good. But no references are given for the modulation approach (AFAICT), though this is referred to as a state-of-the-art approach. Here again, I am unable to follow the discussion linking the formalism to Figure 1b. I am guessing that the formalism is correct but the presentation could be substantially simplified and clarified.

Figures 2: "To take into account that basic activations may have different ranges, the plots show LAs minus average of basic activations, i.e., LA(c | zj )  [...] for c ∈ [−3, 3] and values of j corresponding to the selected users." I cannot relate LA(c | zj) to the label of the x-axis, which is h.

Figure  4. "To take into account that the non-personalized basic activations are relu or swish, the plots show LAs minus average of those two activations, i.e., LA(c | zj ) − (relu(z) + swish(z))/2 for c ∈ [−3, 3] and values of j corresponding to the selected users. " I suggest labeling the x-axis. Would that be z? How should we relate z_j to the z (without the j subscript) argument of relu() and swish()?

Section 4.1, Table 1: Define SDRi before using the term.

Section 4.2, "HPO": "Hyper-parameter optimization"?

Section 4.4, "STFT": define the term.

Section 5.4, "Greedy search": provide a reference? Later, in the results section, the authors state, "Furthermore, the gains can be seen to be more pronounced for greedy than beam search CTC decoders, reaching up to 40% relative improvement for some users which is a significant improvement in the domain of ASR," but greedy search doesn't represent the typical final ASR benchmark, so the greater relative gain than for beam search doesn't seem too significant -- but I am not sure what the authors mean by greedy search, so they should clarify this.
summary_of_the_review: Good originality (AFAICT) and good results; somewhat unclear formalism and presentation of main concepts.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a way of conditioning information on neural networks. In literature a common way to condition a neural with an input would be to either concatenate the conditioning vector to the input vector, or inject it before several layers (modulation approach in Figure 1-b). In this paper they instead propose to pass the conditioning vector through weighted sum of output of neural-net activation functions. The claim is that this way of training the neural network leads to reduction in neural network parameters without sacrificing performance in speech enhancement / ASR tasks.
main_review: The experimental results indicate that the proposed method of training the RNN / CNN based systems for speech enhancement and ASR leads to reduction in number of parameters. The authors argue that this is an important advantage for low-resource computing. I have several questions on this. I have reviewed this paper before, and I see that some of my concerns from earlier, even though were partially adressed in the rebuttal, now seems not to be addressed in the current version of the manuscript.

The number of parameters to assess the computational advantages of a model is questionable in my opinion. I think a better measure for this would be to use the number of FLOPs in forward pass, or the memory usage with respect to input sequence length. Namely, this table was presented in the rebuttal: 

Model 	LA 	FLOPs (M) 	Latency (ms)

RNN 	y 	1254.85 	528.543 (+-36.392)

RNN 	n 	1802.60 	537.766 (+-11.403)

TDS 	y 	1507.43 	116.451 (+-06.992)

TDS 	n 	2906.46 	123.580 (+-03.440)

TDS--RNN 	y 	4406.60 	392.326 (+-15.582)

TDS--RNN 	n 	4543.92 	394.809 (+-20.247)

I think it would be good to include these results. Overall, we see that the improvement is not always clear, especially in terms of latency, when the variance is taken into consideration. 

For the new experiment with ASR, and PASR, (part 2 of the experiments) the presentation is not super clear to me, and I am not sure if an improvement over alternative conditioning techniques are shown here. 
summary_of_the_review: I was a reviewer for this paper for Neurips 2021. Even though I voted for acceptance that time, I now see that some of my concerns and concerns pointed out by other reviewers are not fully addressed. For instance I do not see a comparison with this paper: https://arxiv.org/pdf/1601.02828.pdf  . I also do not see a comparison in terms of # FLOPS and latency for which the improvement seems marginal from the results added to the Neurips rebuttal. 

I therefore vote for marginal rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a method called learned activations for personalized speech enhancement and personalized automatic speech recognition. The learned activations are obtained by a weighted sum of nonlinear activations of hidden layer outputs and the weights are computed from softmax of the projected speaker embeddings such as x-vectors. The main benefit of the model as compared to concatenation or modulation based approaches is that Learned Activations result in smaller number of learnable parameters and hence a smaller model size. Experimental results show that in terms of SDR improvement, the proposed method achieves comparable performance to the baseline models with a smaller model size. In the case of ASR, the proposed approach provides WER reduction as compared to an unadapted model. 

main_review: Strengths:
- Extensive experiments on two application areas (speech enhancement and automatic speech recognition). 
- The paper also tried to provide quantitative results by plotting learned activations for different speakers. 
- The proposed model is comparably smaller than the concatenation based approaches. The method might be suitable for on-device applications.  

Weaknesses:
- As there are many experimental results, the paper may occasionally become harder to read towards the end.
- Some details are not well-described. For example, in Tables 1 and 2, what does #Cond. mean?  
- Even though the model size is smaller, it comes with additional on-the-fly computation of several non-linearities of hidden activations, which may cause other concerns for on-device applications. 
- FiLM approach should be briefly mentioned in the text before reporting the results with this method. 
- HPO acronym should be defined. (Page 6)
- How exactly does the study perform fine-tuning of the speaker embedding model? Details are missing. (Page 8) 
- For PASR, does the study fine-tune the speaker embedding model jointly with the ASR model or are they learned separately?
summary_of_the_review: The paper provides a personalization method by conditioning non-linear activations based on the speaker embeddings. There are extensive experiments. One concern is that the paper may need some additional clarification on details for better reproducibility.
review_title: null

Meta-review: The authors propose a novel method for conditioning deep neural network. They replace the activation function with a linear combination of activation functions (e.g., ReLu). The weights for the activation functions are dynamically computed from the input during inference and training. The approach is evaluated on standard public tasks and shows improvement over well-established alternatives.

Pros
+ A simple novel method for condition that is widely applicable
+ Adequate empirical evaluations to demonstrate it's effectiveness

Cons
- No major weakness

The reviewers provided several feedback. The authors incorporated the suggestions and clarified residual concerns. The revised version of the paper has improved the readability and utility substantially.
Decision: Accept (Poster)