CONDITIONING SEQUENCE-TO-SEQUENCE MODELS
Year: 2021

Abstract:
Conditional neural networks play an important role in a number of sequenceto-sequence modeling tasks, including personalized sound enhancement (PSE), speaker dependent automatic speech recognition (ASR), and generative modeling such as text-to-speech synthesis. In conditional neural networks, the output of a model is often influenced by a conditioning vector, in addition to the input. Common approaches of conditioning include input concatenation or modulation with the conditioning vector, which comes at a cost of increased model size. In this work, we introduce a novel approach of neural network conditioning by learning intermediate layer activations based on the conditioning vector. We systematically explore and show that learned activation functions can produce conditional models with comparable or better quality, while decreasing model sizes, thus making them ideal candidates for resource-efficient on-device deployment. As exemplary target use-cases we consider (i) the task of PSE as a pre-processing technique for improving telephony or pre-trained ASR performance under noise, and (ii) personalized ASR in single speaker scenarios. We find that conditioning via activation function learning is an effective modeling strategy, suggesting a broad applicability of the proposed technique across a number of application domains.

text: Conditional neural networks play an important role in a number of sequenceto-sequence modeling tasks, including personalized sound enhancement (PSE), speaker dependent automatic speech recognition (ASR), and generative modeling such as text-to-speech synthesis. In conditional neural networks, the output of a model is often influenced by a conditioning vector, in addition to the input. Common approaches of conditioning include input concatenation or modulation with the conditioning vector, which comes at a cost of increased model size. In this work, we introduce a novel approach of neural network conditioning by learning intermediate layer activations based on the conditioning vector. We systematically explore and show that learned activation functions can produce conditional models with comparable or better quality, while decreasing model sizes, thus making them ideal candidates for resource-efficient on-device deployment. As exemplary target use-cases we consider (i) the task of PSE as a pre-processing technique for improving telephony or pre-trained ASR performance under noise, and (ii) personalized ASR in single speaker scenarios. We find that conditioning via activation function learning is an effective modeling strategy, suggesting a broad applicability of the proposed technique across a number of application domains.

heading: 1 INTRODUCTION
text: The use of latent or embedding vectors to condition neural networks is common in a number of machine learning tasks. For example, in personalized sound enhancement (PSE) prior knowledge of a speaker’s voice profile is used to filter out noise from an audio recording (Choi et al., 2021; Hu et al., 2020; Wang et al., 2020; 2019), in speaker-dependent or personalized Automatic Speech Recognition (ASR) a speaker’s voice profile is also used to improve the overall transcription quality (Bell et al., 2021; He et al., 2019; Samarakoon et al., 2016; Saon et al., 2013), in personalized Text-To-Speech (TTS) systems synthesized voice profile can be purposefully adapted (van den Oord et al., 2018), in computer vision tasks, such as in visual reasoning applications (Chen et al., 2019; Perez et al., 2018), questions are used as the conditioning vectors, and similarly in conditional domain adaptation techniques (Long et al., 2018). More generally, conditional neural networks accept an additional conditioning vector, that influences how the networks process inputs to produce outputs.
Most common approaches for conditioning neural networks either concatenate or modulate the conditioning vector with one or more intermediate representations of the network, before the updated representations are processed by a layer, e.g., see Figure 1a-b for an overview. Such conditioning techniques come at the cost of significantly increasing the network parameters by inflating the dimensionality of intermediate features or by requiring additional network parameters proportional to the dimensionality of the conditioning vectors and the number of channels present in the representations.
In this work, we introduce a new approach for neural network conditioning and show that effective conditioning can be achieved by learning layer-specific activation functions based on the conditioning vectors. Unlike previous approaches, our learned activation (LA) functions approach does not increase the model parameters significantly, since the number of additional parameters it requires depends only on the size of the conditioning vector, and not on the shape of the feature representations nor on the output shapes of any subsequent layers. LA uses the conditioning vectors to dynamically modify intermediate activation functions, which are R→ R (1-D) mappings, see Figure 1c. Dynamic LAs allow us to extract important insights on how the model is being adapted for different conditioning vectors by simply analyzing the learned activation functions, which are agnostic to the input.
In this work we consider two popular and timely sequence-to-sequence modeling tasks: personalized sound enhancement (PSE) and personalized ASR (PASR) as exemplary tasks for modeling conditional neural networks. PSE leverages a speaker embedding vector, i.e., a unique encoded voice profile of a speaker, for filtering out everything but the speaker’s voice from an audio recording corrupted with babble or ambient noise (Wang et al., 2020; 2019). Speaker embeddings can generally be obtained by using an embedding network once during an enrolment phase, e.g., from recorded utterances like “Hey Alexa” or “Hey Bixby”. PSE networks can be evaluated for improving signal quality for telephony or can also be used to improve the quality of downstream tasks, such as a pre-trained ASR. PASR on the other hand, aims to improve ASR quality in single speaker scenarios, by leveraging user’s enrollment data to enhance performance directly without any pre-processing.
We evaluate LA on three PSE model families (§4.2), on two datasets Librispeech (Panayotov et al., 2015) and Voxforge (vox, 2006). We show that the use of LA on all models achieves competitive performance to conditioning using concatenation and modulation approaches. At the same time, concatenation approaches require 10–95% more model parameters over our solution. For PASR, we compare the performance of non-personalized and personalized versions of the conformer encoder model architecture (Gulati et al., 2020) with a dense layer for character or subword text encoder trained with CTC loss and decoded via CTC decoder (§5.2). Results on Librispeech (Panayotov et al., 2015) show that LA yields effective ASR personalization, i.e., increased performance across all users while using two seconds of enrollment data, with up to 19% relative WER improvement across all users, and up to 40% for some users.

heading: 2 RELATED WORK
text: Conditioning through tensor concatenation The concatenation of intermediate features with latent vectors before being passed to subsequent layers is the most common form of conditioning, but leads to inflated model sizes, which hinder their deployment to low-resource constrained devices (Wang et al., 2019; 2020; Valin et al., 2020; Hu et al., 2020; Comon & Jutten, 2010; Wang et al., 2021; Lam et al., 2021; Vincent et al., 2018; Luo & Mesgarani, 2019; Xu et al., 2020; Abdulatif et al., 2020; Wisdom et al., 2020; Isik et al., 2020; Luo et al., 2020; Choi et al., 2021; Snyder et al., 2018; Panayotov et al., 2015; Bell et al., 2021; He et al., 2019; van den Oord et al., 2018). Our approach overcomes this issue and learns smaller models with competitive quality. Conditioning via normalization layers Another family of techniques is based on having per channel learned linear transformation based on the conditioning vector, which yields a form of conditional normalization (Perez et al., 2018; de Vries et al., 2017; Huang & Belongie, 2017). Our approach distinguishes itself from this line of work since it learns the non-linear components of the network rather than the linear ones, and thus offers greater modelling at lower cost. Learning activations without conditioning Outside the context of conditional neural networks, there is work on learning an optimal activation function as a good default to be used in all layers of most neural network architectures (Apicella et al., 2021; Liu et al., 2020; Ramachandran et al., 2018; Agostinelli et al., 2015). Our work instead i) learns conditional activations based on conditioning vectors, and ii) uses differently learnt activations throughout the network.

heading: 3 CONDITIONING BY LEARNING ACTIVATIONS
text: Conditioning a neural network corresponds to using some form of latent vector or additional prior knowledge, e.g., encoded typically in the form of an embedding vector, to influence the output of the network for a given input. We show in the evaluation that the influence of the conditioning vector can be successfully modeled by learning a weighted combination of a set of basic activation functions (§3.1) and carefully selecting hyperparameters (§3.2).

heading: 3.1 DYNAMICALLY LEARNED ACTIVATIONS
text: To introduce our conditioning approach using dynamically weighted activation functions, we begin by defining a few key variables: z ∈ Rb×d is a 2-D tensor representing b number of d-dimensional conditioning vectors, where b is the batch size, x ∈ Rb×··· is the input data with arbitrary tensor dimensions (e.g., vectors, matrices, higher order tensors), h ∈ Rb×··· is the hidden pre-activation of the previous layer, and {Ai : R → R}ai=1 is an ordered family of a basic activations, e.g., {relu, sigmoid, tanh} (we will discuss below more on the elements of this ordered set of activations).
Dynamically learned activation (LA) functions require a small number of trainable variables w ∈ Rd×a and b ∈ R1×a, and are defined by transforming input tuples (z,h) into output tensors y through (note the parallel between the following equations and the pictorial representation in Figure 1c, where there is a direct correspondence on the roles of z, h and y between the formulas and the diagram):
s := softmaxrowwise(z ×w + b),
h ∈ R 7→ LA(h | zj) := a∑
i=1
sj,i ·Ai(h),
yj,... := LAelementwise(hj,... | zj),
where softmaxrowwise denotes the matrix-matrix operation with entries defined rowwise via softmaxrowwise(m)j := softmax(mj), and LAelementwise(hj,... | zj) denotes the result of applying LA(· |zj) elementwise to each entry of hj,.... Note that during training, batch size is greater than one in which case z is a matrix, whereas during inference batch size is one for each user in which case z is a row vector. So during training zj is the conditioning vector corresponding to the j-th sample, but during inference zj is the conditioning vector corresponding to the j-th user in the discussion below.
Figure 1 highlights the qualitative and quantitative differences between our approach (Figure 1c) and existing state-of-the-art (SOTA) solutions (Figure 1a-b). Concatenation approaches as depicted in Fig. 1a can be defined by Aelementwise(Fθ([z>j ,x > j,...,:]
>)), where Fθ denotes a common learnable transformation such as a dense, conv1d, conv2d or LSTM layer without activation and A : R→ R denotes a generic activation such as the rectified linear unit. Whereas, as shown in Fig. 1b, modulation approaches are defined as Aelementwise(Gη(Uµ(zj) xj,...,: + Vν(zj))), where Gη represents a common learnable transformation (which may be slightly different than Fθ given their inputs have different shapes) and Uµ : Rd → Rf and Vν : Rd → Rf are often taken as linear mappings (where d denotes the conditioning vectors dimension and f denotes the number of channels of x).
In comparison to these SOTA approaches, our learned activation functions mechanism can simply be formulated as LAelementwise(Gη(xj,...) | zj). For simplicity, we use a dense layer in the figure (i.e., we take Fθ and Gη to be dense layers and f = 1), however, our approach is general and can be applied to any layer type, e.g, dense, conv1d, conv2d and LSTM. From a qualitative point of view, LAs conditions the non-linearities rather than the linearities of the network. From a quantitative point of view, Figure 1 also highlights the superior compression abilities achieved by our proposed method.
In particular, concatenation approaches (Figure 1a) require (d+ u) ∗ v parameters and modulation approaches (Figure 1b) require u ∗ v + d ∗ 2 ∗ f , where f denotes the number of channels in the input tensor x, which per channel is multiplied by a scaling factor and offset by a bias term. The learned activation functions approach on the other hand, requires only u ∗ v + d ∗ a parameters. For example, in most audio and video applications, our approach yields significant savings of d ∗ (v − a) given that d 1 and v a, when compared to concatenation approaches. The saving becomes d∗ (2∗f −a) when compared to modulation approaches, given that often in such cases 2∗f −a > 0. For temporal data, such as audio and video, the savings of our approach are even more significant as in those applications the conditioning vector is concatenated with each time frame of the inputs and
for f 1 the model requires larger parameter size. In our approach the operational requirements do not change across data type, i.e., tensor dimensions.

heading: 3.2 HYPER-PARAMETERS
text: Basic Activations As discussed above, our conditioning approach relies on a set of standard activations provided for modelling. Therefore, the selection of these basic activations is an important hyper-parameter in our approach. Following are the key characteristics that should be taken into account when searching for an optimal set of activations:
• Dynamically learned activation coefficients sj are all positive and sum to one. Thus, some families of activations will generate learned activations that remain in the same family, however, this not always true. For example, if all basic activations are all increasing functions, so is the learned activation. Monotonic increasing property of the learned activation may not be retained if we include both increasing and decreasing functions in the basic activation set.
• The fact that sj are in the probability simplex means that the ranges of the basic activations play an important role in the final learned activations. One could consider basic activations with i) unbounded, ii) upper bounded, iii) lower bounded or iv) bounded ranges.
• Finally, the basic activations could instead be picked from well known basis functions. For instance, Fourier basis or Chebyshev polynomials could be used to create bounded learned activations with compact support, and Hermite polynomials to create unbounded learned activations on the real line. Note that this direction of investigation is outside the scope of our paper.
Initialization Choices Another important hyper-parameter to consider is what are the values of the dynamically learned coefficients sj at initialization time for most conditioning vectors zj . Given sj is a probability distribution, to start training with an unbiased model, they could be at initialization time approximately uniform, i.e., providing roughly the same weight to each of the basic activations for most zj . Alternatively, they could be such that for most zj at the beginning of training the learned activations have a desired shape. Therefore, trainable variables w and b should be initialized considering i) the statistics of the conditioning vectors zj , and ii) the shapes of the basic activations.

heading: 4 EXPERIMENTS PART I: PSE FOR TELEPHONY AND PRE-TRAINED ASR
text: In this section we consider the task of PSE, with the aim of enhancing a user’s voice to improve telephony and pre-trained ASR quality under babble (i.e., other speaker’s voices) or ambient noise, in real-world scenarios with only two seconds of enrolment audio of non-constrained speech.

heading: 4.1 DATASETS
text: We consider two representative datasets: LibriSpeech (Panayotov et al., 2015) and Voxforge (vox, 2006), which cover English and Spanish respectively. For the English training set, we take 100h and 360h of clean speech from LibriSpeech, which are recorded using close-talk microphones, without any background noise. Although, supervised learning approaches have shown great performance in sound enhancement tasks, it is often impossible to collect the mixed audio and its ground-truth sound sources from the same audio environment (Wisdom et al., 2020). Furthermore, to evaluate performances of learned activation on realistic data, we create a training dataset by taking 32.5h of Spanish audio from Voxforge, which are mainly recorded with a far-field microphone and contain noise. Both datasets have audio recorded at 16kHz.
The raw datasets consist of tuples (speech, transcript, speaker identifier). For each pair of different speakers a and b, we collect two speech samples from a and one from b to create 4-tuples (speecha, speech ′ a + speechb, speech ′ a, text ′ a), with elements being: i) a reference speech from user a to be passed through a speaker embedding model (Snyder et al., 2018) to create the conditioning vector, ii) a speech from user a, different from that given to the embedding model, which is corrupted (at various db levels) with speech from user b, iii) the ground-truth speech by user a present in the corrupted speech, and, iv) a transcript of the ground-truth speech by user a. All PSE models are trained in a supervised fashion, where the goal is to recover speech′a from speech ′ a + speechb as closely as possible for telephony applications and such that ASR(PSE(speech′a+speechb)) = text ′ a for ASR applications. We also evaluate performances under ambient acoustic noise by constructing (speecha, speech ′ a + noise, speech ′ a, text ′ a) tuples, where noise is sampled from the DEMAND dataset (Thiemann et al., 2013).

heading: 4.2 BASELINE MODELS
text: A variety of architectures have been proposed lately for speech enhancement. In this work, we consider architectures suitable for low-latency on-device deployment that have small memory footprints and operate in real-time. With that in mind, we use the following three architectures as baselines:
• RNN. These are RNN models with skip-connections (Takeuchi et al., 2020; Wang et al., 2020) (c.f., Figure 5a in Appendix A).
• TDS. These models (Figure 5b in Appendix A) comprise of stacked time-depth-separable (TDS) convolution blocks (Isik et al., 2020; Luo & Mesgarani, 2019).
• TDS-RNN. These are encoder-decoder style models with a separator block in between (Figure 5c in Appendix A). Encoder and decoder are made of stacked TDS blocks and the separator of stacked RNN blocks (Choi et al., 2021).
Furthermore, we consider a different number of conditioning layers (#Cond.) for each model family. In particular, for RNN 8, for TDS 21 and for TDS–RNN 2 conditioning layers leading to 8, 42 and 2 nodes in the models being influenced by the conditioning information (c.f., Figure 5 in Appendix A). We have done so to investigate also how the number of conditioning layers influences the personalization effects from the conditioning vectors (c.f., Appendix A.2).
In our experiments we apply LAs in all three families of models and perform extensive comparisons with the baselines in terms of performance, model size and latency. In particular, we take the best performing baseline concatenation architectures, not the weights, and for every layer with a conditioning concatenation, we replace that with a modulation layer or a conditionally learned activation. Finally we train the modulation models and the conditionally learned activation models from scratch, without performing any architectural hyper-parameter optimization (HPO). As a representative of light weight-modulation approaches, we consider the Feature-wise Linear Modulation (FiLM) (Perez et al., 2018) mechanism throughout. Notably, our evaluation experiments give more advantages to the baseline concatenation models.

heading: 4.3 HYPER-PARAMETERS
text: Basic Activations The choice of basic activation functions is an important hyper-parameter in our approach and we consider a comprehensive list of popular activation functions used in deep learning. Specifically, we use the following ordered set as the basic activation functions: {Ai : R→ R}ai=1 = {elu, exponential, hard_sigmoid, linear, relu, selu, sigmoid, softplus, softsign, swish, tanh}. Initialization Choices We consider enrolment data (e.g., a short pre-defined utterance spoken by the user once) to extract (using a `2-normalized X-Vector model (Snyder et al., 2018)) a speech embedding vector of the target user, i.e., the owner of the device. This speech embedding is used as the conditioning vector in PSE models to enhance the user’s voice and suppress unwanted voices and background noise. Although, the embedding vectors in the training set are not perfectly zero mean and unit variance, the fact that they are `2-normalized motivates us to take for initialization of the weight matrix in the LAs the Glorot uniform initializer and for the bias vector the zero initializer.

heading: 4.4 TRAINING SETTINGS
text: All models considered in this work operate in the time-frequency domain. We apply Short Time Fourier Transform (STFT) to extract 512 coefficients using a 32 ms window and with a stride of 16 ms. Models are trained in a data-parallel way with four GPUs using TensorFlow and Horovod. We use batch size of 64, learning rates in [1e−5, 1e−3], exponential decay learning rate scheduler and early stopping (max epochs 100).

heading: 4.5 EVALUATION METRICS
text: We evaluate our models in terms of source-to-distortion ratio improvement (SDRi) (Comon & Jutten, 2010) and word-error-rate (WER) metrics. A high SDRi indicates the improvement in signal quality for telephony and a low WER quantifies good transcription quality of the downstream ASR task. We use open-source Silero (sil, 2021) ASR models (English and Spanish) in our evaluations.

heading: 4.6 RESULTS
text: This subsection presents evaluation results for quantifying the effectiveness of learned activations (LA), compared to concatenation- and modulation-based conditioning approaches, on LibriSpeech (Table 1) and Voxforge (Table 2) datasets. The results show that models with all three conditioning approaches improve telephony (high SDRi) and ASR (low WER) performances on both datasets. We observe that LA and concatenation-based conditioning approaches yield very similar performances, indicating high efficacy of the proposed LA approach. Note that when evaluating ASR, it is also important that a good PSE model shows almost no degradation in WER, when clean audio is presented. Interestingly in some cases in the concatenation-based approach, we see that higher success in removing babble noise, comes with an adverse impact in clean speech WER, mainly due to over-suppression (Wang et al., 2020). Furthermore, Table 1–2 show that both concatenation and modulations produce significantly larger models, i.e., up to 95% larger, compared to LA approaches.
Tables 1–2 highlight the trade-off between the number of parameters and quality metrics. However, it is important also to consider the impact of the different approaches in terms of FLOPS and latency,
as shown in Table 3 (where latency was measured by passing one second of data one hundred times by the models and summarized via mean and standard deviation), which demonstrates LAs are competitive in this dimension. Specifically, LA is faster than standard concatenation approach with comparable quality, and slower than FiLM but with higher quality (i.e., higher SDRi and lower WER).
To gain a better understanding of how LAs adapt based on conditioning vectors, Figure 2 presents plots centered at the origin of the LAs for a PSE application conditioned on speaker embedding vectors extracted from enrolment audio from randomly selected users. In particular, we can see that for all models i) in each layer, different users have different LAs, and ii) each user has different LAs across different layers. As such the visualization of the internal mechanisms of LAs, provides a tangible way to improve how we understand the manner in which models leverage conditioning vectors to influence predictions.

heading: 5 EXPERIMENTS PART II: PERSONALIZED ASR
text: In this section we consider the problem of personalized ASR, with the objective of improving ASR performance in single speaker scenarios. This is a challenging task, especially in realistic setups with only a couple of seconds of enrolment audio with non-constrained speech which we explore below.

heading: 5.1 DATASETS
text: We consider LibriSpeech (Panayotov et al., 2015) 1000h of English annotated audios with transcripts. We augment each sample tuple (audio, transcript) to include two seconds enrollment audio from the same user (audio, transcript, enrollment). We ensure the enrollment audio is taken from a different sample from the same user. For diversity during training the enrollment audio is selected randomly under the aforementioned constraint, whereas for evaluation it is always the same.

heading: 5.2 BASELINE MODELS
text: ASR. For non-personalized ASR models we take the conformer encoder model architecture (Gulati et al., 2020) with a dense layer for character (with 28 characters, namely: ‘A’–‘Z’, ‘ ’ and ‘'’) or subword (with 753 subwords) text encoder trained with CTC loss and decoded via CTC decoder. These models take as input an audio and output a predicted transcript. PASR. For personalized ASR we take the ASR architecture but replace the ReLU and Swish activations with our proposed LAs (see §3.1). We consider basic activations (see §3.2) containing
ReLU and Swish, and an initialization of the bias (see, §3.2) so that at initialization time, the LAs replacing ReLU are approximately ReLU, and the LAs replacing Swish are approximately Swish, a profile which changes during training as a mechanism to enable personalization. As a similar mechanism to the bias offset in LAs, we altered FiLM’s definition Aelementwise(Gη(Uµ(zj) xj,...,: + Vν(zj))) to Aelementwise(Gη((1 +α · tanh(Uµ(zj))) xj,...,: +β · tanh(Vν(zj)))) with α and β as trainable variables, so that at initialization FiLM is approximately the identity function for most conditioning vectors. As with non-personalized ASR, we consider character and subword text encoders trained with CTC loss and decoded by CTC decoder. These models take as input an audio and a fixed speaker embedding vector for each user computed from enrollment data by a speaker embedding network (Snyder et al., 2018), and output a predicted transcript.

heading: 5.3 HYPER-PARAMETERS
text: For consistency with previous experiments, we take as the basic activations those used in §4.3. For initialization choices as mentioned earlier we introduce a bias offset so that at initialization the softmax s described in §3.1 is approximately one-hot to leverage the inductive bias of activations proven to work well in non-personalized settings. Interestingly, we observed in our experiments that subword models seem to be more sensitive to this hyper-parameter than character models.

heading: 5.4 EVALUATION METRICS
text: For character and subword models we consider greedy (beam search with tops paths and beam width equal to one) and beam search (with a beam width of four) CTC decoders. Once predicted transcripts are produced from the models, they are compared with ground-truth transcripts in terms of word error rate (WER) and character error rate (CER).

heading: 5.5 RESULTS
text: We consider the performance of PASR models created from existing ASR models by replacing their fixed activations with LAs. To delve deeper into the effect of LAs on PASR, we consider two dimensions to our experiments, namely, in terms of whether i) training is performed only on the speaker embedding model as well as on the weights and biases of LA, or ii) training is done on all parameters. In both cases, optimization is done with respect to lower CTC loss. These settings are designed on the one hand to understand what are the effects of taking pre-trained ASR models and personalizing them by replacing their activations with LAs, and on the other hand to understand the capacity and convergence properties that LAs provide for PASR. Before a fine-grain analysis of the results below, note that similarly to Figure 2 for PSE, Figure 3 shows that LAs consistently create different activation profiles across users and layers for PASR as well.

heading: 5.5.1 TRAIN SELECTED PARAMETERS
text: Leaving the parameters of the original ASR intact, allows for switching between the original static activations and the LAs, which might be beneficial for deployment scenarios lacking clean enrolment data. We investigate this scenario by fine-tuning only the speaker embedding model and LA parameters, which results in an overall 2% WER relative improvement which is a modest but significant improvement given an already low WER/CER of the original ASR. Figure 4 (top) shows that for some users this improvement is even more significant, as high as 15% WER relative improvement.

heading: 5.5.2 TRAIN ALL PARAMETERS
text: In this section we investigate the performance of PASR models derived from training all their parameters after initializing their non-personalized components from existing ASR models. Results for all users together Table 4 contains WER/CER results aggregated across all users of LibriSpeech test-clean. Overall the results show a consistent improvement in performance, i.e., lower WER/CER for personalized ASR via LAs compared with non-personalized ASR. The results also show that subword models yield better WER, but character models give better CER, with the gains through personalization being higher for character than subword models, being as high as 19% relative WER improvement across all users. Results for each user separately Figure 4 (bottom) shows plots of WER relative (in terms of percentage) and absolute improvement per each user of LibriSpeech test-clean for character models with CTC greedy and beam search decoders. Each plot shows three curves, namely: i) the horizontal blue line y = 0 representing the performance of the non-personalized ASR, ii) the orange curve representing the performance of the non-personalized ASR with its activations replaced by randomly initialized LAs, and iii) the green curve representing the performance of this personalized ASR fine-tuning speaker embedding model, weights and biases of LAs, as well as the parameters of the
non-personalized ASR. The results show an initial mixed degradation/improvement per user when replacing the original activations with LAs as expected without training (ensured by an appropriate choice of the bias offset hyper-parameter as described in §5.3), and the benefits of training the personalized ASR, with significant gains in WER per user, in terms of relative and absolute WER. Furthermore, the gains can be seen to reach up to 40% and 35% relative improvement for some users for CTC greedy decoder and CTC beam search decoder respectively, which is a significant improvement in the domain of ASR.

heading: 6 LIMITATIONS
text: Our evaluations are limited to two supervised audio domains, namely: i) PSE to improve telephony and pre-trained ASR quality for English and Spanish, under babble and ambient noise, and ii) personalized ASR for single speaker scenarios over English, based only on two seconds of enrollment data. We nevertheless see value in assessing our approach in generative or image domains. Another limitation stems from the fact that the evaluation is performed with speaker embeddings computed from clean enrolment data, which might not be available at deployment time. So the robustness to this source of issues is not explored in our experimental setting. This limitation is common in most, if not all studies, in these application domains.

heading: 7 CONCLUSIONS
text: We introduced a new general method for designing conditional neural networks. Unlike state-of-theart techniques, which are based on concatenation or modulation strategies, we presented a new form of conditioning by dynamically learning activations. By conducting an extensive set of experiments, we highlighted the efficacy of our modeling technique in terms of quality metrics and model sizes. Although, we mainly focus on PSE under babble and ambient scenarios and personalized ASR for single speaker cases as the main application domains, we believe our approach will open ways for the generalized use of dynamically learned activation functions in many application domains.

Review summaries:
This paper proposes a way of conditioning information on neural networks. In literature a common way to condition a neural with an input would be to either concatenate the conditioning vector to the input vector, or inject it before several layers (modulation approach in Figure 1-b). In this paper they instead propose to pass the conditioning vector through weighted sum of output of neural-net activation functions. The claim is that this way of training the neural network leads to reduction in neural network parameters without sacrificing performance in speech enhancement / ASR tasks.

This paper presents a very interesting idea (if I understand it correctly), which is to implement conditioning on e.g. a speaker embedding vector via a parametrization of the layer-to-layer activation functions of an overall deep neural net architecture. The motivation is to achieve the same effect as the well-known concatenation approach, but without requiring as many new parameters.  I found the formalism hard to follow, impeding my grasp of the fundamental proposal. However, eventually I got the idea. The results are promising, showing good preservation of e.g. WER quality on the unconditioned scenario, with good WER on the specialized scenario, and with fewer parameters than the concatenation method.  I think the name "learned activations" somewhat undersells the idea -- wouldn't it be better named as "learned activation _functions_"? That immediately would clarify the fundamental concept -- and highlight it's originality.

This paper introduces a method called learned activations for personalized speech enhancement and personalized automatic speech recognition. The learned activations are obtained by a weighted sum of nonlinear activations of hidden layer outputs and the weights are computed from softmax of the projected speaker embeddings such as x-vectors. The main benefit of the model as compared to concatenation or modulation based approaches is that Learned Activations result in smaller number of learnable parameters and hence a smaller model size. Experimental results show that in terms of SDR improvement, the proposed method achieves comparable performance to the baseline models with a smaller model size. In the case of ASR, the proposed approach provides WER reduction as compared to an unadapted model.