Conditioning Sequence-to-sequence Networks with Learned Activations
Authors: Alberto Gil Couto Pimentel Ramos
Abhinav Mehrotra
Nicholas Donald Lane
Sourav Bhattacharya
Venue: ICLR 2022 Poster (2022)
Keywords: Conditional Neural Networks
Sound Enhancement
Personalized ASR

Abstract: Conditional neural networks play an important role in a number of sequence-to-sequence modeling tasks, including personalized sound enhancement (PSE), speaker dependent automatic speech recognition (ASR), and generative modeling such as text-to-speech synthesis. In conditional neural networks, the output of a model is often influenced by a conditioning vector, in addition to the input. Common approaches of conditioning include input concatenation or modulation with the conditioning vector, which comes at a cost of increased model size. In this work, we introduce a novel approach of neural network conditioning by learning intermediate layer activations based on the conditioning vector. We systematically explore and show that learned activation functions can produce conditional models with comparable or better quality, while decreasing model sizes, thus making them ideal candidates for resource-efficient on-device deployment. As exemplary target use-cases we consider (i) the task of PSE as a pre-processing technique for improving telephony or pre-trained ASR performance under noise, and (ii) personalized ASR in single speaker scenarios. We find that conditioning via activation function learning is an effective modeling strategy, suggesting a broad applicability of the proposed technique across a number of application domains.

Published as a conference paper at ICLR 2022
CONDITIONING SEQUENGE-TO-SEQUENNC MODELS WITH LEARNED ACTIVATION FUNCTIONS
Alberto Gil C. P. AAbiiav Mehrotra Nicholls D. Lane '2, Sourav Bhattacharyal Samsung Al Centre, Cambridge, UK ?University of Cambridge, UK fa. gilramos, wwhrotrall, nic. lane, sourav. bo http::ll)ssssssssss amsauug... com
Abhinay
ABSTRACT
Conditional neural networks play an important role in a number of sequence- to-sequence modeling tasks, including personalized sound enhancement (PSE), speaker dependent automatic speech recognition (ASR), and generative modeling such text-to-speech synthesis. In conditional neural networks, the output of a model is often influenced by a conditioning vector, addition to the input. Common approaches of conditioning include input concatenation modulation with the conditioning vector, which comes at a cost of increased model size. In this work, we introduce novel approach of neural network conditioning by learning intermediate layer activations based on the conditioning vector. We systematically explore and show that learned activation functions can produce conditional models with comparable or better quality, while decreasing model sizes, thus making them ideal candidates for resource-efficient on-device deployment. As exemplary target use-cases we consider (i) the task of PSE as pre-processing technique for improving telephony or pre-trained ASR performance under noise, and (ii) personalized ASR in single speaker scenarios. We find that conditioning via activation function learning is an effective modeling strategy. suggesting a broad applicability of the proposed technique across number of application domains.
INTRODUCTION
The use of latent embedding vectors to condition neural networks is common in number of machine learning tasks. For example, personalized sound enhancement (PSE) prior knowledge of a speaker's voice profile is used filter out noise from audio recording (Choi al. 202 Hu et al.,
2020 Wang et al. 2020; 2019), in speaker-dependent or personalized Automatic Speech Recognition
(ASR) speaker's voice profile is also used to improve the overall transcription quality (Bell et al.
2021 He et al. 2019 Samarakoon et al. 2016 [Son et al., 2013), in personalized Text- To-seTo-Speech
(TTS) systems synthesized voice profile can be purposefully adapted (van den Oord al. 2018))
in computer vision tasks, such as in visual reasoning applications (Chen et al.) 2019 Perez et al. 2018), questions are used as the conditioning vectors, and similarly in conditional domain adaptation techniques (Long et al. 2018). More generally, conditional neural networks accept an additional conditioning vector, that influences how the networks process inputs produce outputs.
Most common approaches for conditioning neural networks either concatenate or modulate the conditioning vector with one or more intermediate representations of the network, before the updated representations are processed by layer, e.g., see Figure b for an overview. Such conditioning techniques come at the cost of significantly increasing the network parameters by inflating the dimen- sionality of intermediate features by requiring additional network parameters proportional to the dimensionality the conditioning vectors and the number of channels present in the representations.
In this work, introduce new approach for neural network conditioning and show that effective conditioning can be achieved by learning layer-specific activation functions based on the conditioning vectors. Unlike previous approaches, our learned activation (LA) functions approach does not increase the model parameters significantly, since the number of additional parameters it requires depends only the size of the conditioning vector, and not the shape of the feature representations nor on the output shapes of any subsequent layers. LA uses the conditioning vectors to dynamically modify intermediate activation functions, which are + R (1-D) mappings, see Figure Ic. Dynamic
"qqual contribution

---

Published as a conference paper at ICLR 2022
Activation, B.g Ralu
Activation, e.g. Reiu
Learned Activation
Denie: Linear
Dense, Linear
Weights
Dense, Linear
Wasted
Weights
Weights
Dense, Linear
Unsstack
Bou Concatenate
(b)
(c)
Figure 1: Overview of neural network conditioning methods with two inputs, Rbxd conditioning
vector and æ E Rgx- input feature: (a) conditioning via concatenation, (b) modulation and (c) the proposed learned activation (LA) functions, illustrated here together with dense layer.
LAS allow us to extract important insights on how the model is being adapted for different conditioning vectors by simply analyzing the learned activation functions, which are agnostic to the input.
In this work we consider two popular and timely sqquence-to-sequees sound enhancement (PSE) and personalized ASR (PASR) as exemplary tasks for modeling conditional neural networks. PSE leverages a speaker embedding vector, i.c., a unique encoded voice profile of a speaker, for filtering out everything but the speaker's s voice from audio recording corrupted with babble or ambient noise (Wang et al., 2020 2019). Speaker embeddings can generally be obtained by using an embedding network once during an enrolment phase, e-g., from recorded utterances like "Hey Alexa" or "Hey Bixby". PSE networks can be evaluated for improving signal quality for telephony or can also be used to improve the quality of downstream tasks, such as a pre-trained ASR. PASR on the other hand, aims to improve ASR quality in single speaker scenarios, by leveraging user's enrollment data to enhance performance directly without any pre-processing
modeling tasks: personalized
We evaluate LA on three PSE model families ($4.2), on two datasets Librispeech (Panayotov et al. 2015) and Voxforge (vox (006). We show that the use of LA on all models achieves competitive performance to conditioning using concatenation and modulation approaches. At the same time, concatenation approaches require 10-95% more model parameters over our solution. For PASR, we compare the performance of personnsnnd and personalized versions of the conformer encoder model architecture (Gulati et al., (020)) with a dense layer for character or subword text encoder trained with CTC loss and decoded via CTC decoder (:5.2). Results on Librispeech (Panayotov et al.) 2015) show that LA yields effective ASR personalization, i.e., increased performance across all users while using two seconds of enrollment data, with up 19% relative WER improvement across all users, and up to 40% for some users.
RELATED WORK
Conditioning through tensor concatenation The concatenation of intermediate features with latent vectors before being passed to subsequent layers is the most common form of conditioning, but leads to inflated model sizes, which hinder their deployment to low-resource constrained devices
(Wang et al., 2019 2020; Valin et al.) 22000; Hu et al.) 2020
Comon & Jutten [2010] Wang et al.
2021
Lam et al. [202 Vincent et al.] 2018 Luo & Mesgarani
2019; Xu 20000 Abdulatif et al.
2020 Wisdom et al. 2020 Isik et al. 2020 Luo et al. 2020 Choi et al. 2021 (Snddr et al. 2018)
Panayotov et al. 2015 Bell al., 2021 He al. 2019 van den Oord et al. 2018). Our approach
overcomes this issue and learns smaller models with competitive quality.
Conditioning via normalization layers Another family of techniques is based having per channel learned linear transformation based on the conditioning vector, which yields a form of conditional normalization (Perez et al. 20188 de Vries et al., 2017 [Huang & Belongie 2017). Our approach distinguishes itself from this line of work since it learns the non-linear components of the network rather than the linear ones, and thus offers greater modelling lower cost.
Learning activations without conditioning Outside the context of conditional neural networks, there is work on learning an optimal activation function good default to be used in all layers of
2

---

Published as a conference paper at ICLR 2022
2020: Ramachandran et al. 2018 Agostinelli et al.] 2015). Our work instead i) learns conditional activations based on conditioning vectors, and ii) uses differently learnt activations throughout the network.
most neural network architectures (Apicella et al. 2021 Liu et al.
3 CONDITIONING BY LEARNING ACTIVATIONS
Conditioning neural network corresponds to using some form of latent vector or additional prior knowledge, e.g., encoded typically in the form of an embedding vector, to influence the output of the network for a given input. We show in the evaluation that the influence of the conditioning vector can be successfully modeled by learning a weighted combination of set of basic activation functions (3..) and carefully selecting hyperparameters (33.2)
3.1 DYNAMICALLY LEARNED ACTIVATIONS
To introduce our conditioning approach using dynamically weighted activation functions, we begin by defining few key variables: 2 E 1x 2-D tensor representing b number of d-dimensional conditioning vectors, where b is the batch size, x E T is the input data with arbitrary tensor dimensions (e.g., vectors, matrices, higher order tensors), h R the hidden pre- retaviivt of the previous layer, and {A, R + R is an ordered family of a basic activations, e.g., frelu, sigmoid, tanh) (we will discuss below more on the elements of this ordered set of activations). Dynamically learned activation (L.A) functions require small number of trainable variables w pd and b E R and are defined by transforming input tuples (z, h) into output tensors y through (note the parallel between the following equations and the pictorial representation in Figure Te. where there is direct correspondence on the roles of z, h and between the formulas and the diagram):
= softmaxxowwised (z x w + b),
h EER- R (((h/2))
Σ
8, Ai(h);
yy... LA Leeeementwise (hj.... 1z)):
where softmax denotes the matrix- matrix operation with entries defined rowwise via softmaxrowwise (m); == softmax(m, (m,). and L.Aeementtisee (hj... zz) denotes the result of applying L.(.1 zj) elementwise to each entry of h.....
Note that during training. batch size is greater than one in which case is a matrix, whereas during inference batch size one for each user which case z is a row vector. So during training zj is the conditioning vector corresponding to the j-th sample, but during inference zj is the conditioning vector corresponding to the j-th user in the discussion below.
Figure D highlights the qualitative and quantitative differences between our approach (Figure and existing state-of-the-art (SOTA) solutions (Figure Ta-b), Concatenation approaches as depicted in Fig. ll can be defined by Aelementwiso (Fe((z) D. j...... ("). where Fe denotes common learnable transformation such as a dense, convld, conv2d or LSTM layer without activation and A R - R denotes generic activation such the rectified linear unit. Whereas, shown in Fig. [pp modulation approaches are defined as Aelementwise (G,(U (Un(()) @ æy....: + Vu(z)))). where Go represents a common learnable transformation (which may be slightly different than Fo given their inputs have different shapes) and UP Rd + / and Vu Rd + R are often taken as linear mappings (where d denotes the conditioning vectors dimension and f denotes the number of channels of x))
In comparison to these SOTA approaches, our learned activation functions mechanism can simply be formulated as LAglementwiee (........ 2,). For simplicity, we use a dense layer in the figure (i.e., we take Fe and G, to be dense layers and = 1), however, our approach is general and can be applied to any layer type, e.g. dense, conv I,, conv2d and LSTM. From a qualitative point of view, LAS conditions the non-linearities rather than the linearities of the network. From a quantitative point of view, Figure T also highlights the superior compression abilities achieved by our proposed method.
In particular, concatenation approaches (Figure D require (d+ u) parameters and modulation approaches (Figure [p)) require u * v + d x * J. where f denotes the number of channels in the input tensor which per channel is multiplied by scaling factor and offset by bias term. The learned activation functions approach on the other hand, requires only u + d * a parameters. For example, in most audio and video applications, our approach yields significant savings of d (v - a) given that d > and > a, when compared to concatenation approaches. The saving becomes d (2+ f a) when compared to modulation approaches, given that often in such cases 2
0.
3

---

Published as a conference paper at ICLR 2022
For temporal data, such audio and video, the savings of our approach more significant as in those applications the conditioning vector is concatenated with each time frame of the inputs and for f > 1 the model requires larger parameter size. In our approach the operational requirements do not change across data type, i.e., tensor dimensions.
3.2 HYPER-PARAMER PARAMETERS
Basic Activations As discussed above, conditioning approach relies set of standard activations provided for modelling. Therefore, the selection of these basic activations is an important hyper-parameter in our approach. Following are the key characteristics that should be taken into account when searching for an optimal set of activations:
Dynamically learned activation coefficients 8j all positive and sum one. Thus, some families of activations will generate learned activations that remain in the same family, however, this not always true. For example, if all basic activations are all increasing functions, so is the learned activation. Monotonic increasing property of the learned activation may not be retained if we include both increasing and decreasing functions in the basic activation set.
The fact that 8, are in the probability simplex means that the ranges of the basic activations play an important role in the final learned activations. One could consider basic activations with i) unbounded, ii) upper bounded, iii) lower bounded iv) bounded ranges.
Finally, the basic activations could instead be picked from well known basis functions. For instance, Fourier basis or Chebyshev polynomials could be used create bounded learned activations with compact support, and Hermite polynomials to create unbounded learned activations on the real line. Note that this direction of investigation is outside the scope of our paper.
Initialization Choices Another important syper-parameter to consider is what are the values of the dynamically learned coefficients 8, at initialization time for most conditioning vectors 2,. Given 8j is probability distribution, start training with an unbiased model, they could be at initialization time approximately uniform, i.e., providing roughly the same weight to each of the basic activations for most zj. Alternatively, they could be such that for most 2j at the beginning of training the learned activations have desired shape. Therefore, trainable variables w and b should be initialized considering i) the statistics of the conditioning vectors Zj. and ii) the shapes of the basic activations.
4 EXPERIMENTS PART I: PSE FOR TELEPHONY AND PRE-TRAINED ASR
In this section we consider the task of PSE, with the aim of enhancing a user's voice to improve telephony and pre-trained ASR quality under babble (i.e., other speaker's 's voices) or ambient noise, in real-world scenarios with only two seconds of enrolment audio of non-constrained speech.
4.1 DATASETS
We consider two representative datasets: LibriSpeech (Panayotov et al. 2015) and Voxforge (vox) 2006), which cover English and Spanish respectively. For the English training set, we take 100h and 360h of clean speech from Libriippechh, which are recorded using close-talk microphones, without any background noise. Although, supervised learning approaches have shown great performance in sound enhancement tasks, it is often impossible to collect the mixed audio and its ground-truth evaluate sound sources from the same audio environment (Wisdom al., (020). Furthermore, performances of learned activation on realistic data, we create training dataset by taking 32.56 of Spanish audio from Voxforge, which are mainly recorded with a far-field microphone and contain noise. Both datasets have audio recorded at 16kHz.
The raw datasets consist of tuples (speech, transcript, speaker identifier). For each pair of dif- ferent speakers a and b, we collect two speech samples from and one from b to create 4-tuples (speech,, sppeech,,.peech, + speech,,
text')) ). with elements being: i) a reference speech from user to be passed through speaker embedding model (Snyder et al. 2018) to create the conditioning vector, it) a speech from user a, different from that given to the embedding model, which corrupted (at various db levels) with speech from user b, dii) the ground-trui -ruth speech by user present in the corrupted speech, and, iv) a transcript of the ground-trutte speech by user a. All PSE models are trained in supervised fashion, where the goal is recover speech,, from speech', + speech, as closely as possible for telephony applications and such that ASR (PSE (PSE(speech, + +specch,) = text', for ASR applications. We also evaluate performances under ambient acoustic noise by constructing (speech,,, speech,, + noise, speech', text',) ) tuples, where noise is sampled from the DEMAND dataset (Thiemann et al. 2013).
4

---

Published as a conference paper at ICLR 2022
-2
Figure 2: Learned activations (LA) for models trained on Voxforge, for speaker embedding vectors of randomly selected users in a PSE application. To take into account that basic activations may have different ranges, the plots show LAs minus average of basic activations, i.c. LA(h) xz) a a'\ A, (h) for h [-3, 3] and values of corresponding to the selected users. The plots highlight that LA exhibit different profiles across different and layers.
Table 1: Souste-to- Ratio improvement (SDRi) (higher the better) and Word-Error-Rate Distortion (WER) (lower the better) performances on LibriSpeech (English) for models with conditioning via learned activations (LAs), modulation (FiLM) and concatenation, across models with different number conditioning layers (#Cond.).
SDRi
WER
Model
Type
#Cond.
Params (M)
Babble
Ambient
Clean
Babble
Ambient
Identity
8.2
89.4
17.3
RNN
LA
8
4.31
8.47
12.02
8.3
24.8
15.2
RNN
FiLM
5.34
7.39
11.41
9.4
31.2
16.4
RNN
Concat.
6.13
8.57
12.19
8.3
24.5
15.0
TDS
LA
42
2.51
7.33
11.58
8.2
33.0
15.7
TDS
FiLM
4.10
6.74
11.34
8.3
37.5
16.9
TDS
Concat,
4.89
7.78
11.72
8.3
30.0
15.7
TDS-RNN
9.56
8.96
12.44
8.3
21.9
14.3
TDS-RNN
FiLM
9.87
8.36
12.08
9.0
25.4
15.2
TDS-RNI
Concat.
10.48
9.49
12.49
9.3
18.2
14.3
Table 2: Source-to Couure-to-Distoriie Distorrtttinn Ratio improvement (SDRi) (higher the better) and Word-Error-Rate (WER) (lower the better) performances on VoxForge (Spanish) for models with conditioning via learned activations (LAs), modulation (FiLM) and concatenation, across models with different number of conditioning layers (#Cond.).
SDRi
WER
Model
Type
#Cond.
Params (M)
Babble
Ambient
Clean
Babble
Ambient
Identity
1.2
89.1
9.2
RNN
LA
8
4.31
6.96
9.81
1.9
20.8
6.7
RNN
FiLM
5.34
5.92
8.33
3.0
25.5
8.2
RNN
Concat.
6.13
6.84
9.64
1.4
22.8
6.4
TDS
LA
42
2.51
5.37
8.21
1.4
33.3
7.7
TDS
FiLM
4.10
6.09
8.45
1.8
24.3
7.3
TDS
Concat.
4.89
5.19
8.02
1.6
35.6
7.6
TDS-RNN
LA
9.56
7.17
10,36
1.8
19.5
6.0
TDS-RNN
FiLM
9.87
7.23
9.91
2.3
18.0
6.4
TDS-RNN
Concat.
10.48
7.69
10.50
1.9
15.2
5.9
4.2 BASELINE MODELS
variety of architectures have been proposed lately for speech enhancement. In this work, we consider architectures suitable for low-latency on-device deployment that have small memory footprints and operate in real-time. With that in mind, we use the following three architectures as baselines:
RNN. These are RNN models with skip-connections Takeuchi et al. 2020 Wang et al.) 2020 (c.f., Figure 5a in Appendix A).
TDS. These models (Figure 5b in Appendix A) comprise stacked -ime-depth-spprrabe (TDS) convolution blocks (Isik et al. 2020 Luo & Mesgarani 2019).
5

---

Published as a conference paper at ICLR 2022
TDS-RNN. These encoder-decoder style models with separator block in between (Figure 5c in Appendix A). Encoder and decoder are made of stacked TDS blocks and the separator of stacked RNN blocks (Choi et al. 2021).
Furthermore, we consider a different number of conditioning layers (#Cond.) for each model family, In particular, for RNN 8, for TDS 21 and for EDS-RNN22 2 conditioning layers leading to 8, 42 and nodes in the models being influenced by the conditioning information (c.f., Figure in Appendix A). We have done so to investigate also how the number of conditioning layers influences the personalization effects from the conditioning vectors (c.f., Appendix A.2).
In our experiments we apply LAs in all three families of models and perform extensive comparisons with the baselines in terms of performance, model size and latency. In particular, take the best performing baseline concatenation architectures, not the weights, and for every layer with a condi- tioning concatenation, we replace that with modulation layer or a conditionally learned activation. Finally we train the modulation models and the conditionally learned activation models from scratch, without performing any architectural hyper-parameter optimization (HPO). As a representative of approaches, we consider the Feature-wis wise Linear Modulation (FiLM) (Perez et al., 2018) mechanism throughout. Notably, our evaluation experiments give more advantages to the baseline concatenation models.
light
4.3 HYPER-PARAMETER
Basic Activations The choice of basic activation functions is an important hyper-parameter in our approach and we consider comprehensive list of popular activation functions used in deep learning. Specifically, we use the following ordered set as the basic activation functions: [A, : - R): :-1
felu, exponential, hard_ sigmoid, linear, relu, selu, sigmoid, softplus, softsign, swish, tanh].
Initialization Choices We consider enrolment data (e.g., short pre-defined utterance spoken by the user once) to extract (using a fz-normalized X- K-eector model (Snyder et al., (2018) a speech embedding vector of the target user, i.c., the owner of the device. This speech embedding is used as the conditioning vector in PSE models to enhance the user's 's voice and suppress unwanted voices and background noise. Although, the embedding vectors in the training set are not perfectly zero mean and unit variance, the fact that they are Ez-normalized motivates us to take for initialization of the weight matrix in the LAs the Glorot uniform initializer and for the bias vector the zero initializer.
4.4 TRAINING SETTINGS
All models considered in this work operate in the fime-freyuency domain. We apply Short Time Fourier Transform (STFT) to extract 512 coefficients using a 32 ms window and with a stride of 16 ms. Models are trained in data-paraliel way with four GPUs using TensorFlow and Horovod. We use batch size of 64, learning rates in [1c-5, 3c,3], exponential decay learning rate scheduler and early stopping (max epochs 100).
4.5 EVALUATION METRICS
We evaluate our models in terms of source-to-distortion ratio improvement (SDRi) (Comon & Jutten 2010) and word-error-rate (WER) metrics. A high SDRi indicates the improvement in signal quality for telephony and a low WER quantifies good transcription quality of the downstream ASR task. We use open-source Silero (sil 202T ASR models (English and Spanish) in our evaluations.
4.6 RESULTS
This subsection presents evaluation results for quantifying the effectiveness of learned activations (LA), compared concatenation- and modulation-based conditioning approaches, on LibriSpeech (Table and Voxforge (Table 2 datasets. The results show that models with all three conditioning approaches improve telephony (high SDRi) and ASR (low WER) performances on both datasets. We observe that I.A and concatenation- indicating high efficacy of the proposed LA approach. Note that when evaluating ASR, it is also important that good PSE model shows almost degradation in WER, when clean audio is presented. Interestingly in some cases in the concatenation- based approach, we see that higher success in removing babble noise, comes with an adverse impact in clean speech WER, mainly due to over-suppression (Wang et al. (2020). Furthermore, Table 2 show that both concatenation and modulations produce significantly larger models, i.e., up 95% larger, compared to LA approaches.
conditioning approaches yield very similar performances,
Tables I 2 highlight the trade-off between the number of parameters and quality metrics. However, is important also to consider the impact of the different approaches in terms of FLOPS and latency,

---

Published as a conference paper at ICLR 2022
LA
0.1
0.-
UserA
-2
Figure 3: Learned activations (LA) for models trained on LibriSpeech, for speaker embedding vectors of randomly selected users in PASR application. To take into account that the non-personalized basic activations are relu or swish, the plots show LAs minus average of those two activations, i.c., LA(h) 23) (relu(k) + swish(h)) 2 for h [-3,3] and values of j corresponding to the selected users. The plots highlight that LA exhibit distinct profiles across different users and layers.
Table 3: FLOPs and latency (for one second audio) for PSE models with conditioning via learned acti- vations (LAs), modulation (FiLM) and concatenation (Concat.). Note FLOPs may not be completely accurate given the common lack of support of profiling tools for non-standard layers or operations.
Model
Type
FLOPs (M)
Latency CPU (ms)
Latency GPU (ms)
RNN
LA
1254.85
462.997
(=14.719)
39.156 (+31.339)
RNN
FiLM
1254.85
465.785
35.769 (=01.344)
RNN
Concat.
1802.60
537.766
(=11.403)
46.716 (+00.903)
TDS
LA
1507.43
127.420
(=02.996)
78.998 (=#1.963)
TDS
FiLM
1507.43
95.535
(=1),861)
70.265 (=01.975)
TDS
Concat.
2906.46
123.580
(=03.440)
84.614 (=02.479)
TDS-RNN
LA
4406.60
380.014
(1116.604)
233.748
(105,3115
TDS-RNY AS-RNN
FiLM
4406.60
387.325
101)
238.787
(=11.882)
TDS-RNN
Concat.
4543.92
394.809 (=20.247)
237.437 (=06.928)
shown in Table (where latency was measured by passing one second of data one hundred times by the models and summarized via mean and standard deviation), which demonstrates LAs are competitive this dimension. Specifically. LA is faster than standard concatenation approach with comparable quality, and slower than FiLM but with higher quality (i.c., higher SDRi and lower WER).
To gain better understanding of how LAs adapt based on conditioning vectors, Figure 2 presents plots centered at the origin of the LAs for a PSE application conditioned on speaker embedding vectors extracted from enrolment audio from randomly selected users. In particular, we can see that for all models i) in each layer, different users have different LAs, and ii) each user has different LAs across different layers. As such the visualization of the internal mechanisms of LAs, provides tangible way to improve how we understand the manner in which models leverage conditioning vectors to influence predictions.
5 EXPERIMENTS PART II: PERSONALIZED ASR
In this section consider the problem of personalized ASR, with the objective of improving ASR performance in single speaker scenarios. This is a challenging task, especially in realistic setups with only couple of seconds of enrolment audio with non-constrained speech which explore below.
5.1 DATASETS
We consider LibriSpeech (Panayotov et al. 2015) 1000h of English annotated audios with transcripts. We augment each sample tuple (audio, transcript) to include two seconds enrollment audio from the same user (audio, transcript, enrollment). We ensure the enrollment audio is taken from a different sample from the same user. For diversity during training the enrollment audio is selected randomly under the aforementioned constraint, whereas for evaluation always the same.
5.2 BASELINE MODELS
ASR. For non personasndd ASR models we take the conformer encoder model architecture Gulati et al. 2020) with a dense layer for character (with 28 characters, namely: 'A'-ZZ' and or subword (with 753 subwords) text encoder trained with CTC loss and decoded via CTC decoder. These models take input audio and output a predicted transcript.
PASR. For personalized ASR take the ASR architecture but replace the ReLU and Swish activations with our proposed LAs (see [3.1). We consider basic activations (see (3.2) containing ReL.U and Swish, and an initialization of the bias (see, (.2)) that at initialization time, the LAs
7

---

Published as a conference paper at ICLR 2022
replacing ReLU are approximately ReLU, and the LAs replacing Swish are approximately Swish, a profile which changes during training as a mechanism to enable personalization. As a similar mechanism to the bias offset in LAs, we altered FiLM's definition Aetementwisel (G, (U, (2g) @
" + (z(z))) to Aelementwise (,,(( and 8 as trainable variables, so that at initialization FiLM is approximately the identity function for most conditioning vectors. As with non- non-personalized ASR, we consider character and subword text encoders trained with CTC loss and decoded by CTC decoder. These models take as input an audio and a fixed speaker embedding vector for each user computed from enrollment data by a speaker embedding network (Snyder et al., 2018), and output a predicted transcript.
a- tanh (z;))) @ "...... + 3 (2) with
5.3 HYPER-PARAMER
For consistency with previous experiments, we take as the basic activations those used in 4.3 For initialization choices mentioned earlier we introduce bias offsset so that at initialization the softmax & described in i3. approximately one-hot to leverage the inductive bias of activations proven to work well in non-personalized settings. Interestingly, we observed in experiments that subword models seem to be more sensitive to this hyper-parameter than character models.
5.4 EVALUATION METRICS
For character and subword models we consider greedy (beam search with tops paths and beam width equal to one) and beam search (with a beam width of four) CTC decoders. Once predicted transcripts are produced from the models, they are compared with ground-truth transcripts in terms of word error rate (WER) and character error rate (CER).
5.5 RESULTS
We consider the performance of PASR models created from existing ASR models by replacing their fixed activations with LAs. To delve deeper into the effect of L.s on PASR, we consider two dimensions to our experiments, namely, in terms of whether i) training is performed only on the speaker embedding model as well as on the weights and biases of LA, or ii) training is done on all parameters. In both cases, optimization is done with respect to lower CTC loss. These settings are designed on the one hand understand what are the effects of taking pre-trained ASR models and personalizing them by replacing their activations with LAs, and on the other hand to understand the capacity and convergence properties that LAs provide for PASR. Before a fine-grain analysis of the results below, note that similarly to Figure 2 for PSE, Figure 3 shows that LAs consistently create different activation profiles across users and layers for PASR as well.
5.5.1 TRAIN SELECTED PARAMETERS
Leaving the parameters of the original ASR intact, allows for switching between the original static activations and the LAs, which might be beneficial for deployment scenarios lacking clean enrolment data. We investigate this scenario by fine-tuning only the speaker embedding model and LA param- eters, which results in an overall 2% WER relative improvement which is a modest but significant improvement given an already low WER/CER of the original ASR. Figure (top) shows that for some users this improvement is even more significant, as high as 15% WER relative improvement.
5.5.2 TRAIN ALL PARAMETERS
In this section we investigate the performance of PASR models derived from training all their parameters after initializing their non-personalized components from existing ASR models.
Results for all users together Pable/ccontaing WER/CER results aggregated across all users of LibriSpeech test-clean. Overall the results show a consistent improvement in performance, i.c.. lower WER/CER for personalized ASR via LAs compared with non-personalized ASR. The results also show that subword models yield better WER, but character models give better CER, with the gains through personalization being higher for character than subword models, being as high as 19% relative WER improvement across all users.
Results for each user separately Figure (bottom) shows plots of WER relative (in terms of percentage) and absolute improvement per each user of LibriSpeech test-clean for character models with CTC greedy and beam search decoders. Each plot shows three curves, namely: i) the horizontal blue line y = 0 representing the performance of the non-personalized ASR, ii) the orange curve representing the performance of the non non-personalized ASR with its activations replaced by randomly initialized LAs, and iii) the green curve representing the performance of this personalized ASR fine-tuning speaker embedding model, weights and biases of LAs, as well as the parameters of the non-personalized ASR. The results show an initial mixed degraddtiion/mppooeeees per user when
8

---

Published as a conference paper at ICLR 2022
Table 4: WER and LER results on LibriSpeech test-clean for ASR and PASR (via FiLM modulation and proposed LA functions approaches) with two seconds of enrollment data, for character and subword text encoders and CTC greedy (G) and beam search (BS) with beam width four decoders.
Models
ASR
PASR
Type
FiLM
LA
Encoder
Char
Subword
Char
Subword
Char
Subword
Decoder
G
BS
G
BS
G
BS
G
BS
G
BS
G
BS
WER
8.93
8.77
7.69
7.62
7.41
7.27
6.99
6.93
7.26
7.15
6.41
6.35
CER
2.65
2.62
5.04
5.01
2.20
2.18
4.78
4.75
2.17
2.14
4.39
4.36
Kaa
Train LA
Train all
Spesker
CTC greedy decoder
(b) CTC bearn search deccder
Figure 4: WER relative and absolute improvement on LibriSpeech test-clean fine-tuning speaker embedding and LAs (top), training everything (bottom) for CTC greedy (a) and beam search with beam width four (b) decoders, with two seconds of enrollment data, for character text encoder.
replacing the original activations with LAs expected without training (ensured by appropriate choice of the bias offset hyper-parameter as described in (5.3). and the benefits of training the personalized ASR, with significant gains in WER per user, in terms of relative and absolute WER. Furthermore, the gains can be seen to reach up to 40% and 35% relative improvement for some users for CTC greedy decoder and CTC beam search decoder respectively, which is significant improvement in the domain of ASR.
6 LIMITATIONS
Our evaluations are limited to two supervised audio domains, namely: i) PSE to improve telephony and pre-trained ASR quality for English and Spanish, under babble and ambient noise, and ii) personalized ASR for single speaker scenarios over English, based only on two seconds of enrollment data. We nevertheless see value in assessing our approach in generative or image domains. Another limitation stems from the fact that the evaluation is performed with speaker embeddings computed from clean enrolment data, which might not be available at deployment time. So the robustness to this source of issues is not explored in experimental setting. This limitation common in most, not all studies, in these application domains.
7 CONCLUSIONS
We introduced new general method for designing conditional neural networks. Unlike state-of-the. art techniques, which are based on concatenation or modulation strategies, we presented a new form of conditioning by dynamically learning activations. By conducting exiensive set of experiments, we highlighted the efficacy of our modeling technique in terms of quality metrics and model sizes. Although, mainly focus on PSE under babble and ambient scenarios and personalized ASR for single speaker cases as the main application domains, we believe our approach will open ways for the generalized use of dynamically learned activation functions in many application domains.

---

Published as a conference paper at ICLR 2022
REFERENCES
VoxForge Corpus.
2006.
Silero models: pre-trained gradr stt / its models and benchmarks. https://gitti com/ /wtaaaaarrr//aaleeooooeeeeeee 2021.
Sherif Abdulatif, Karim Armanious, Jayasankar T. Sajeev, Karim Guirguis, and Bin Yang. Investigat- ing Pross-Domain Losses for Speech Enhancement. arXiv, 2020.
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning Activation Functions to Improve Deep Neural Networks. In ICLR, 2015.
Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, and Roberto Prevete. A survey on modern trainable activation functions, Neural Networks, 138, 2021.
P. Bell, J. Fainberg, O. Klejch, Li, S. Renals, and P. Swietojanski. Adaptation Algorithms for Neural Network-Baser Speech Recognition: An Overview. IEEE Open Journal of Signal Processing, 2, 2021.
Ting Chen, Mario Lucic, Neil Houlsby, and Sylvain Gelly. On Self Modulation for Generative Adversarial Networks. In ICLR, 2019.
Flycong-Seok Choi, Sungjin Park, Jie Hwan Lee, Hoon Heo, Dongsuk Jeon, and Kyogu Lee. Real- time Denoising and Dereverheration with Tiny Recurrent U-Net. In ICASSP, 2021.
P. Comon and C. Jutten. Handbook of Blind Source Separation: Independent Component Analysis and Applications. Springer. 2010.
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating Early Visual Processing by Language. In NeurIPS, 2017.
Anmol Gulati, James Qin, Blung-Chengy Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo transformer for
Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: speech recognition. In Interspeech, 2020.
Yanzhang He, Tara N. Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, Qiao Liang, Deepti Bhatia, Yuan Shangguan, Bo Li, Golan Pundak, Khe Chai Sim, Tom Bagby. Shuo-yiin Chang, Kanishka Rao, and Alexander Gruenstein. Streaming End-to-end Speech Recognition for Mobile Devices. In ICASSP, 2019.
Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancement. Interspeech, 2020.
Xun Huang and Serge Belongie. Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization. ICCV, 2017.
Umut Isik. Ritwik Giri, Neerad Phansalkar, Jean-Mare Valin, Karim Helwani, and Arvindh Kr- ishnaswamy. PoCoNet: Better Speech Enhancement with Deequency-Positionee Embeddings. Semi-Supervised Conversational Data, and Biased Loss. In Interspeech, 2020.
Max W. Y. Lam, Jun Wang, Dan Su, and Dong Yu. Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks. In SLT, 2021.
Hanxiao Liu, Andrew Brock, Karen Simonyan, and Quoe V. Le. Evolving Normalization-Activviio
Layers. In arXiv, 2020.
Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael Jordan. Conditional Adversarial Domain Adaptation. In NeurlPS, 2018.
Yi Luo and Nima Mesgarani. Conv-TasNet: Surpassing Ideal Time- Frequency Magnitude Masking for Speech Separation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 27 (8), 2019.
10

---

Published as a conference paper at ICLR 2022
Yi Luo, Cong Han, and Nima Mesgarani. Uitt-Lightweigtt Speech Separation via Group Communi- cation. arXiv, 2020.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ASR corpus based public domain audio books. In ICASSP, 2015.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with General Conditioning Layer. In AAAI, 2018.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for Activation Functions. In ICLR, 2018.
Lahiru Samarakoon, Khe Chai Sim, Lahiru Samarakoon, and Khe Chai Sim. Factorized hidden layer adaptation for deep neural network based acoustic modeling. 24(12), 2016.
George Saon, Hagen Soltau, David Nahamoo, and Michael Picheny. Speaker adaptation of neural work acoustic models using i-vectors. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, 2013.
David Snyder, Daniel vectors: Robust DNN Embeddings for Speaker Recognition. In ICASSP, 2018.
Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. X-
Daiki Takeuchi, Kohei Yatabe, Yuma Koizumi, Yasuhiro Oikawa, and Noboru Harada. Real-time time speech enhancement using equilibriated mn. In ICASSP. 2020.
Joachim Thiemann, Nobutaka Ito, and Emmanuel Vincent. DEMAND: collection of multi-channe recordings of acoustic noise in diverse environments, June 2013. URL htttsstss:///tt orgg/00 5281/zenodo: 1227121 Supported by Inria under the Associate Team Program VERSAMUS.
Jean-Mare Valin, Umut Isik, Neerad Phansalkar, Ritwik Giri, Karim Helwani, and Arvindh Krish- naswamy. A Perceptually-Motivacee Approach for Low-Complexity, Real-Time Enhancement of Fullband Speech. Interspeech, 2020.
Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel WaveNet: Fast High-Fidelity Speech Synthesis. In ICML, 2018.
Laurens Van der Maaten and Geoffrey Hinton.
Visualizing data using
t-SNE.
Journal of machine learning research, 9, 2008.
Emmanuel Vincent,
Tuomas Virtanen,
and
Sharon
Gannot.
Audio Source Separation and Speech Enhancement. Wiley Publishing, 2018
Kai Wang, Bengbeng He, and Wei-Ping Zhu. TSTNN:
Two-stage Transformer based Neural Network for Speech Enhancement in the Time Domain. In ICASSP, 2021.
Quan Wang, Hannah Muckenhirn, Kevin Wilson, Prashant Sridhar, Zelin Wu, John R. Hershey, Rif A. Saurous, Ron J. Weiss, Ye Jia, and Ignacio Lopez Moreno. VoiceFilter: Targeted Voice Separation by Speaker-Conditionn Spectrogram Masking. In Interspeech, 2019.
Quan Wang, Ignacio Lopez Moreno, Mert Saglam, Kevin Wilson, Alan Chiao, Renjic Liu, Yanzhang He, Wei Li, Jason Pelecanos, Marily Nika, and Alexander Gruenstein. VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition. In Interspeech, 2020.
Scott Wisdom, Efthymios Tzinis, Hakan Erdogan, Ron J. Weiss, Kevin Wilson, and John R. Hershey. Unsupervised Sound Separation Using Mixture Invariant Training. In NeurlPS, 2020.
Ruilin Xu, Rundi Wu, Yuko Ishiwaka, Carl Vondrick, and Changxi Zheng. Listening To Sounds Of Silence For Speech Denoising. In NeurlPS, 2020.

---

Published as a conference paper at ICLR 2022
A
APPENDIX
A.1
BASELINE MODEL FOR PSE
Figure presents pictorial representation of the PSE models introduced in 84.2.
1944 Block
IVUN Nodel
TDS Bleck (TDS3)
Model
INN Gleck
TDS Book (TOG))
TDSS TD2---NI Model
Concaleraaioon (C) Modulaaion (M)
Deise (RANG
Donse
Denoel
CTEEB
Dengo
Dispout
(TDBB
(Bhike
Fileer (RSSEE
CandID
ConriD
Frocrder
C
RVN Block (PPINE)
RWN Modsl
TDB Bleck (TD63)
TDS Model
RVN Block (1R44))
YDB Book (1BB))
TDe 994 Model
PAN
Deyse (Bvan)
Dense
Denise
Learned Activations
Danse
Denos
Crepoun
(PANe
TEAS
Filter
(RISURT)
CondiD
ComrtD
Enscader
(e)
(b)
(c)
Figure 5: A pictorial representation of the RNN (a), TDS (b) and TDS RNN (c) models used in the evaluation of conditioning based on concatenation (top, C), modulation (top, M) and based on the proposed learned activations (LAs) (bottom).
A.2 REGULARIZERS
We also investigate the use of regularizers during training for learning activation functions, in order boost deployment and conditioning efficiency. We consider two regularizers: entropy-based and t-SNE (Van der Maaten & Hinton, 2008). Through entropy regularization we encourage learned activations to approximately select one element of family of predefined activations (with applications to efficiency). Whereas, with t-SNE regularization
promote that similar/dissimilar conditioning vectors yield similar/dissimilar learned activations, thereby instilling the geometry of the speaker embedding space in the conditioning mechanism (for enhanced conditioning quality).
A.2.1 APPROXIMATELY ONE-HOT ASSIGNMENTS AND GEOMETRY
We explore mainly two regularization techniques to promote certain qualitative properties in the learned activations during training. The first technique encourages the learned activations ap- proximately select one element of {A; lR - R based on the conditioning vector. Different non-linear functions require different amount of computational resources, thus biasing the learned activations towards a single known form can improve efficiently. in comparison to their weighted combinations. The second technique guides training so that similar/dissimilar conditioning vectors yield similar/dissimilar learned activations, thereby preserving the geometry of the speaker embedding space the conditioning mechanism. Considering a particular use-case PSE, there might be some variation in the conditioning vectors (i.e., speech samples collected during enrolment). This turns into a challenge of selecting or aggregating these vectors to properly condition the network. The use of our second technique for regularization during training could reduce the impact of such variations in conditioning vectors at deployment phase by increasing the stability of the network against subtle variations in the conditioning vectors.
12

---

Published as a conference paper at ICLR 2022
Learning approximately one-hot assignments Sparse assignments be encouraged by adding known regularizers, such as entroypy or & to the loss during training, scaled by a ie., (H(s) a||s|]1. Alternatively, sparsity can be promoted by considering temperature parameter t S 1. For example, a value lower than one makes the model more confident, whereas higher than one makes it less Thus sparse activation can be learned as: softmaxrowwise (t-1(z w + b))-
a,
Approximately preserving conditioning vectors geometry Learned activations are built based on conditioning vectors, and we may expect these functions have a discriminative property for the conditioning vectors they are based on. In other words, clusters in 2j space would be correlated to clusters in the space of learnt conditioned activations weights 8, for similar/dissimilar conditioning vectors. However, strong correlation may not happen automatically due to: i) the dimension of the conditioning vectors is often much larger than the number of basic activation functions a, i.e., >> a, ii) the manner in which 2j is used to produce 8j may be affected by the fact that softmax is invariant to translation, i.e., softmaxx (v + c) = softmax (v), and iii) the similarityddissiilartty of the basic activations (A, : R- = High correlation can be promoted though by geometry aware high- to low-dimensional embeddings like t-SNE (Van der Maaten & Hinton 2008). Although typically used for data visualization, instead leverage the t-SNE objective function regularizer in our loss function. Specifically, we take t-SNE probabilities p, q E [0, 1]000 defined by p:; == qiçi = 0 and
for i j:
c c-11=:-=,11
Pili + P
(1+ 8: Ex (1 ||sk st 11
8, 1;
Paje Extie 1x: -=:===11 12/2 Pi.j
q.
26
and add [KL (p q) to our loss function, where ,3 is a haper--ppraeee peper-parameter and KL, denotes the Kullback- Leibler divergence. The minimization of the KL regularizer may be useful in applications, where conditioning vectors are naturally clustered such as in audio applications involving speaker embedding vectors, such PSE, TTS, and speaker-dependent ASR. Also in applications with discrete latent vectors (Oord et al. 2017). where the preservation of the geometry of the conditioning vectors in the learned activations may offer superior results or better insights into the modelling internals.
A.2.2 IMPACT OF REGULARIZERS
We now investigate the impact of using regularizers for training PSE models. We trained all models with two regularizes (c.f {A22). which are used individually to analyze their unique benefits (as discussed earlier). Tables present the performance of all models by using each regularizer with models trained on two datasets. To show the effectiveness of the regularizers, we compare these results against those in Tables 1-2 for models learned without regularizers.
The results clearly demonstrate the improvement in performance of RNN and TDS models when trained with either of the regularizers. The performance gain is consistent across both datasets, but there is trade off with the impact on clean speech. Moreover, we observe that entropy regularizer outperformed by t-SNE regularizer. This could be due to the fact that t-SNE regularizer less restrictive allowing a wider range of combinations of the basic activations. Whereas, entropy regularizer promotes the selection of only one of the basic activations.
Interestingly, the TDS-RNN models, which performed best without regularizers, get worse perfor- mance with regularizers. This could be due to limited number of filter layers (i.e., only 2, compared with 8 and 42 for RNN and TDS models respectively) present in the model architecture. Given that LA was given only 11 basic activation to learn from, there could be merely 121 (i.e., 11²) unique separations learned by LA. This makes it difficult for these models to successfully differentiate between all unique conditioning vectors which might contain way more separations.
Note, we considered as typer-parameters only the values in a, B {le-5, 1e-4, le-3, le-2). E {1} and used the same learning rates as for training models without regularizers. Due to limited computational budget, we could not perform an exhaustive search these hyper-parameters.
A.2.3 CORRELATION ANALYSIS
To understand the manner in which the models leverage the conditioning vectors together with inputs to predict outputs, we perform a correlation analysis herween distance matrices of softmax values (of LA) and of conditioning vectors. For a random selection of embedding vectors {
1100 computed
the weights ( with j E (1....,10) 100} of the LAs of model. Now, compute the distance
13

---

Published as a conference paper at ICLR 2022
Table 5: SDRi and WER results LibriSpeech (English) when using learned activations (I.A) trained with entropy (a) and t-SNE (18, regularieess.
Regularizers
SDRi
WER
Model
Entropy
t-SNE
Babble
Ambient
Clean
Babble
Ambient
RNN
le-4
[0. 1]
8.53
11.90
8.5
24.5
15.4
TDS
1e-3
[0, 1]
7.27
11.55
8.3
33.0
16.0
TDS RNN
le-5
[0, 1]
7.95
12.14
8.2
28.8
15.2
RNN
0
[le-4,1]
8,46
12.08
8.4
24.3
15.0
TDS
0
[1e-5.
7.42
11.42
8.5
33.0
16.1
TDS-RNN
[le- 1,
8.51
12.50
8.2
25.4
14.9
Table 6: SDRi and WER results VoxForge (Spanish) when using learned activations (LA) trained with entropy (a) and t-SNE (13, o)) regularizers.
Regularizers
SDRi
WER
Model
Entropy
t-SNE
Babble
Ambient
Clean
Babble
Ambient
RNN
le-5
[0, 1]
7.08
9.74
1.6
18.6
5,9
TDS
le-5
(0. 1]
5.85
8.61
2.1
27.0
7.1
TDS RNN
le-5
[0. 1]
5.51
9.34
1.3
33.3
6,8
RNN
(1e-5, 13
7.36
10.12
2.6
16.9
5.9
TDS
0
[1e-2, 1]
5.68
8.62
1.8
29.0
7.1
TDS-RNN
0
(1e-5.1)
6.33
9.92
1.4
25.2
6.5
matrices for zj and 8) which refer as Z and S Here, the distance matrix is computed by taking the pair-wise cosine distances. To get insights about how LAs perform conditioning, we compute the Spearman correlation between vee Z ) and vec(S()) This correlation is computed for all LAs (i.e., 8, 42 and for RNN, TDS, TDS RNN models). This allows to better measure the impact of the conditioning vector across the depth (i.e., position of the LA) in the network.
(1)
In Figure o (bottom
of each sub-figure) present the correlation values for each LA in all three models. We observe there is a large variability in correlation values as the number of layers increases. We do not very strong correlation with specifie LA, which might be due to the fact that conditioning is performed at different stages.
In addition, we also present (shown in top rows of each sub-figure of Figure 6 the distance matrices (1)) (for I which has the highest correlation), as well as Z (in the last column). The overall structure of approximately block diagonal with three blocks, is not well preserved when learning without regularizers given the blocks collapse to two, but well preserved when learning with regularizers. This is especially significant given that conditioning vectors are high dimensional (in this example 256) but the weight vectors in LAs are low dimensional (in this case 11). We also observe that the models with less number of LAs preserve much similar structure between 5() and Z. Given the networks have similar performance, this indicates that as the number of LAs decrease, the networks enforce higher degree of conditioning per LA.
A.3 IMPACT OF ENROLLMENT DATA ON AMBIENT NOISE
Tables 5 and show the performance of various PSE models based on two seconds of enrollment data in terms of babble and ambient noise. Although babble noise, i.e., other people talking, cannot be expected to be suppressed without knowledge of the owner's voice profile, clean and ambient noise on the other hand does not in principle require such enrollment data. To investigate the performance of the various PSE models in the absence of enrolment data have replaced with vectors of random, zeros and ones, and measured the quality of PSE on ambient noise in Tables 7-8, which demonstrate that all architectures work well (compared concatenation based conditioning approach) even without enrolment data for clean and ambient noise when using ones instead of conditioning vectors. We also see that for the best architecture (TDS-RNN) that is the case not just with replacing by ones but also by random and zeros.
14

---

Published as a conference paper at ICLR 2022
RNN
TDS
TDS ANN
RNN
TDS
TDS RMN
14
14
0.
03
0.1
21 213644
00
21 264
Layer
Layer
trained regularizers
(b) Model
1-SNE regularizer
(c) Embd. vector
Figure 6: Correlation analysis of conditioning vectors with softmax values of LA for models trained on Librispeech. sub-figure (a) and (b), the bottom row shows the overall correlation all LAs in model with conditioning vectors, and the top row depicts the distance matrix corresponding to LA with the highest overall correlation for each model well for conditioning vectors (fig c).
Table Results for models trained on Librispeech (English) data with proposed LA and baseline concatenation and evaluated with random, zeros and ones speaker embedding vectors.
SDRi
WER
Model
LA
SE
Babble
Ambient
Clean
Babble
Ambient
Identity
8.2
89.4
17.3
RNN
LA
Random
3,08
9.24
RNN
LA
Zeros
0.47
8.41
25.9
60.2
21.8
RNN
LA
Ones
4.61
11.64
9.7
44.4
15.6
RNN
Concat.
Random
1.17
6.32
28.2
55.2
20.0
RNN
Concat.
Zeros
3.70
10.76
20.2
43.8
17.3
RNN
Concat.
Ones
2.20
11.44
$1.1
TDS
LA
Random
-3.48
4.36
11.3
79.6
19.7
TDS
LA
Zeros
2.65
4.58
14.4
85.5
25.5
TDS
LA
Ones
-0.59
10.59
8.4
74.0
16.1
TDS
Concat.
Random
-2.90
5.63
32.7
74.6
24.1
TDS
Concat.
Zeros
-0.68
8.38
30.9
68.5
21.7
TDS
Concat.
Ones
-1.13
9.06
24.8
19.5
TDS IDS-RNN
RNN
LA
Random
1.99
12.23
$2.2
14.4
TDS RNN
LA
Zeros
1.91
12.19
17.0
52.8
14.5
TDS-RNN
LA
Ones
3,90
12.33
9.5
44.8
14.4
TDS-RNN
Concat.
Random
1.79
7.42
58.2
46.1
21.0
TDS-RNN
Concat.
Zeros
7.33
11.84
25.4
25.6
15.2
TDS-RNN
Concat.
Ones
3.66
11.75
23.1
15.4
15

---

Published as a conference paper at ICLR 2022
Table 8: Results for models trained on VoxForge (Spanish) data with proposed LA and baseline concatenation and evaluated with random, zeros and ones speaker embedding vectors.
SDRi
WER
Model
LA
SE
Babble
Ambient
Clean
Babble
Ambient
Identity
1.2
89.1
9.2
RNN
LA
Random
-3.23
-3.00
76.6
77.8
52.7
RNN
LA
Zeros
-1.60
2.45
48.7
64.9
33.3
RNN
LA
Ones
-3.53
1.37
61.5
69.7
37.7
RNN
Concat.
Random
-1.77
-0.44
56.9
73.6
41.0
RNN
Concat.
Zeros
-1.26
2.34
42.9
63.4
30.7
RNN
Concat.
Ones
-2.09
1.02
57.9
67.8
39.2
TDS
LA
Random
-3.91
-3.21
49.8
85.8
TDS
LA
Zeros
-3.91
-3.21
49.8
85.8
40.3
TDS
LA
Ones
-3.84
0.18
33.4
78.7
30.4
TDS
Concat.
Random
-3.80
-2.48
34.1
84.0
33,6
TDS
Concat.
Zeros
-4.05
-1.06
39.9
39,5
TDS
Concat.
Ones
-4066
-1.51
47.2
78.1
42.2
IDS-RNN
LA
Random
-1.07
8.37
13.0
56.8
10.3
TDS-RNN
LA
Zeros
-1.13
8.26
14.0
56.9
10.9
IDS-RNN
LA
Ones
-0.85
8.58
10.7
56.4
9.4
TDS-RNN
Concat.
Random
-0.10
0.88
86.6
56.3
39,9
TDS RNN
Concat.
Zeros
1.59
6.09
56.2
42.7
19,9
TDS RNN
Concat.
Ones
0.47
3.75
66.3
51.1
31.7
16


Reviews:
[{"review_id": "vRpYx_1PG0_", "reviewer": "Reviewer_gotN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "See my comments above in the Summary for strengths and weaknesses. In an nutshell, I think the paper suffers from a lack of clarity that undersells what is otherwise a nice idea.\n\nI suggest introducing the specifics of the core of the idea earlier on in the paper. There's a sense of anticipation that is a bit frustrating for me as a reader.\n\nSome more specific comments follow.\n\nRe: sentence ending in, \"... and a family of `a` basic activations {Ai : R → R} a i=1 of which a particular realization could be the set of the most commonly used activations in deep learning (e.g. relu, sigmoid, tanh, and so on)\": check the grammar, I believe this is not a complete sentence given what comes before the sub-clause I just quoted.\n\nWhat is \"softmax_rowwise\"?\n\nRe: First 3 equations in Section 3.1: What is the relationship between LA_elementwize(c | z_j) and LA(h_j | z_j)? (Numbering the equations would be helpful). What is c?\n\nI did not quite follow the formalism. I am unable to clearly relate the equations in 3.1 with Figure 1c.\n\nA number of references are given for the concatenation approach, which is good. But no references are given for the modulation approach (AFAICT), though this is referred to as a state-of-the-art approach. Here again, I am unable to follow the discussion linking the formalism to Figure 1b. I am guessing that the formalism is correct but the presentation could be substantially simplified and clarified.\n\nFigures 2: \"To take into account that basic activations may have different ranges, the plots show LAs minus average of basic activations, i.e., LA(c | zj )  [...] for c ∈ [−3, 3] and values of j corresponding to the selected users.\" I cannot relate LA(c | zj) to the label of the x-axis, which is h.\n\nFigure  4. \"To take into account that the non-personalized basic activations are relu or swish, the plots show LAs minus average of those two activations, i.e., LA(c | zj ) − (relu(z) + swish(z))/2 for c ∈ [−3, 3] and values of j corresponding to the selected users. \" I suggest labeling the x-axis. Would that be z? How should we relate z_j to the z (without the j subscript) argument of relu() and swish()?\n\nSection 4.1, Table 1: Define SDRi before using the term.\n\nSection 4.2, \"HPO\": \"Hyper-parameter optimization\"?\n\nSection 4.4, \"STFT\": define the term.\n\nSection 5.4, \"Greedy search\": provide a reference? Later, in the results section, the authors state, \"Furthermore, the gains can be seen to be more pronounced for greedy than beam search CTC decoders, reaching up to 40% relative improvement for some users which is a significant improvement in the domain of ASR,\" but greedy search doesn't represent the typical final ASR benchmark, so the greater relative gain than for beam search doesn't seem too significant -- but I am not sure what the authors mean by greedy search, so they should clarify this.", "summary_of_the_paper": "This paper presents a very interesting idea (if I understand it correctly), which is to implement conditioning on e.g. a speaker embedding vector via a parametrization of the layer-to-layer activation functions of an overall deep neural net architecture. The motivation is to achieve the same effect as the well-known concatenation approach, but without requiring as many new parameters.\n\nI found the formalism hard to follow, impeding my grasp of the fundamental proposal. However, eventually I got the idea. The results are promising, showing good preservation of e.g. WER quality on the unconditioned scenario, with good WER on the specialized scenario, and with fewer parameters than the concatenation method.\n\nI think the name \"learned activations\" somewhat undersells the idea -- wouldn't it be better named as \"learned activation _functions_\"? That immediately would clarify the fundamental concept -- and highlight it's originality.", "summary_of_the_review": "Good originality (AFAICT) and good results; somewhat unclear formalism and presentation of main concepts.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oWVdFbeYffK", "reviewer": "Reviewer_dsMi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Extensive experiments on two application areas (speech enhancement and automatic speech recognition). \n- The paper also tried to provide quantitative results by plotting learned activations for different speakers. \n- The proposed model is comparably smaller than the concatenation based approaches. The method might be suitable for on-device applications.  \n\nWeaknesses:\n- As there are many experimental results, the paper may occasionally become harder to read towards the end.\n- Some details are not well-described. For example, in Tables 1 and 2, what does #Cond. mean?  \n- Even though the model size is smaller, it comes with additional on-the-fly computation of several non-linearities of hidden activations, which may cause other concerns for on-device applications. \n- FiLM approach should be briefly mentioned in the text before reporting the results with this method. \n- HPO acronym should be defined. (Page 6)\n- How exactly does the study perform fine-tuning of the speaker embedding model? Details are missing. (Page 8) \n- For PASR, does the study fine-tune the speaker embedding model jointly with the ASR model or are they learned separately?", "summary_of_the_paper": "This paper introduces a method called learned activations for personalized speech enhancement and personalized automatic speech recognition. The learned activations are obtained by a weighted sum of nonlinear activations of hidden layer outputs and the weights are computed from softmax of the projected speaker embeddings such as x-vectors. The main benefit of the model as compared to concatenation or modulation based approaches is that Learned Activations result in smaller number of learnable parameters and hence a smaller model size. Experimental results show that in terms of SDR improvement, the proposed method achieves comparable performance to the baseline models with a smaller model size. In the case of ASR, the proposed approach provides WER reduction as compared to an unadapted model. \n", "summary_of_the_review": "The paper provides a personalization method by conditioning non-linear activations based on the speaker embeddings. There are extensive experiments. One concern is that the paper may need some additional clarification on details for better reproducibility.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XMkjo8a921", "reviewer": "Reviewer_QdFv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The experimental results indicate that the proposed method of training the RNN / CNN based systems for speech enhancement and ASR leads to reduction in number of parameters. The authors argue that this is an important advantage for low-resource computing. I have several questions on this. I have reviewed this paper before, and I see that some of my concerns from earlier, even though were partially adressed in the rebuttal, now seems not to be addressed in the current version of the manuscript.\n\nThe number of parameters to assess the computational advantages of a model is questionable in my opinion. I think a better measure for this would be to use the number of FLOPs in forward pass, or the memory usage with respect to input sequence length. Namely, this table was presented in the rebuttal: \n\nModel \tLA \tFLOPs (M) \tLatency (ms)\n\nRNN \ty \t1254.85 \t528.543 (+-36.392)\n\nRNN \tn \t1802.60 \t537.766 (+-11.403)\n\nTDS \ty \t1507.43 \t116.451 (+-06.992)\n\nTDS \tn \t2906.46 \t123.580 (+-03.440)\n\nTDS--RNN \ty \t4406.60 \t392.326 (+-15.582)\n\nTDS--RNN \tn \t4543.92 \t394.809 (+-20.247)\n\nI think it would be good to include these results. Overall, we see that the improvement is not always clear, especially in terms of latency, when the variance is taken into consideration. \n\nFor the new experiment with ASR, and PASR, (part 2 of the experiments) the presentation is not super clear to me, and I am not sure if an improvement over alternative conditioning techniques are shown here. ", "summary_of_the_paper": "This paper proposes a way of conditioning information on neural networks. In literature a common way to condition a neural with an input would be to either concatenate the conditioning vector to the input vector, or inject it before several layers (modulation approach in Figure 1-b). In this paper they instead propose to pass the conditioning vector through weighted sum of output of neural-net activation functions. The claim is that this way of training the neural network leads to reduction in neural network parameters without sacrificing performance in speech enhancement / ASR tasks.", "summary_of_the_review": "I was a reviewer for this paper for Neurips 2021. Even though I voted for acceptance that time, I now see that some of my concerns and concerns pointed out by other reviewers are not fully addressed. For instance I do not see a comparison with this paper: https://arxiv.org/pdf/1601.02828.pdf  . I also do not see a comparison in terms of # FLOPS and latency for which the improvement seems marginal from the results added to the Neurips rebuttal. \n\nI therefore vote for marginal rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel method for conditioning deep neural network. They replace the activation function with a linear combination of activation functions (e.g., ReLu). The weights for the activation functions are dynamically computed from the input during inference and training. The approach is evaluated on standard public tasks and shows improvement over well-established alternatives.

Pros
+ A simple novel method for condition that is widely applicable
+ Adequate empirical evaluations to demonstrate it's effectiveness

Cons
- No major weakness

The reviewers provided several feedback. The authors incorporated the suggestions and clarified residual concerns. The revised version of the paper has improved the readability and utility substantially.
Decision: Accept (Poster)