Conditioning Sequence-to-sequence Networks with Learned Activations

Conditional neural networks play an important role in a number of sequenceto-sequence modeling tasks , including personalized sound enhancement ( PSE ) , speaker dependent automatic speech recognition ( ASR ) , and generative modeling such as text-to-speech synthesis . In conditional neural networks , the output of a model is often influenced by a conditioning vector , in addition to the input . Common approaches of conditioning include input concatenation or modulation with the conditioning vector , which comes at a cost of increased model size . In this work , we introduce a novel approach of neural network conditioning by learning intermediate layer activations based on the conditioning vector . We systematically explore and show that learned activation functions can produce conditional models with comparable or better quality , while decreasing model sizes , thus making them ideal candidates for resource-efficient on-device deployment . As exemplary target use-cases we consider ( i ) the task of PSE as a pre-processing technique for improving telephony or pre-trained ASR performance under noise , and ( ii ) personalized ASR in single speaker scenarios . We find that conditioning via activation function learning is an effective modeling strategy , suggesting a broad applicability of the proposed technique across a number of application domains . 1 INTRODUCTION . The use of latent or embedding vectors to condition neural networks is common in a number of machine learning tasks . For example , in personalized sound enhancement ( PSE ) prior knowledge of a speaker ’ s voice profile is used to filter out noise from an audio recording ( Choi et al. , 2021 ; Hu et al. , 2020 ; Wang et al. , 2020 ; 2019 ) , in speaker-dependent or personalized Automatic Speech Recognition ( ASR ) a speaker ’ s voice profile is also used to improve the overall transcription quality ( Bell et al. , 2021 ; He et al. , 2019 ; Samarakoon et al. , 2016 ; Saon et al. , 2013 ) , in personalized Text-To-Speech ( TTS ) systems synthesized voice profile can be purposefully adapted ( van den Oord et al. , 2018 ) , in computer vision tasks , such as in visual reasoning applications ( Chen et al. , 2019 ; Perez et al. , 2018 ) , questions are used as the conditioning vectors , and similarly in conditional domain adaptation techniques ( Long et al. , 2018 ) . More generally , conditional neural networks accept an additional conditioning vector , that influences how the networks process inputs to produce outputs . Most common approaches for conditioning neural networks either concatenate or modulate the conditioning vector with one or more intermediate representations of the network , before the updated representations are processed by a layer , e.g. , see Figure 1a-b for an overview . Such conditioning techniques come at the cost of significantly increasing the network parameters by inflating the dimensionality of intermediate features or by requiring additional network parameters proportional to the dimensionality of the conditioning vectors and the number of channels present in the representations . In this work , we introduce a new approach for neural network conditioning and show that effective conditioning can be achieved by learning layer-specific activation functions based on the conditioning vectors . Unlike previous approaches , our learned activation ( LA ) functions approach does not increase the model parameters significantly , since the number of additional parameters it requires depends only on the size of the conditioning vector , and not on the shape of the feature representations nor on the output shapes of any subsequent layers . LA uses the conditioning vectors to dynamically modify intermediate activation functions , which are R→ R ( 1-D ) mappings , see Figure 1c . Dynamic LAs allow us to extract important insights on how the model is being adapted for different conditioning vectors by simply analyzing the learned activation functions , which are agnostic to the input . In this work we consider two popular and timely sequence-to-sequence modeling tasks : personalized sound enhancement ( PSE ) and personalized ASR ( PASR ) as exemplary tasks for modeling conditional neural networks . PSE leverages a speaker embedding vector , i.e. , a unique encoded voice profile of a speaker , for filtering out everything but the speaker ’ s voice from an audio recording corrupted with babble or ambient noise ( Wang et al. , 2020 ; 2019 ) . Speaker embeddings can generally be obtained by using an embedding network once during an enrolment phase , e.g. , from recorded utterances like “ Hey Alexa ” or “ Hey Bixby ” . PSE networks can be evaluated for improving signal quality for telephony or can also be used to improve the quality of downstream tasks , such as a pre-trained ASR . PASR on the other hand , aims to improve ASR quality in single speaker scenarios , by leveraging user ’ s enrollment data to enhance performance directly without any pre-processing . We evaluate LA on three PSE model families ( §4.2 ) , on two datasets Librispeech ( Panayotov et al. , 2015 ) and Voxforge ( vox , 2006 ) . We show that the use of LA on all models achieves competitive performance to conditioning using concatenation and modulation approaches . At the same time , concatenation approaches require 10–95 % more model parameters over our solution . For PASR , we compare the performance of non-personalized and personalized versions of the conformer encoder model architecture ( Gulati et al. , 2020 ) with a dense layer for character or subword text encoder trained with CTC loss and decoded via CTC decoder ( §5.2 ) . Results on Librispeech ( Panayotov et al. , 2015 ) show that LA yields effective ASR personalization , i.e. , increased performance across all users while using two seconds of enrollment data , with up to 19 % relative WER improvement across all users , and up to 40 % for some users . 2 RELATED WORK . Conditioning through tensor concatenation The concatenation of intermediate features with latent vectors before being passed to subsequent layers is the most common form of conditioning , but leads to inflated model sizes , which hinder their deployment to low-resource constrained devices ( Wang et al. , 2019 ; 2020 ; Valin et al. , 2020 ; Hu et al. , 2020 ; Comon & Jutten , 2010 ; Wang et al. , 2021 ; Lam et al. , 2021 ; Vincent et al. , 2018 ; Luo & Mesgarani , 2019 ; Xu et al. , 2020 ; Abdulatif et al. , 2020 ; Wisdom et al. , 2020 ; Isik et al. , 2020 ; Luo et al. , 2020 ; Choi et al. , 2021 ; Snyder et al. , 2018 ; Panayotov et al. , 2015 ; Bell et al. , 2021 ; He et al. , 2019 ; van den Oord et al. , 2018 ) . Our approach overcomes this issue and learns smaller models with competitive quality . Conditioning via normalization layers Another family of techniques is based on having per channel learned linear transformation based on the conditioning vector , which yields a form of conditional normalization ( Perez et al. , 2018 ; de Vries et al. , 2017 ; Huang & Belongie , 2017 ) . Our approach distinguishes itself from this line of work since it learns the non-linear components of the network rather than the linear ones , and thus offers greater modelling at lower cost . Learning activations without conditioning Outside the context of conditional neural networks , there is work on learning an optimal activation function as a good default to be used in all layers of most neural network architectures ( Apicella et al. , 2021 ; Liu et al. , 2020 ; Ramachandran et al. , 2018 ; Agostinelli et al. , 2015 ) . Our work instead i ) learns conditional activations based on conditioning vectors , and ii ) uses differently learnt activations throughout the network . 3 CONDITIONING BY LEARNING ACTIVATIONS . Conditioning a neural network corresponds to using some form of latent vector or additional prior knowledge , e.g. , encoded typically in the form of an embedding vector , to influence the output of the network for a given input . We show in the evaluation that the influence of the conditioning vector can be successfully modeled by learning a weighted combination of a set of basic activation functions ( §3.1 ) and carefully selecting hyperparameters ( §3.2 ) . 3.1 DYNAMICALLY LEARNED ACTIVATIONS . To introduce our conditioning approach using dynamically weighted activation functions , we begin by defining a few key variables : z ∈ Rb×d is a 2-D tensor representing b number of d-dimensional conditioning vectors , where b is the batch size , x ∈ Rb×··· is the input data with arbitrary tensor dimensions ( e.g. , vectors , matrices , higher order tensors ) , h ∈ Rb×··· is the hidden pre-activation of the previous layer , and { Ai : R → R } ai=1 is an ordered family of a basic activations , e.g. , { relu , sigmoid , tanh } ( we will discuss below more on the elements of this ordered set of activations ) . Dynamically learned activation ( LA ) functions require a small number of trainable variables w ∈ Rd×a and b ∈ R1×a , and are defined by transforming input tuples ( z , h ) into output tensors y through ( note the parallel between the following equations and the pictorial representation in Figure 1c , where there is a direct correspondence on the roles of z , h and y between the formulas and the diagram ) : s : = softmaxrowwise ( z ×w + b ) , h ∈ R 7→ LA ( h | zj ) : = a∑ i=1 sj , i ·Ai ( h ) , yj , ... : = LAelementwise ( hj , ... | zj ) , where softmaxrowwise denotes the matrix-matrix operation with entries defined rowwise via softmaxrowwise ( m ) j : = softmax ( mj ) , and LAelementwise ( hj , ... | zj ) denotes the result of applying LA ( · |zj ) elementwise to each entry of hj , .... Note that during training , batch size is greater than one in which case z is a matrix , whereas during inference batch size is one for each user in which case z is a row vector . So during training zj is the conditioning vector corresponding to the j-th sample , but during inference zj is the conditioning vector corresponding to the j-th user in the discussion below . Figure 1 highlights the qualitative and quantitative differences between our approach ( Figure 1c ) and existing state-of-the-art ( SOTA ) solutions ( Figure 1a-b ) . Concatenation approaches as depicted in Fig . 1a can be defined by Aelementwise ( Fθ ( [ z > j , x > j , ... , : ] > ) ) , where Fθ denotes a common learnable transformation such as a dense , conv1d , conv2d or LSTM layer without activation and A : R→ R denotes a generic activation such as the rectified linear unit . Whereas , as shown in Fig . 1b , modulation approaches are defined as Aelementwise ( Gη ( Uµ ( zj ) xj , ... , : + Vν ( zj ) ) ) , where Gη represents a common learnable transformation ( which may be slightly different than Fθ given their inputs have different shapes ) and Uµ : Rd → Rf and Vν : Rd → Rf are often taken as linear mappings ( where d denotes the conditioning vectors dimension and f denotes the number of channels of x ) . In comparison to these SOTA approaches , our learned activation functions mechanism can simply be formulated as LAelementwise ( Gη ( xj , ... ) | zj ) . For simplicity , we use a dense layer in the figure ( i.e. , we take Fθ and Gη to be dense layers and f = 1 ) , however , our approach is general and can be applied to any layer type , e.g , dense , conv1d , conv2d and LSTM . From a qualitative point of view , LAs conditions the non-linearities rather than the linearities of the network . From a quantitative point of view , Figure 1 also highlights the superior compression abilities achieved by our proposed method . In particular , concatenation approaches ( Figure 1a ) require ( d+ u ) ∗ v parameters and modulation approaches ( Figure 1b ) require u ∗ v + d ∗ 2 ∗ f , where f denotes the number of channels in the input tensor x , which per channel is multiplied by a scaling factor and offset by a bias term . The learned activation functions approach on the other hand , requires only u ∗ v + d ∗ a parameters . For example , in most audio and video applications , our approach yields significant savings of d ∗ ( v − a ) given that d 1 and v a , when compared to concatenation approaches . The saving becomes d∗ ( 2∗f −a ) when compared to modulation approaches , given that often in such cases 2∗f −a > 0 . For temporal data , such as audio and video , the savings of our approach are even more significant as in those applications the conditioning vector is concatenated with each time frame of the inputs and for f 1 the model requires larger parameter size . In our approach the operational requirements do not change across data type , i.e. , tensor dimensions .

Summary: This paper presents a very interesting idea (if I understand it correctly), which is to implement conditioning on e.g. a speaker embedding vector via a parametrization of the layer-to-layer activation functions of an overall deep neural net architecture. The motivation is to achieve the same effect as the well-known concatenation approach, but without requiring as many new parameters.  I found the formalism hard to follow, impeding my grasp of the fundamental proposal. However, eventually I got the idea. The results are promising, showing good preservation of e.g. WER quality on the unconditioned scenario, with good WER on the specialized scenario, and with fewer parameters than the concatenation method.  I think the name "learned activations" somewhat undersells the idea -- wouldn't it be better named as "learned activation _functions_"? That immediately would clarify the fundamental concept -- and highlight it's originality.