SlowGuess/NAACL_2021_OCR huggingface dataset:
We follow a leave-one-speaker-out evaluation scheme and report the average performance across all test speakers on all four downstream tasks. For each test speaker, we pick the model that gives the best performance on a held-out validation set. The hyper-parameters that we optimize on the validation set include the number of hidden layers  $\{1,2,3\}$ , the width of each hidden layer  $\{64,128,256\}$ , and the activation unit  $\{\mathrm{Tanh}, \mathrm{ReLU}\}$ . We construct the networks in PyTorch and train them with batches of size 32 using the ADAM optimizer with learning rate of  $10^{-4}$  and a cross-entropy loss function. We train each model for a maximum of 100 epochs and apply early stopping if the validation loss does not improve for five consecutive epochs. We repeat each experiment with 30 different random seeds and report the average and standard deviation to account for performance fluctuation due to random initialization and training.
In this section, we provide the results of our four experiments (Section 5.1).
Is the proposed framework capable of inserting expressiveness into synthetic speech? Table 1 shows that we obtain an MCD of 24.01 and an
Table 2: Performance obtained using different features for emotion recognition and speaking style classification. Emotion recognition performance is measured using the unweighted average recall (UAR) while speaking style detection performance is measured using accuracy. Performance is evaluated using a leave-one-speaker-out scheme and the numbers reported are averages ( $\pm 1$  standard deviation) from 30 runs to account for randomness in initialization and training. * indicates that the marked performance is significantly higher than MFBs.  $\dagger$  indicates that the marked performance is significantly higher than best APC model. Significance is assessed at  $p < 0.05$  using the Tukey's honest test on the ANOVA statistics.
Features	IEMOCAP	Emotion (UAR) MSP-IMPROV	VESUS	Style (Accuracy) IEMOCAP
Baseline - Surface Features
Chance	25.0	25.0	20.0	52.3
MFBs	53.0 ± 0.6	43.6 ± 1.2	36.0 ± 1.4	67.0 ± 0.7
Baseline - Unsupervised
Autoencoder	50.6 ± 0.9	38.7 ± 1.0	33.6 ± 1.1	64.2 ± 0.6
APC (5-steps)	51.7 ± 0.8	42.2 ± 0.8	33.5 ± 1.2	68.3 ± 0.6
APC (10-steps)	53.9 ± 0.9	44.6 ± 0.9	35.5 ± 1.6	69.7 ± 0.6
APC (20-steps)	54.3 ± 0.9	44.1 ± 0.9	36.1 ± 1.5	69.7 ± 0.6
Paralinguistic Embeddings (ours)
Base EVoCA	56.4 ± 0.6*†	46.0 ± 0.6*†	44.2 ± 0.9*†	71.7 ± 0.5*†
w/ 128-dim para. enc.	55.4 ± 0.8*†	45.3 ± 0.9*	42.6 ± 1.4*†	69.6 ± 0.5*
w/ 64-dim para. enc.	53.0 ± 0.6	42.9 ± 0.8	38.2 ± 0.9*†	67.2 ± 0.5
w/ 32-dim para. enc.	51.7 ± 0.6	41.0 ± 0.4	36.0 ± 1.3	65.7 ± 0.5
w/ 128-dim converter	57.1 ± 0.5*†	46.3 ± 0.9*†	43.5 ± 1.3*†	70.4 ± 0.5*†
w/ 64-dim converter	57.0 ± 0.7*†	44.9 ± 0.9*	41.0 ± 0.9*†	69.6 ± 0.6*
w/ 32-dim converter	54.9 ± 0.6*	44.6 ± 0.7*	38.1 ± 1.0*†	68.8 ± 0.5*
F0 RMSE of 146.20 when computing the performance using the synthetic reference speech and ground-truth expressive speech. In comparison, we obtain an MCD of 10.71 and an F0 RMSE of 64.36 when computing the performance using the converted speech and the ground-truth expressive speech. This suggests that the proposed EVoCA framework converts the synthetic speech so that its closer to the expressive speech. We note that it is possible to obtain better conversion performance if we increase the capacity of the model and utilize a more sophisticated vocoder. However, as the results for question 3 will suggest, increasing the capacity of the voice converter might not necessarily yield better embeddings for downstream classification tasks.
Can the learned paralinguistic embeddings be used for emotion and style classification? Table 2 shows that our paralinguistic embeddings significantly outperform MFB surface features on both the emotion recognition and the speaking style
detection tasks. This suggests that the paralinguistic encoder learns a feature transformation that highlights latent paralinguistic attributes in surface acoustic features.
How do changes to EVoCA's structure affect the converted speech quality as well as the quality of the extracted embeddings for downstream tasks? Figure 3 visually demonstrates the effect of each input on the quality of a converted utterance. Figure 3a shows that the converted speech has higher quality when the paralinguistic embedding is provided as an input compared to Figure 3b. Specifically, the harmonic structure in Figure 3a is well defined and dynamic while that in Figure 3b is relatively static and not well separated. Figure 3c shows that the model is unable to generate speech solely from paralinguistic embeddings. We hypothesize that this is due to the embeddings' limited capacity to encode both linguistic and paralinguistic information present in the original signal to allow for accurate reconstruction.
(a) all components
(b) w/o paralinguistic embedding
(c) w/o synthetic reference
Figure 3: Sample converted test utterance with three model setups.
Additionally, we believe paralinguistic embeddings struggle to model time-varying phenomena like rhythm and speech activity because they are computed using a global average over LSTM outputs.
Table 1 quantitatively shows the effect of each of these two inputs on the conversion performance. We find that the synthesized reference input is more important to the conversion task than the paralinguistic embedding is. This is highlighted by the larger impact that reducing the capacity of the voice converter has on the converted speech quality compared to the impact of reducing the capacity of the of the paralinguistic encoder. This can be due to the fact that the paralinguistic embeddings do not have enough capacity to encode the linguistic attributes in speech that are necessary for obtaining good voice conversion performance.
Tables 1 and 2 show the results obtained on the voice conversion task and the downstream classification tasks, respectively. We find that while a high capacity voice converter improves the quality of the converted speech, it can also degrade the quality of the extracted embeddings as measured on the classification tasks. For instance, we find that reducing the capacity of the voice converter from 256 to 128 decreases the conversion performance on the voice conversion task but improves the classification performance on two out of the four downstream tasks. The results suggest that using a high-capacity voice converter can reduce EVoCA's reliance on the paralinguistic encoder for providing style and emotion information, causing the encoder to perform poorly when used to transform features for downstream applications.
How does the performance of paralinguistic embeddings compare to the embeddings learned from other unsupervised methods? Table 2 shows that paralinguistic embeddings encode information that is more suited to paralinguistic tasks than those extracted from other unsupervised meth
ods, namely APC and a traditional autoencoder. The APC model provides improvements over surface features on all four downstream tasks when using the 20-step setup and shows improvements over surface features on three downstream tasks when using the 10-step setup. In contrast, a standard autoencoder fails to provide any improvements over surface features on all tasks. We believe that the success of the extracted embeddings from EVoCA demonstrate the importance of targeted unsupervised tasks.
We proposed EVoCA, a framework for learning a surface feature transformation that highlights paralinguistic content needed for detecting emotion and speaking style. We first showed that speech synthesizers can be used to strip away paralinguistic attributes from speech while retaining linguistic information. We demonstrated how a neural voice conversion model can be adapted to facilitate the extraction of paralinguistic features by converting synthetic neutral speech to real expressive speech. Finally, we showed that these extracted embeddings improve performance over surface features and can outperform other embeddings extracted from existing unsupervised methods on emotion recognition and speaking style detection tasks. Future work will consider how the choice of the synthesis model, the number of speakers in the training set, and the architecture used for the encoder affect the quality of the extracted embeddings.
Potential Benefits. A variety of applications can benefit from the automatic detection of paralinguistic attributes (e.g., emotion) from speech; some of these applications include: human-robot interaction, medical applications, and speaker verification to name a few. The framework that we introduce
can impact these applications by enabling the utilization of data that are not labeled for paralinguistic attributes when building the detection models for these domains.
Potential Risks. The behavior and performance of all data-driven models heavily depend on the data that are used for building them. Thus, the decisions that these models make will reflect any biases that exist in the data. Some attributes that can bias speech data include: age, gender, dialect, accent, language, recording conditions, and environment. We encourage the deployment of our framework with full consideration of these biases and their consequences on the target application.
Jacob Benesty, M Mohan Sondhi, and Yiteng Huang. 2007. Springer Handbook of Speech Processing. Springer.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42(4):335.
Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. 2016. MSP-IMPROV: An acted corpus of dyadic interactions to study emotion perception. IEEE Transactions on Affective Computing, 8(1):67-80.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James R Glass. 2019. An unsupervised autoregressive model for speech representation learning. In Proc. Interspeech 2019, pages 146-150.
Nicholas Cummins, Stefan Scherer, Jarek Krajewski, Sebastian Schnieder, Julien Epps, and Thomas F Quatieri. 2015. A review of depression and suicide risk assessment using speech analysis. Speech Communication, 71:10-49.
Sefik Emre Eskimez, Zhiyao Duan, and Wendi Heinzelman. 2018. Unsupervised learning approach to feature analysis for automatic speech emotion recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5099-5103.
Florian Eyben, Klaus R Scherer, Björn W Schuller, Johan Sundberg, Elisabeth André, Carlos Busso, Laurence Y Devillers, Julien Epps, Petri Laukka, Shrikanth S Narayanan, et al. 2015. The Geneva Minimalistic Acoustic Parameter Set (GeMAPS) for voice research and affective computing. IEEE Transactions on Affective Computing, 7(2):190-202.
Jian Gao, Deep Chakraborty, Hamidou Tembine, and Olaitan Olaleye. 2019. Nonparallel Emotional Speech Conversion. In Proc. Interspeech 2019, pages 2858-2862.
Wei-Ning Hsu, Yu Zhang, and James Glass. 2017. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances In Neural Information Processing Systems, pages 1878-1889.
Zahi N Karam, Emily Mower Provost, Satinder Singh, Jennifer Montgomery, Christopher Archer, Gloria Harrington, and Melvin G Mcinnis. 2014. Ecologically valid long-term mood monitoring of individuals with bipolar disorder using speech. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4858-4862.
Reza Lotfian and Carlos Busso. 2015. Emotion recognition using synthetic speech as neutral reference. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4759-4763.
Reza Lotfian and Carlos Busso. 2017. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 10(4):471-483.
Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Ni- eto. 2015. Librosa: Audio and music signal analysis in python. In Proceedings of the 14th Python in Science Conference, volume 8, pages 18-25.
Seyed Hamidreza Mohammadi and Alexander Kain. 2017. An overview of voice conversion systems. Speech Communication, 88:65-82.
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa. 2016. WORLD: a vocoder-based high-quality speech synthesis system for real-time applications. IEICE Transactions on Information and Systems, 99(7):1877-1884.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A large-scale speaker identification dataset. Proc. Interspeech 2017, pages 2616-2620.
Andrew Rosenberg. 2012. Classifying skewed data: Importance weighting to optimize average recall. In Thirteenth Annual Conference of the International Speech Communication Association.
Jacob Sager, Ravi Shankar, Jacob Reinhold, and Archana Venkataraman. 2019. VESUS: A crowd-annotated database to study emotion production and perception in spoken english. In Proc. Interspeech 2019, pages 316-320.
Björn Schuller and Anton Batliner. 2013. Computational Paralinguistics: Emotion, Affect and Personality in Speech and Language Processing, 1st edition. Wiley Publishing.
Björn Schuller, Stefan Steidl, and Anton Batliner. 2009. The Interspeech 2009 emotion challenge. In Tenth Annual Conference of the International Speech Communication Association.
Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian Müller, and Shrikanth S Narayanan. 2010. The Interspeech 2010 paralinguistic challenge. In Eleventh Annual Conference of the International Speech Communication Association.
Björn Schuller, Stefan Steidl, Anton Batliner, Florian Schiel, and Jarek Krajewski. 2011. The interspeech 2011 speaker state challenge. In Twelfth Annual Conference of the International Speech Communication Association.
Björn Schuller, Stefan Steidl, Anton Batliner, Alessandro Vinciarelli, Klaus Scherer, Fabien Ringeval, Mohamed Chetouani, Felix Weninger, Florian Eyben, Erik Marchi, et al. 2013. The interspeech 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism. In Proc. Interspeech 2013.
Ravi Shankar, Hsi-Wei Hsieh, Nicolas Charon, and Archana Venkataraman. 2019a. Automated emotion morphing in speech based on diffeomorphic curve registration and highway networks. In Proc. Interspeech 2019, pages 4499-4503.
Ravi Shankar, Jacob Sager, and Archana Venkataraman. 2019b. A multi-speaker emotion morphing model using highway networks and maximum likelihood objective. In Proc. Interspeech 2019, pages 2848-2852.
David Snyder, Daniel Garcia-Romero, Gregory Sell, Daniel Povey, and Sanjeev Khudanpur. 2018. X-vectors: Robust dnn embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5329-5333.
Alessandro Vinciarelli, Anna Esposito, Elisabeth Andre, Francesca Bonin, Mohamed Chetouani, Jeffrey F Cohn, Marco Cristani, Ferdinand Fuhrmann, Elmer Gilmartin, Zakia Hammal, et al. 2015. Open challenges in modelling, analysis and synthesis of human behaviour in human-human and human-machine interactions. Cognitive Computation, 7(4):397-413.
Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. 2018. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning, pages 5180-5189. PMLR.
Jing-Xuan Zhang, Zhen-Hua Ling, and Li-Rong Dai. 2019a. Non-parallel sequence-to-sequence voice conversion with disentangled linguistic and speaker representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:540-552.
Jing-Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, and Li-Rong Dai. 2019b. Sequence-to-sequence acoustic modeling for voice conversion. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(3):631-644.
Ya-Jie Zhang, Shifeng Pan, Lei He, and Zhen-Hua Ling. 2019c. Learning latent representations for style control and transfer in end-to-end speech synthesis. In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6945-6949.