Insights from Interspeech 2K20 — A Letter to Prospective, Busy & Junior (PB&J) research students
By: Sai Krishna Rallabandi
Tags: Speech Processing, Machine Learning, Deep Learning, Interspeech, Self Supervised Learning

Dear PB&J,

I vividly remember a question from ISCA General Assembly Q/A session last Interspeech: about making Interspeech remote. I recall instantly thinking about you. Coz attending my first Interspeech(2015) in person is an experience I am likely to reminisce for a long long time. There are a ton of memories associated with a trip like that but something I found super infotaining was the (impromptu)discussions with senior PhD students and researchers both from academia and industry. This year we are hit with something that forced us remote. Am sure you have made the most of all the parallel sessions this year, great job! Here are some things I would have loved discussing with you in person.

Disclaimer: (1) These are my subjective opinions. (2) This post was co-edited by Judith. (3) My blind spot areas are: speech production & perception, phonology, brain stuff.

I like binning insights from conferences into three trend categories: trends that are (A) yet to play out, (B) playing out and (C) have played out. I will indicate each trend with its identifier in this post.

Towards objective metrics for evaluation of synthetic speech (A):

A huge bottleneck in development of neural generative models for speech(also text, image) is the evaluation. Typically metrics are subjective and are time consuming. Also such metrics make it hard to incorporate techniques such as RL seamlessly. I came across works this year attempting to predict naturalness of synthetic speech. They did quite an extensive evaluation. Interestingly, they found this approach language independent. There is also a work on learning a differentiable perceptual metric. On a side note, do you know that in Interspeech, someone is predicting paper reviews using a deep learning model? Read about it. Its cool :)

Learned Features are better than traditional features (A):

In paper “Investigating Self-supervised Pre-training for End-to-end Speech Translation”, authors show that self supervised features are better than traditional filterbank features. Let me tell you why this is fascinating. MFCCs or filterbanks working so well for every task was always the case, even though a ton of signal processing features were proposed over the last couple of decades. We seem to now have neural representations that are better than these traditional features. Alright, to be fair we knew this from a couple of years now, but this year we seem to be adding nuances to it: learned features are better in (1)low resource scenarios, (2) for phoneme discrimination (3) for spk id, etc. Other work in this direction is “Unsupervised Subword Modeling Using Autoregressive Pretraining and Cross-Lingual Phone-Aware Modeling”. This observation seems to hold across different training data sizes.

Error Rates of learned features vs MFCCs wrt training data from the paper “Unsupervised Subword Modeling Using Autoregressive Pretraining and Cross-Lingual Phone-Aware Modeling”

Easing the dependence on ASR in SLU (A)

Requirement of ASR was a strict one for many SLU applications and it seems to be going away. Removing ASR requirement (or easing it) can make approaches language independent as well as less error prone due to propagation of issues. Other advantages are inference speed up and memory efficiency. Papers “A low latency ASR-free end to end spoken language understanding system” and “ASR-Free Pronunciation Assessment” are examples. In the paper “An Audio-enriched BERT-based Framework for Spoken Multiple-choice Question Answering”, authors show that adding acoustic level statistics can offset the errors made by ASR in a cascased system. Their task was spoken multiple-choice question answering. But I guess this observation can be extended. Still early works these though. Looks like more will be in this direction.

Confidence Measures in Neural Models (B)

A huge advantage of traditional models was the ability to comment on model confidence. Posterior probability was a good proxy for model confidence. But thats difficult in neural models. In addition, there have been published works that argue that the hidden layers of deterministic autoregressive models have unknown marginal distributions. This makes it hard to employ them directly as confidence measures. There have been works towards confidence measures in our lovely neural models. Came across the paper “Confidence measure for speech-to-concept end-to-end spoken language understanding” in this year’s conference where authors show an interesting approach in this direction. They use multiple layers to alleviate the previous concerns.

Universal Approaches to Tasks (B)

There has been a constant rise in approaches towards universal solutions — both in terms of models as well as scenarios. Papers in this direction are:

“Towards Learning a Universal Non-Semantic Representation of Speech”

“Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions”

“Listen to What You Want: Neural Network-based Universal Sound Selector”

“Towards Universal Text-to-Speech”

“Universal Adversarial Attacks on Spoken Language Assessment Systems” Check this space out.

Discrete Latent Representations are here (B)

Perhaps VQVAE was the first comprehensive neural based model learning discrete representations in an unsupervised fashion from speech. Since then there have been numerous attempts at learning such representations. The papers “Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation”, “Learning Syllable-Level Discrete Prosodic Representation for Expressive Speech Generation” are in this direction. Watch out the works from Zero Resource speech challenge for more ideas in this space.

Meta Learning works in Speech (B)

Lots of paradigms proposed in core ML do not typically directly work in speech processing tasks. One of things we noticed as a trend last ICASSP and Interspeech was meta learning. And it seems to work very well in speech. There was a tutorial session which showed application in Spoken Language Understanding on Sunday. One of the papers I liked from the many in this year’s conference is “Learning Fast Adaptation on Cross-Accented Speech Recognition” where the authors showed that meta was better than doing multi/mixed accent training in the context of ASR.

Few shot results from “Learning Fast Adaptation on Cross-Accented Speech Recognition”

Seq2Seq Models to perform a combination of tasks (A)

The Chain rule based decomposition and subsequent end to end is usually employed in single tasks. I havent seen people doing multiple tasks using end to end models. This is true even in the case of HMMs or CRFs(may be coz they cant model long term dependencies well). I saw a paper titled “Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of any Number of Speakers” where they manipulate the objective function to perform multiple tasks in an end to end fashion. I believe this is the first time I am seeing this in a bunch of slightly unrelated tasks and perhaps there is more to come from here.

Data Augmentation and Privileged Data (B)

People have been using cleaner versions of data to assist in different tasks from a while now. Google used a mix of noisy data and clean version of data for speech synthesis / denoising. Amazon used noisy data to learn a noise invariant representation for ASR. I came across a paper “Implicit Transfer of Privileged Acoustic Information in a Generalized Knowledge Distillation Framework” where they name this as using privileged information (did I get this wrong?) This seems an interesting space. Last year’s ICASSP had a tutorial on data augmentation, perhaps indicating an increased interest in this direction. I have not seen many approaches manipulating the latent space of a speech utterance for augmentation. This conference, we a have a paper “SpeechMix — Augmenting Deep Sound Recognition Using Hidden Space Interpolations” in this direction. An interesting paper I found has something I did not expect with respect to augmentation: In “Large scale weakly and semi-supervised learning for low-resource video ASR” authors showed that their approach was better than the strongest data augmentation method they evaluated against.

Graph based Methods (B)

Graphs have interesting properties and they seem to have finally arrived in speech. In “Conversational Emotion Recognition Using Self-Attention Mechanisms and Graph Neural Networks”, authors propose relation

reduction process in the graph and demonstrate that only three kinfs of relation types need to be modeled for emotion recognition.

“Predicting Collaborative Task Performance using Graph Interlocutor Acoustic Network in Small Group Interaction”, authors employ a graph based approach to model inter and intra group perspectives in the context of personality behavior prediction. Other papers along these lines are:

“Entity Linking for Short Text Using Structured Knowledge Graph via Multi-grained Text Matching” and

“Improving Speech Emotion Recognition Using Graph Attentive Bi-directional Gated Recurrent Unit Network”

Measuring Disentanglement and Interpreting Models (B)

Speech has been slow compared to vision regards to disentanglement and perhaps due to the lack of visibly demonstrable/separable latent spaces. Saw a paper from CSAIL titled “Unsupervised Methods for Evaluating Speech Representations” towards this direction. While there are kinks to be addressed, this is I believe one of the earliest works and I anticipate more work to come. In “Understanding Self-Attention of Self-Supervised Audio Transformers”, authors show that self supervised audio transformers have three explainable attention categories — pertaining to two characteristics: phonetic and speaker. Along the same lines, authors in “Vector-Quantized Autoregressive Predictive Coding” limit model capacity while training and find that bottleneck-ed autoregressive stochastic models preserve phonetic and speaker information

Three attention patterns and the two characteristics from “Understanding Self-Attention of Self-Supervised Audio Transformers”

The trends I believe have almost played out are “Controllable Text to Speech”, “building data/compute/memory/inference-time efficient models(beyond just ASR and TTS)”. Finally there are certain areas that I dont consider trends, but are important to watch out for: Speech for Health applications and Privacy in speech. There are very interesting papers in these directions and you are gonna love them.

I will close this post with an example trend from notes I wrote after ICASSP last year. There was this paper by DeepMind about efficient coding for speech and one of my fav. Postdocs(whom I met at one such in person conferences) Gustav suggested this as something to look out for — Speech Coding. I wasn't long on this. You see, I believed then that there was little in terms of academic research w.r.t coding and has been for more than a couple of decades. I thought, “We have reliable codecs. What else do we need?”. You can verify that there has been a lot of work in this direction recently. This highlights the importance of having such notes: They provide an ability to look at things retrospectively. I highly encourage you to develop this habit.

My takeaway from the whole conference came from the keynote talk by Prof Lin-shan Lee “This maybe the golden age of (speech)research incentivized by deep learning, big data, powerful machines and a strong industrial workforce.” I will extend his quote by saying this: “…and realize that YOU are at the center of this. Do amazing research and enjoy every step of the way :)”

Okay, I have said enough. I will go back to work on my dissertation now. Looking forward to reading your next paper!