Unsupervised Audiovisual Synthesis via Exemplar Autoencoders
Authors: Kangle Deng
Aayush Bansal
Deva Ramanan
Venue: ICLR 2021 Poster (2021)
Keywords: unsupervised learning
autoencoders
speech-impaired
assistive technology
audiovisual synthesis
voice conversion

Abstract: We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers. Our approach builds on simple autoencoders that project out-of-sample data onto the distribution of the training set. We use exemplar autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target exemplar speech. In contrast to existing methods, the proposed approach can be easily extended to an arbitrarily large number of speakers and styles using only 3 minutes of target audio-video data, without requiring any training data for the input speaker. To do so, we learn audiovisual bottleneck representations that capture the structured linguistic content of speech. We outperform prior approaches on both audio and video synthesis.


Published as a conference paper at ICLR 2021
UNSUPERVISED AUDIOVISUAL SYNTHESIS VIA EXEMPLAR AUTOENCODERS
Kangle Deng, Aayush Bansal, Deva Ramanan Carnegie Mellon University Pitssbrgh,, PA 15213, USA (kangled, aayushb, deva (cs. cmu cmu.edu
ABSTRACT
We present an unsupervised approach that converts the input speech of any in- dividual into audiovisual streams of potentially-infinice ers. Our approach builds on simple autoencoders that project out-of-sample data onto the distribution of the training set. We use exemplar autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target exemplar speech. In contrast to existing methods, the proposed approach can be easily ex- tended to an arbitrarily large number of speakers and styles using only 3 minutes of target audio-video data, without requiring any training data for the input speaker. To do so, we learn audiovisual bottleneck representations that capture the struc- tured linguistic content of speech. We outperform prior approaches on both audio and video synthesis. Please visit our project website for summary video and more information.
many output speak-
INTRODUCTION
We present unsupervised approach to retargeting the speech of any unknown speaker to diovisual stream of a known target speaker. Using our approach, one can retarget a celebrity video clip say the words "Welcome to ICLR 2021" in different languages including English, Hindi, and Mandarin (please see our associated video). Our approach enables variety of novel applications because eliminates the need for training on large datasets; instead, is trained with unsupervised learning only few minutes of the target speech, and does not require any training examples of the input speaker. By retargeting input speech generated by medical devices such as electrolarynxs and text-to-speech (TTS) systems, our approach enables personalized voice generation for voice- impaired individuals (Kain et al., 2007] Nakamura et al., 2012). Our work also enables applications can create interactive documentaries about historical figures in
in education and entertainment; their voice, or generate the sound of actors who are no longer able to perform. We highlight such representative applications in Figure T
Prior work typically independently looks at the problem of audio conversion (Chou et al. 2019 Kaneko et al. 2019a]b] Mohammadi & Kim 2019: Qian et al.) 2019) and video generation from audio signals Yehia et al., 2002 [hung et al. 2017 Suwajanakoor et al., 2017] Zhou et al., 2019 Zhu et al., (0118). Particularly relevant are zero-shot audio translation approaches (Chou et al. 2019 Mohammadi & Kim 2019; Pollyk & Wolf 2019: Qian et al., 2019) that learn a generic low- dimensional embedding (from a training set) that are designed
agnostic to speaker identity (Fig. aa). We will empirically show that such generic embeddings may struggle to capture stylistic details of in-the-wild speech that differs from the training set. Alternatively, one can directly learn an audio translation engine specialized to specific input and output speakers, often requiring data of
the two speakers either aligned/paired (Chen et al.] 2014 Nakashika et al., 2014 (unn et al. 2015
Toda et al. 2007) or maaligeed/unpaired (Chou et al. 2018 Fang et al. 2018 Kameoka et al. 2018
Kaneko & Kameoka 2017] Kaneko 20.12019A| by Serrà et al. 2019). This requirement restricts
such methods to known input speakers at test time (Fig. 2)b). In terms of video synthesis from
audio input, zero-shot facial synthesis approaches (Chung et al. 2017 Zhou et al., 2019) Zhu et al.)
2018) animate the lips but struggle to capture realistic facial characteristies of the entire person.
https: https://dunbrr12211 . www.ggittuu.... httpggghhh..//pppecpppee///////e
sual/

---

Published as a conference paper at ICLR 2021
We Exemplar Autoencoders for infirilely speakers =5 minutes speech for individusi speaker without additiional information.
inputi Electiokanyn
Inout Text nol--o--o Speech System
nerken Ehrosen
Input matlow Hind
Input "Mi the and, this
हिदी
S X
output: natural volce Micheile Doame.
Takeo Kanade
ratural lau
John Clivr's vaisa.
John Oirr's vals.
output
Winston Chundh Audio-Vbull Speeth
Tool Impaired: Zero-Shat Natural
Constranns::
(e). GlacationalErtetaarrre Purposes:
Syrthesia
Zero-Shat Lutti-Lingul Translation
Zero-Shol Audio-Vizull Content Creation
Figure 1: We train audiovisual (AV) exemplar autoencoders that capture personalized in-the-wild web speech as shown in the top-row. We then show three representative applications of Exemplar Autoencoders: (a) Our approach enables zero-shot natural voice synthesis from an Electrolarynx or TTS used by approach can generate Chinese and Hindi speech for John Oliver, an English-speaken late-night show host; and (c) We generate audio-visual content for historical documents that could not be otherwise captured.
person; (b) Without any knowledge of Chinese and Hindi,
Other approaches (Gioosar al 2019 Shlizerman et al. 2018 Suwajanakorn et al. 2017 restrict themselves to known input speakers at test time and require large amounts of data to train a model in supervised manner.
Our work combines the zero-shot nature of generic embeddings with the stylistic detail of person- specific translation systems. Simply put, given target speech with particular style and ambient environment, we learn an autoencoder specific to that larget speech (Fig. ()). We deem our ap- proach "Exemplar Autoencoders". At test time, we demonstrate that one translate any input speech into the target simply by passing it through the target exemplar autoencoder. We demon- strate this property tend to cluster quite well in spectrogram space a)); and (2) autoencoders with sufficiently small bottlenecks act as projection operators that project out-of-sample source data onto the target training distribution, allowing to preserve the content (words) of the source and the style of the target (Fig, (tc). Finally, we jointly synthesize audiovisual (AV) outputs by adding a visual stream to the audio autoencoder. Importantly, our approach is data-efficient and can be trained using minutes of audio-video data the target speaker and no training data for the input speaker. The ability to train exemplar autoencoders small amounts of data is crucial when learning specialized models tailored to particular target data. Table contrasts our work with leading approaches in au- dio conversion (Kaneko al. 2019c Qian et al. 2019) and audio-to-video synthesis (Chung al., 2017 Suwajanakorn et al. (0177).
consequence of two curious facts, shown in Fig. 3 (I) linguistic phonemes
Contributions: (1) We introduce exemplar autoencoders, which allow for any input speech to be converted into an arbitrarily-large number of target speakers ("any-to-many" AV synthesis). (2) We move beyond well-curated datasets and work with in-the-wild web audio-video data in this paper. We also provide a new CelebAudio dataset for evaluation. (3) Our approach can be used an off- the-shelf plug and play tool for target-speeific voice conversion. (4) Finally, because our approach generates high-fidelity audio and video content that could be potentially misused, we discuss broader impacts the appendix, including forensic experiments that suggests fake content can be identified with high accuracy.
2 RELATED WORK
A tremendous interest in audio-video generation for health-care, quflity-of-life improvement, educa- tional, and entertainment purposes has influenced a wide variety of work in audio, natural language processing, computer vision, and graphics literature. In this work, seek to explore standard representation for a user-controllable "any-to-many" audiovisual synthesis.
Speech Synthesis & Voice Conversion: Earlier works (Hunt & Black 1996; Zen al. 2009) in speech synthesis use text inputs to create Text-to-Speech (TTS) systems. Sequence-to teaeiceett--uuuueeee (Seq2seq) structures (Sutskever al., 2014) have led significant advancements in TTS sys-
2

---

Published as a conference paper at ICLR 2021
Generic-User
Person-Specific
Generic-User & Person-Specilic
Offiie training with fiee number speskerss:
Offline training speskers:
On-the-fly training with number of speakers:
(St,
((S.), (Se)... (8k).
Test time: any unknown speaker
Test time: (Si, So)
Test time: any unimoww speaker
input speech
fhom sry uset
content embedding
So
Sa
inuu style to
target oporier
decoder
S1
Su
Swi
S
speaker embodding
translation on a par of speakers
loamning speaker speakar-spoctte model
zero-shot conversion
(b) one-on-one conversion
(c) ours
exemplar autoencoders
Figure 2: Prior approaches can be classified into two groups: (a) Zero-shot conversion learns generic low-dimensional embedding from a training set that is designed to be agnostic to speaker identity. We empirically observe that such generic embeddings may struggle to capture stylistic details of in-the-wild speech that differs from the training set. (b) Person-specific one-on-one con- version learns translation engine specialized to specific input and output speakers, restricting them to known input speakers at test time. (c) In this work, we combine the zero-shot nature of generic embeddings with the stylistic detail of person-specific translation systems. Simply put, given tar- get speech with a particular style and ambient cuvironment, we learn an autoencoder specific to that target speech. At test time, one translate any input speech into the target simply by passing through the target exemplar autoencoder.
tems (Oord et al., 2016; Shen et al., 2018; Wang et al. 2017). Recent works (Jia et al., 2018) have extended these models to incorporate multiple speakers. These approaches enable audio version by generating text from the input via a speech-to-tex (STT), and use a TTS for target audio. Despite enormous progress in building TTS systems, it is not trivial to embody perfect emotion and prosody due to the limited expressiveness of bare text (Pitrelli et al.] 2006). In this work, we use the raw speech signal of target speech to encode the stylistic nuance and subtlety that wish to synthesize.
Audio-to-Audio Conversion: The problem of audio-to-audio conversion has largely been confined
to one-to-one translation, be it using a paired (Chen et al. 2014 Nakashika et al. 2014 Sun et al.
2015 Toda et al. 2007) or unpaired (Chen al.] 2019 Kaneko & Kameoka [011]] 7 Kaneko al.
2019a b Serrà et al. 2019 Tobing et al. 2019; Zhao et al. 2019) data setup. Recent works (Chou al.) 2019 Mohammadi have begun to explore any-to-any translation, where the goal is to generalize to any input and any target speaker. To do so, such approaches learn a speaker-agnostic embedding space that meant to generalize to never-before-seen identities. Our work is closely inspired by such approaches, but is based on the observation that such generic embeddings tend to fow-dimensional, making diffi- cult to capture the full range of stylistic prosody and ambient environments that can exist in speech. We can, however, capture these subtle but important aspects via an exemplar autoencoder that is trained for specific target speech exemplar. Moreover, it is not clear how extend such generic embeddings to audiovisual generation because visual appearance highly multimodal (people can appear wildly different due clothing). Exemplar autoencoders, on the other hand, are straightfor- ward to extend to video because they inherently are tuned to the particular visual mode (clothing) in the target speech.
Kim 2019 Paul et al. 2019 Polyak & Wolf 2019) Qian et al. 2019)
Audio-Vidae Synthesis: There is a growing interest (Chung et al. 2018 Nagrani et al. 2018 Oh
al. 2019) to jointly study audio and video for better recognition (Kazakos et al.) 2019), localizing
sound (Gao et al. 2018; [Senocak et al. 2018). or learning better visual representations Owenss
Efros 2018 Owens et al.) (016). There is a wide literature (Berthouzoz et al., 2012, Bregler
et al. 1997.
Fan et al.j2015 Fried et al., 2019 Greenwood et al.) [ Taylor et al. 2017 Thies
ct al. 2020 Wang et al. 2021| Kumar et al. (020) on synthesizing videos (talking-heads, from an audio signal. Early work Yehia et al. 2002) links speech acoustics with coefficients necessary to
animate natural face and head motion. Recent approaches (Chung et al. 2017 Zhou et al.) 2019
Zhu et al. 2018) have looked at shot facial synthesis from audio signals. These approaches register human faces using facial keypoints so that one can expect a certain facial part at a fixed location. At test time, image of the target speaker and audio is provided. While these approaches
3

---

Published as a conference paper at ICLR 2021
Method
input: audio,
unknown test
speaker-specillc model
output:
speaker
Auto-VC Qian ot 2019
StuGAN VC Kaneko et
2019b
Sid Chung
Synthesizing Obama Suwajaanakorn et al.
2017
V
x
Ours (Exemplar Autoencoders)
Table 1: We contrast our work with leading approaches in audio conversion (Kaneko et al. Qian et al. 2019) and audio-to-video generation (Chung et al., 2017 (Suwajanakorn et al. (017)) Unlike past work, we generate both audio-video output from an audio input (left). Zero-shot meth- ods are attractive because they can generalize unknown target speakers at test time (middle). However, in practice. models specialized to particular pairs of known input and target speakers tend produce more accurate results (right). Our method combines the best of both worlds by tuning an exemplar autoencoder mn-the-fly to the target speaker of interest.
2019b]
can animate lips of the target speaker, it is still difficult to capture other realistic facial expressions. Other approaches (Ginosar et al. 2019 Shlizerman et al. 2018 Suwajanakom et al., 2017) use large amount of training data and supervision (in the form of keypoints) to learn person person-specific synthesis models. While these generate realistic results, they are restricted to known input users at test time. In contrast, our goal is to synthesize audiovisual content (given any input audio) in data-efficient manner (using 2-3 minutes of unsupervised data).
Autoencoders are unsupervised neural networks that learn reconstruct inputs via a bottleneck layer (Bengio et al.] (017). They have traditionally been used for dimensionality reduction or feature learning, though variational formulations have extended them to full generative models that can be used for probabilistic synthesis (Kingma & Welling 2013). Most related to us are denoising autoencoders (Vincent et al. 2008), which learn to project noisy input data onto the underlying training data manifold. Such methods are typically trained on synthetically-comee inputs with the goal of learning useful representations. Instead, we exploit the projection property of autoencoders itself, by repurposing them as translation engines that "project" the input speech of an unknown speaker onto the target speaker manifold. It is well-known that linear autoencoders, which can be learned with PCA (Baldi & Hornik 1989). produce the best reconstruction of any sample input datapoint, in terms of squared error from the subspace spanned by the training dataset (Bishop 2006). We empirically verify that this property approximately holds for nonlinear autoencoders (Fig. 33. When trained on a target (exemplar) dataset consisting of a particular speech style, we demonstrate that reprojections of out-of-sample inputs tend to preserve the content of the input and the style of the target.
Autoencoders:
3 EXEMPLAR AUTOENCODERS
There is an enormous space of audiovisual styles spanning visual appearance, prosody, pitch, emo- tions, and environment. It is challenging for single large model to capture such diversity for accurate audiovisual synthesis. However, many small models may easily capture the various nu- ances. In this work, we learn a separate autoencoder for individual audiovisual clips that are limited particular style. We call our technique Exemplar Autoencoders. One of our remarkable findings are that, when trained appropriately, Exemplar Autoencoders can be driven with the speech input of a never-before-se speaker, allowing for the overall system to function as a audiovisual translation engine. We study in detail why audiovisual autoencoders preserve the content (words) of never- before-seen input speech but preserve the style of the target speech on which it was trained. The heart of our approach relies on the compressibility of audio speech, which describe below.
Speech contains two types of information: (i) the content, or words being said and (ii) style infor- mation that describes the scene context, person- sppein- characteristics, and prosody of the speech delivery. It is natural to assume that speech is generated by the following process. First, a style s is drawn from the style space S. Then a content code is drawn from the content space W. Fi-
4

---

Published as a conference paper at ICLR 2021
projected perzze
out of sample legut
out-ofaampl input
CAT
projected point
training dats sabepare
training data manidod
(a) Linear Autoencoder (PCA)
Autoencoder
(c) I-SNE for acoustic features
Figure 3: Autoencoders with sufficiently small bottlenecks project out-of-sample data onto the man- ifold spanned by the training set (Bengio al. 2017). which can be easily seen in the case of linear autoencoders learned via PCA (Bishop) 2006). We observe that acoustic features (MEL spec- tograms) of words spoken by different speakers (from the VCTK dataset (Veaux et al. 2016 easily cluster together irrespective of who said them and with what style. We combine these observations design simple style transfer engine that works by training an autoencoder on single target style with a MEL-spectogram reconstruction loss. Such exemplar autoencoders project out-of-sample input speech content onto the style-specific manifold of the target.
nally, x f(s, w) denotes the speech of the content w spoken in style s, where f is the generating function of speech.
3.1 STRUCTURED SPEECH SPACE
In human acoustics, one uses different shapes of their vocal trac to pronounce different words with their voice. Interestingly, different people use similar, or ideally the same, shapes of vocal tract to pronounce the same words. For this reason, in the speech space the acoustic features the same words different styles are very close (also shown in Fig. a).. Given styles 81 and 82 for example, [(1, wo) is one spoken word in style 81. Then (82, wo) should be closer to f(11, Wo) than any other word in style 82. We can formulate this property as follows:
find built-in structure that
Efror(f(s1, wo), (s2, wo)) < (sr wo), f(2,, w)), Vw W, where $1, 82 S.
This can be further presented in equation form:
f(22, wp) = arg min Error(æ, f(11, wo)), where = w)) WEW. aEM
(I)
3.2 AUTOENCODERS FOR STYLE TRANSFER
We now provide a statistical motivation for the ability of exemplar autoencoders to preserve content while transferring style. Given training examples (z,), one learns an encoder E and decoder D so as to minimize reconstruction error:
Error 2ú D(F().)))
min E.D
In the linear case (where E(x) = Ar, D(x) Br, and Error = L2), optimal weights are given by the eigenvectors that span the input subspace of data (Baldi & Hornik 1989). Given sufficiently small bottlenecks, linear autoencoders project out-of-sample points into the input subspace, so as to minimize the reconstruction enror of the output (see Fig. (a)). Weights of the autoencoder (eigen- vectors) capture dataset-specific style common to all samples from that dataset, while the bottleneck activations (projection coefficients) capture sample-specific content (properties that capture individ- ual differences between samples). In the appendix, we empirically verify that similar property
"The vocal tract is the cavity in human heings where the sound produced at the sound source is filtered. The shape of the vocal tract is mainly determined by the positions and shapes of the tongue, throat and mouth.
5

---

Published as a conference paper at ICLR 2021
Audio
80
64
64"(T/32)
Audio Decoder
Down Samp Sampping
ID-CNN=3
Bi-LSTMx2
Content Code
Unsampled
Content Code
Up Sampling
Bi-ISTM
ID-CNN*3
Bi-LSTMx3
WaveNet
STET
m
772
Overview
2564(T/4)
512"512((T/4)
3D An-transCNNNxG
128
Up- Rlockes
Audio
3D-CNN=3
1D-CNN
Audio Encoder
Audio Decoder
Audio Encoder
Decoder
Video
Decoder
V Video Decoder
WVoice-Conversiog
(bJAudio-Video Syntheiss
Figure 4: Network Architecture: (a) The coice-conversion network consists of a content encoder, and an audio decoder (denoted as green). This network serves person & attributes-spedt specific auto- encoder at training time, but is able to convert speech from anyone personalized audio for the target speaker at inference time. (b) The audio-video synthesis network incorporates a video decoder (denoted as yellow) into the voice-conversion system. The video decoder also regards the content encoder as its front-end, but takes the unsampled content code input due to time alignment. The video architecture is mainly borrowed from StackGAN (Radford et al. 2016 Zhang et al.) 2017). which synthesizes the video through 2 resolution- stages.
holds for nonlinear autoencoders: given sufficiently-small bottlenecks, they approximately project satple data onto the nonlinear manifold M spanned by the training set:
D(E(E)) arg min Error(m, f)), where M (D(E())) E Rd).
mEM
where the approximation (f(82, w) w E W} be the manifold spanning particular style 82. Let f f(11, wy) be par- ticular word we spoken with any other style $1. The output D(E()) will be a point on the target manifold M (roughly) closest to â. From Eq. [ and Fig. the closest point the target manifold tends to be the same word spoken by the target style:
exact the linear case. the nonlinear case, let M
D(E(t)) arg min (t, #) arg min Error(t, f(s,, [((2, wo). BEA
ZEA
Note that nothing in our analysis is language-specific, so still holds, in principle, for other lan- guages such as Mandarin and Hindi. We posit that the "content" can be represented by phonemes that can be shared by different languages. our summary video, we verify this by showing that audiovisual translation is possible across different languages; one can drive John Oliver to speak in Chinese or Hindi.
3.3 STYLISTIC AUDIOVISUAL (AV) SYNTHESIS
We now operationalize our previous analysis into an approach for stylistic AV synthesis. To do so, we learn AV representations with autoencoders tailored to particular target speech. To enable these representations be driven by the audio input of any user, we learn in manner that ensures bottleneck activations capture structured linguistic "word" content: we pre-train an audio-oniy au- toencoder, and then learn a video decoder while finetuning the audio component. This adaptation allows us to train AV models for target identity with as little as 3 minutes of data (in contrast to the 14 hours used (Suwajanakorn et al. 2017
Audio autoencoder: Given the target audio stream of interest we first convert it to a mel spec- togram m F(x) using a short-time Fourier Transform (Oppenheim et al. 1996). We train an encoder E and decoder D that reconstructs Mel-spectograms m, and finally V (Oord et al. 2016) to convert m back to speech ž. Importantly, we train all components (the encoder E, decoder D, and vocoder V) with joint reconstruction loss in both frequency and audio space:
a WaveNet vocoder
ErrorAudee (I, .) ElJm mil1 ax,

---

Published as a conference paper at ICLR 2021
VCTK Veaux
2016
Satra-Data
(8)
StarGAN VC Kaneko
2019b
69.5
582.1
VQ. VAE
den Oord
2017
69.9
663.4
Chou
Chou Chou el.
2019
V
57.0
491.1
Auto-VC
2019
/
98.5
408.8
Ours
X 99.6
420.3
Table 2: Objective Evaluation for Audio Translation: We evaluate on VCTK, which provides paired data. The speaker-llassificttee accuracy (SCA) criterion enables us study the naturalness of generated audio samples and similarity to the target speaker, where higher is better. Typical methods measure reoconstruction error with Mel-Cepstral distortion (MCD) (where lower is bet- ter), but our prior analysis show that this will be dominated by matching the content words (Fig. 3. Our approach achieves competitive performance to prior state-of-the-arr without requiring additional data from other speakers and yet be zero-shot. We do more comprehensive human study in Table using CelebAudio dataset to study the influence of data.
where LWaveNed is the standard cross- cross-entropy loss used to train Wavenet (Oord et al. 2016). Fig. (top-row) summarizes our network design for audio-only autoencoders. Additional training param- eters are provided in the appendix.
Video decoder: Given the trained audio autoencoder E(x) and target video e, we now train video decoder that reconstructs the video e from Exx) (see Fig. Specifically, we train using joint loss:
Error AV (Ir, U, 8) Error Andio (2,,
T|l + Lad (t,
where Adv is an adversarial loss (Goodfellow et al. 2014) used to improve video quality. We found helpful to simultaneously fine-tune the audio autoencoder and add equation to the overall AV loss.
4 EXPERIMENTS
We now quantitatively evaluate the proposed method for audio conversion in Sec. 4.1 and audio- video synthesis in Sec. 4.2 We attempted find all public codebases that could serve as baselines for audio conversion and audiovisual generation. We use existing datasets for fair evaluation with prior art. We also introduce a new challenging dataset that consists of recordings in non-studio environments. Many of our motivating applications, such education and assistive technology, require processing unstructured real-worle data collected outside a studio. We urge the reader to see our summary video see/hear our results, well as the appendix for additional ablative analysis.
4.1 AUDIO TRANSLATION
We use the publicly available VCTK dataset (Veaux et al. 2016), which contains 44 hours of utterances from 109 native speakers of English with various accents. Each speaker reads a different set of sentences, except for two paragraphs. While the conversion setting is unpaired, there exists small amount of paired data that enables to conduct objective evaluation. We also introduce the new in-the-wild CelebAudio dataset for audio translation to validate the effectiveness as well as the robustness of various approaches. This dataset consists of speeches (with an average of 30 minutes, but some as low as 3 minutes) of various public figures collected from YouTube, including native and non-native speakers. The content of these speeches entirely different from one another, thereby forcing the future methods be unpaired and unsupervised. There are 20 different identities. We provide more details about CelobAudio dataset the appendix.
Datasets:
Quantitative Evaluation: We rigorously evaluate our output with variety of metrics. When ground- truth paired examples are available for testing (VCTK), we follow past work (Kaneko et al.) 2019alb) b) and compute Mel-Cepstral distortion (MCD), which measures the squared distance be- tween the synthesized audio and ground-truth in frequency space. Inspired by "FCN- score" metrics for image generation, propose speaaer-classfficttonn accuracy (SCA), which is defined as

---

Published as a conference paper at ICLR 2021
CelehAudio
Naturalness
Voice Similarity f MOS
Content Consistency
Geometric Mean
Data
MOS
Pref
Pref (%)
MOS
Pref (56
of VS-CC
Auto-VC VC QQia Vian al 2019
off-Ibe-shelf
1.21
1.31
1.60
0
1.45
fine-tuned
2.35
13.3
1.97
2.0
4.28
48.0
2.90
scratch
x
2.28
1.90
2.0
4.05
18.0
2.77
Ours
2.78 66.7
3.32
94.0
4.00
22.0
3.64
Table 3: Human Studies for Audio: We conduct extensive human studies on Amazon Mechanical Turk. We report Mean Opinion Score (MOS) and user preference (percentage of time that method ranked best) for naturalness, target voice similarity (VS), source content consistency (CC), and the geometric mean of VS-CC (since either can be trivially maximized by reporting a target/input sam- ple). Higher the better, on a scale of 1-5. The off-the-shelf Auto- VC model struggles to generalize to CelebAudio, indicating the difficulty of wn-the-wild zero-shot conversion. Fine-tumm timeettuing on Cele- bAudio dataset significantly improves performance. When restricting Auto-VC to the same training data as our model (scratch), performance drops a small but noticeable amount. Our results strongly outperform Auto-VC for VS-CC, suggesting exemplar autoencoders are able to generate speech that maintains source content consistency while being similar to the target voice style. We provide more ablation analysis in the Appendix.
the percentage of times a translation correctly classified by a speaker-classifie (trained with Serrà ct al. (2019)).
Human Study Evaluation: We conduct extensive human studies on Amazon Mechanical Turk (AMT) to analyze the generated audio samples from CelebAudio. The study assesses the natural- ness of the generated data, the voice similarity (VS) to the target speaker, the content/word consis- tency (CC) to the input, and the geometric mean of VS-CC (since either can be trivially maximized by reporting a target/input speech sample). Each is measured on a scale from 1-5. We select 10 CelebAudio speakers as targets and randomly choose 5 utterances from the other speakers inputs. We then produce 5 x 10 = 50 translations, each of which is evaluated by 10 AMT users.
Baselines: We contrast our method with several existing voice conversion systems (Chou al.,
2018 Kaneko et al. 2019b)] van den Oord et al. 2017]] Qian et al.) 2019 Serrà et al. (019). Because StarGAN-VC (Kaneko et al. (001b)), VQ-VAE (van den Oord et al. 2017). Chou al. (Chou et al. 2018), and Blow (Serrà et al. (2199) do not claim zero-shot voice conversion, we train these models on 20 speakers from VCTK and evaluate voice conversion on those speakers. Sablee]shone that we outperform these approaches for speaker classification, while being competitive for MCD reconstruction error.
Auto-VC VC (Qian et al. 2019): ACto-VC is a zero-shot translator. It performs quite well in SCA and MCD metrics in Table 2 and so we extensively explore it via AMT human studies in Table 3 Since Auto-VC claims any-to-any audio conversion, we first use off-the-shelf model for evalu- ation. We observe poor performance, both quantitatively and qualitatively. We then fine-tune the existing model audio data from 20 CelebAudio speakers. We observe significant performance improvement in Auto-V VC when restricting it to the same set of examples as ours. We also trained the Auto-VE VC model from scratch, for apples-to-apples comparison with The performance on all three criterion dropped with lesser data. On the other hand, our approach generates significantly better audio which sounds more like the target speaker while still preserving the content.
4.2 AUDIO- VIDEO SYNTHESIS
Dataset: In addition to augmenting CelebAudio with video clips, we also evaluate on VoxCeleb Chung et al. 2018), an audio-visual dataset of short celebrity interview clips from YouTube.
We compare to Speech2Vid (Chung et al. 2017). using their publicly available code. This approach requires the face region to be registered before feeding it to a pre-trained model for facial synthesis. We find various failure cases where the human face from in-the-wild videos of VoxCeleb dataset cannot be correctly registered. Additionally, Speech2 generates the video of a cropped face region by default. We composite the output on a still background to make it more compelling and comparable ours. Finally, we also compare to the recent work of LipGAN (KR)
Baselines:
8

---

Published as a conference paper at ICLR 2021
et al. 2019). This approach synthesizes region around lips and paste this generated face crop into the given video. We observe that the paste is not seamless and leads to artifacts especially when working with in-the-wild web vidcos. We show comparisons with both Speech2Vid (Chung al, 2017) and LipGAN (KR et al., 2019) in Figure [6] The artifacts from both approaches are visible when used for in-the-wild video examples from CelehAudio and VoxCeleb dataset. However, approach generates the complete face without these artifacts and captures articulated expressions.
Human Studies: We conducted AB-Test AMT: show a pair of our video output and another method to the user, and ask the user to select which looks better. Each pair is shown to users. Our approach is preferred 87. 3%.2% over Speech2Vid (Chung et 2017) and 59.3% over LipGAN (KR et al. 2019).
5 DISCUSSION
In this paper, we propose exemplar autoencoders for unsupervised audiovisual synthesis from speech input. Our model able to generalize to in-the-wild web data and diverse forms of speech input, enabling novel applications in entertainment, education, and assistive technology.
Our work opens up rethinking of autoencoders' modeling capacity. Under certain conditions, autoencoders serve as projective operators to some specific distribution.
In the appendix, we provide an extensive discussion of broader impacts including potential abuses of such technology. We explore mitigation strategies, including empirical evidence that forensic classifiers can be used to detect synthesized content from our method.

---

Published as a conference paper at ICLR 2021
REFERENCES
Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 1989.
Yoshua Bengio, Ian Goodfellow, and Aaron Courville. Deep learning, volume 1. MIT press Mas- sachusetts, USA:, 2017.
Floraine Berthouzoz, Wilmot Li, and Maneesh Agrawala. Tools for placing cuts and transitions in interview video. ACM Trans. Graph., 2012.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statis- fics). Seringer-Verlapn 2006.
Christoph Bregler, Michele Covell, and Malcolm Slaney. Video rewrite: Driving visual speech with audio. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques, pp. 353-360, 1997.
Li-Wei Chen, Hung- Yi Lee. and Yu Tsao. Generative adversarial networks for umpaired voice trans- formation on impaired speech. Proc. Interspeech 2019. 2019.
Ling-Hui Chen, Hua Ling, Li-Juan Liu, and Li-Rong Dai. Voice conversion using deep neural networks with layer-wise generative training. JEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 2014.
Ju-Chieh Chou, Cheng-Chieh Ych, Hung-Yi Yi Lee, and Lin-Shan Lee, Multi-target Voice Conver- sion without Parallel Data by Adversarially Learning Disentangled Audio Representations. Proc. Interspeech, 2018.
Ju-chich Chou, Cheng-chich Yeh, and Hung-yi Lec. One-shot voice conversion by separating speaker and content representations with instance normalization. In INTERSPEECH, 2019.
Joon Son Chung, Amir Jamaludin, and Andrew Zisserman. You said that? arXiv preprint
a7Xiv:100..02966, 2017.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. VoxCeleb2: Deep speaker recognition. In INTERSPEECH, 2018
Bo Fan, Lijuan Wang. Frank K Soong, and Lei Xie. Photo-real talking head with deep bidirec- tional Istm. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4884 4888. IEEE, 2015.
Fuming Fang, Junichi Yamagishi, Isao Echizen, and Jaime Lorenzo-Trueba. High-Quality Non- parallel Voice Conversion Based on ColleCConsstent Adversarial Network. In IEEE ICASSP, 2018.
Ohad Fried, Ayush Tewari, Michael Zollhöfer, Adam Finkelstein, Eli Shechtman, Dan Gold- man, Kyle Genova, Zeyu Jin, Christian Theobalt, and Maneesh Agrawala. Text-based editing of talking-head video. ACM Trans. Graph., 2019.
Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. ECCV, 2018.
S. Ginosar, A. Bar, Kohavi, C. Chan, A. Owens, and Malik. Learning individual styles of conversational gesture. In CVPR, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
David Greenwood, Iain Matthews, and Stephen Laycock. Joint learning of facial expression and head pose from speech. Interspeech, 2018.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. JEEE Transactions Acoustics, Speech, and Signal Processing, 32(2):236-243, 1984.
10

---

Published as a conference paper at ICLR 2021
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
A. J. Hunt and A. W. Black. Unit selection in concatenative speech synthesis system using large speech database. In IEEE ICASSP, 1996.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML,, 2015.
Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang. Ignacio Lopez Moreno, Yonghui Wu, et al. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In NeurIPS, 2018.
Alexander Kain, John-Paul Hosom, Xiaochuan Niu, Jan PH Van Santen, Melanie Fried-Oken, and
Janice Staehely. Improving the intelligibility of dysarthric speech. Speech communication, 2007.
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Acvae-ve: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder. arXiv preprint 0rXx:::/08.00509, 2018.
Takuhiro Kaneko and Hirokazu Kameoka. Parallel-data-fre voice conversion using cycle-consistent adversarial networks. arXiv preprint 171.. 1293, 2017.
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. Cyclegan-ve2. Improved cyclegan-based non-parallel voice conversion. In IEEE ICASSP, 2019a.
Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, and Nobukatsu Hojo. StarGAN-VC2: Rethink- ing Conditional Methods for StarGAN-Based Voice Conversion. Proc. Interspeech, 2019b.
Takuhiro Kaneko, Hirokazu Kamcoka, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve2: Rethinking conditional methods for blaed voice conversion. Proc. Interspeech 2019, 2019c.
Evangelos Kazakos, Arsha Nagrami, Andrew Zisserman, and Dima Damen. Epic-fusion: Audio- visual temporal binding for egocentric action recognition. In ICCV, 2019.
Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. Deep video por- traits. ACM Transactions on Graphics 370G),37(4):1144 2018.
Diederik P Kingma and Max Welling. arXiv: 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Prajwal KR, Rudrabha Mukhopadhyay, Jerin Philip, Abhishek Jha, Vinay Namboodiri, and CV Jawahar. Towards automatic face-to-face translation. In Proceedings of the 27th ACM In- ternational Conference on Multimedia, pp. 1428-1436, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012.
Neeraj Kumar, Srishti Goel, Ankur Narang, and Mujtaba Hasan. Robust one shot audio to video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR) Workshops, June 2020.
Maxim Maximov, Ismail Elezi, and Laura Leil-Taixé. Ciagan: Conditional identity anonymization the IEEE/CVF Conference on Computer Vision
generative adversarial networks. In Proceedings the and Puttern Recognition, pp. 5447-5456, 2020.
Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. Pulse: Self- supervised photo upsampling via latent space exploration generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2437-2445, 2020.
Scyed Hamidreza Mohammadi and Tachwan Kim. One-shot voice conversion with disentangled representations by leveraging phonetic posteriorgrams. Proc. Interspeech, 2019.
Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Seeing voices and hearing faces: Cross- modal biometric matching. In CVPR, 2018.

---

Published as a conference paper at ICLR 2021
Keigo Nakamura, Tomoki Toda, Hiroshi Saruwatari, and Kiyohiro Shikano. Speaking-aie -ii systems using gmm-based voice conversion for electrolaryngeal speech. Speech Commun., 2012.
Toru Nakashika, Tetsuya Takiguchi, and Yasuo Ariki. High-order sequence modeling using speaker- dependent recurrent temporal restricted boltzmann machines for voice conversion. In Proc. Inter- speech, 2014.
Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T. Freeman, Michael Rubinstein, and Wojciech Matusik. Speech2face: Learning the face behind a voice. In CVPR, 2019.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.034990 2016,
Alan V. Oppenheim, Alan S. Willsky, and S. Hamid Nawab, Signals & Systems (Second Edition). Prentice-Han Hall, Inc., 1996.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. ECCV, 2018.
Andrew Owens, Jiajun Wu, Josh McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016.
Dipjyoti Paul, Yannis Pantazis, and Yannis Stylianou. Non-parallel voice conversion using weighted generative adversarial networks. Proc. Interspeech 2019, 2019.
John F Pitrelli, Raimo Bakis, Ellen M Eide, Raul Fernandez, Wael Hamza, and Michael A Picheny.
The ibm expressive text-to-speech synthesis system for american english. IEEE Transactions on Audio, Speech, and Language Processing, 2006.
Adam Polyak and Lior Wolf. Attention-based wavenet autoencoder for universal voice conversion. In IEEE ICASSP, 2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In ICML, 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. FaceForensics++: Learning to detect manipulated facial images. In International Con- ference on Computer Vision (ICCV), 2019.
Arda Senocak, Tae-Hyun Oh, Junsik Kim, Ming-Hsuan Yang, and In So Kweon. Learning to local- ize sound source in visual scenes. In CVPR, 2018.
Joan Serrà, Santiago Pascual, and Carlos Segura. Blow: single-scale hyperconditioned flow for non-parallel raw-audio voice conversion. In NeurIPS, 2019.
Jonathan Shen, Ruoming Pang. Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. Natural its synthesis by con- ditioning wavenet on mel spectrogram predictions. In IEEE ICASSP, 2018.
Eli Shlizerman, Lucio Dery, Hayden Schoen, and Ira Saaeemmaahhe--Shizemaaa Audio body dynamics. In CVPR, 2018.
Gina Marie Stevens and Charles Doyle. Privacy: An overview of federal statutes governing wire- tapping and electronic eavesdropping. Lulu. com, 2011.
Lifa Sun, Shiyin Kang, Kun Li, and Helen Meng. Voice conversion using deep bidirectional long short-term memory based recurrent neural networks. In IEEE ICASSP, 2015.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In NIPS, 2014.
12

---

Published as a conference paper at ICLR 2021
Supasorn Suwajanakom, Steven M. Seitz, and Ira Kemelmacher-Shhizemeee Synthesizing obama: Learning lip syne from audio. ACM Trans. Graph., 2017.
Sarah Taylor, Tachwan Kim, Yisong Yue, Moshe Mahler, James Krahe, Anastasio Garcia Rodriguez, Jessica Hodgins, and lain Matthews. deep learning approach for generalized speech animation. ACM Transactions on Graphics (TOG), 36(4):1-11,
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias Nießner. Neural voice puppetry: Audio-driven facial reenactment. European Conference on Computer Vision, pp. 716-731. Springer, 2020.
Patrick Lumban Tobing, Yi-Chiao Wu, Tomoki Hayashi, Kazuhiro Kobayashi, and Tomoki Toda. Non-parallel voice conversion with cyclic variational autoencoder. Proc. Interspeech 2019, 2019.
Tomoki Toda, Alan W Black, and Keicchi Tokuda. Voice conversion based on Bexximum-likellioon estimation of spectral parameter trajectory. IEEE Transactions on Audio, Speech, and Language Processing, 2007.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Christophe Veaux, Junichi Yamagishi, and Kirsten Macdonald. Superseded CSTR VCTK Corpus:
English multi-speaker corpus for CSTR voice cloning toolkit. 2016.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103 2008.
Sheng-Yt Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Can-generated images surprisingly easy to spot...for now. In CVPR, 2020.
Ting-Chun Wang, Arun Mallya, and Ming- Yuing-Yulaan Liu. One-shot free-view neural talking-head synthesis for video conferencing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Yuxuan Wang. RJ Skerry-Ryan. Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. Proc. Interspeech, 2017.
Hani Yehia, Takaaki Kuratate, and Eric and speech acousties. Journal Phonetics, 30(3):555-568. 2002.
Bateson. Linking facial animation, head motion
Heiga Zen, Keiichi Tokuda, and Alan W Black. Statistical parametric speech synthesis. Speech Communication, 2009.
Han Zhang, Tao Xu, and Hongsheng Li. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Shengkui Zhao, Trung Hicu Nguyen, Hao Wang, and Bin Ma. Fast learning for non-parallel many- (o-many voice conversion with residual star generative adversarial networks. Proc. Interspeech 2019, 2019.
Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by ad- versarially disentangled audio-visual representation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 9299-9306, 2019.
Hao Zhu Aihua Zheng, Huaibo Huang, and Ran He. High-resolution talking face generation via mutual information approximation. arXiv preprint arXiv: 18/2.06589, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX BROADER IMPACTS
Our work falls in line with a body of work on content generation that retargets video content, often considered in the context of facial puppeteering. While there exist many applications in entertain- ment, there also exist many potentials for serious abuse. Past work in this space has included broader impact statements (Fried et al. 2019 Kim et al., 2018), which we build upon here.
Our Setup: Compared to prior work, unique aspects of our setup are (a) the choice of input and output modalities and (b) requirements on training data, In terms of (a), our models take audio as input, and produce audiovisual (AV) output that captures the personalized style of the target indi- vidual but the linguistic content (words) of the source input. This factorization of style and content across the source and target audio is key technical aspect of our work. While past work has dis- cussed broader impacts of visual image/video generation, there is less discussion on responsible practices for audio editing. We begin this discussion below. In terms of (b), our approach is unique in that we require only a few minutes of training on target audio+video, and do not require train- ing on large populations of individuals. This has implications for generalizability and ease-of-use for non-experts, increasing the space of viable target applications, Our target applications include entertainmen//educatio and assistive technologies, each of which is discussed below. We conclude with a discussion of potential abuses and strategies for mitigation.
An important application of voice synthesis is voice generation for the speak- ing impaired. Here, generating speech in an individuals personalized style can be profoundly im- portant for maintaining a sense of identity We demonstrate applications in this direction in our summary videos. We have also begun collaborations with clinicians to explore personalized and stylistic speech outputs from physical devices (such as an electrolarynx) that directly sense vocal utterances. Currently, there are considerable data and privacy considerations in acquiring such sen- sitive patient data. We believe that illustrating results on well-known individuals (such as celebrities) can be used to build trust in clinical collaborators and potential patient volunteers.
Assistive Technology:
Creating high-quality AV content is a labor intensive task, often re- quiring days/weeks to create minutes of content for production houses. Our work has attracted the attention of production houses that are interested in creating documentaries or narrations of events by historical figures in their own voice (Winston Churchill, J F Kennedy, Nelson Mandela, Martin Luther King Jr). Because our approach can be trained on small amounts of target footage, it can be used for personalized storytelling (itself a potentially creative and therapeutic endeavour) as well as educational settings that have access to less computational/artisiic resources but more immediate classroom feerbacc/retrainnngg.
Abuse: It is crucial to acknowledge that audiovisual retargeting can be used maliciously, including spreading false information for propaganda purposes or pomographic content production. Address- ing these abuses requires not only technical solutions, but discussions with social scientists, policy makers, and ethicists to help delineate boundaries of defamation, privacy, copyright, and fair use. We attempt to begin this important discussion here. First, inspired by (Fried et al. 2019), we introduce recommended policy for AV content creation using our method. In our setting, retargeting involves two pieces of media; the source individual providing an audio signal input, and target individual whos audiovisual clip will be edited to match the words of the source. Potentials for abuse include plagiarism of the source (e.g., representing somcone elses words your own), and misrepresenta- tion / copyright violations of the target. Hence, we advocate a policy of always citing the source content and always obtaining permission from the target individual. Possible exceptions the pol- icy, if any, may fall into the category of fair us which typically includes education, commentary, or parody (eg., retargeting& a zebra texture on Putin (Zhu et al.) (017))) In all cases, retargeted content should acknowledge itself as edited, either by clearly presenting itself as parody or via watermark. Importantly, there may be violations of this policy. Hence, a major challenge will be identifying such violations and mitigating the harms caused by such violations, discussed further below.
pices - htlp::/:ppggggeaeheeeeeeeeeaaaaa connect inn http:////anneesseeocc/e//////rrr
14

---

Published as a conference paper at ICLR 2021
Audiovisual Forensics: We describe 3 strategies that attempt to identify misuse and the harms caused by such misuse. We stress that these are not exhaustive: (a) identifying "fake" content, (b) data amonymization, and (c) technology awareness. (a) As approaches for editing AV content ma- ture, society needs analogous approaches for detecting such "fake" content. Contemporary forensic detectors tend be data-driven, themselves trained to distinguish original versus synthesized media via a classification task. Such forensic real-vs-fake classifiers exist for both images (Maximov et al.) 2020 Rössler al. 2019; Wang et al. 2020) and audic Because access to code for generating "fake training examples will be crucial for learning to identify fake content, we commit to making our code freely available. Appendix B provides extenstive analysis of audio detection of Exemplar Autoencoder fakes. (b) Another strategy for mitigating abuse is controlling access to private media data. Methods for image anonymization, via face detection and blurring, are widespread and crucial part of contemporary data collection, even mandated via the EUs General Data Protection Regu- lation (GDPR). In the US, audio restrictions are even more severe because of federal wiretapping regulations that prevent recordings of oral communications without prior consent (Stevens & Doyle, 2011]. Recent approaches for image anonymization make use of generative models that de-identify data without degradive blurring by retargeting each face generic identity (e.g., make everyone in dataset look like PersonA) (Maximov et al., 2020). Our audio retargeting approach can poten- tially be used for audio de-identification by making everyone a recording sound like PersonA. (c) Finally, we point out that audio recordings are currently used in critical societal settings including legal evidence and biometric recoonecoonntionn (c.g.. accessing ones bank account via automated recognition of speech over-the- the--hneeehoeeee (Stevens & Doyle ( Our results suggest that such use cases need to be re-evaluated and thoroughly tested in context of stylistic audio synthesis. other
terms, it is crucial for society to understand what information can be reliably deduced from audio, and our approach can be used to empirically explore this question.
Finally, we point out unique property of our technical approach that differs prior approaches for audio and video content generation. Our exemplar approach reduces the re- liance population-scall training datasets. Facial synthesis models trained on existing large-scale datasets- that may be dominated by english-speaking celebrities - may produce higher accuracy on sub-populations with skin tones, genders, and languages over-represented in the training dataset (Menon et al. 2020). Exemplar-based learning may exhibit different properties because models trained on the target exemplar individual. That said, we do find that pre-training individual (but not a population) can speed up convergence of learning on the target individual. Be- cause of the reduced dependency on population-scale training datasets (that may be dominated by English), exemplar models may better generalize across dialects and languages underrepresented in such datasets. In our sunmary video, we present results for stylized multilingual translations (re- targeting Oliver to speak in Mandarin and Hindi) without ever training on any Mandarin or Hindi speech.
Training datasets:
a single
B APPENDIX FORENSIC STUDY
In the previous section, we outline both positive and negative outcomes of our research. Compared to prior art, most of the novel outcomes arise from using audio additional modality. As such, in this section, we conduct a study which illustrates that Exemplar Autoencoder fakes can be detected with high accuracy by forensic classifier, particularly when trained on such fakes. We hope our study spurs additional forensic research on detection of manipulated audiovisual content.
Speaker Agnostic Discriminator: We begin by training a real/fake audio classifier 6 identities from CelebAudio. The model structure is the same as (Serrà et al. 2019). The training datasets contains (1) Real speech of identities: John F Kennedy, Alexei Efros, Nelson Mandela, Oprah Winfrey, Bill Clinton, and Takeo Kanade. Each has 200 sentences. (2) Fake speech: generated by turning each sentence in (1) into a different speaker in those 6 identities. So there total of 1200 generated sentences. We test performance under 2 scenarios: (1) speaker within-the-training set: John F Kennedy, Alexei Efros, Nelson Mandela, Oprah Winfrey, Bill Clinton, and Takeo Kanade; (2) speaker-outtoof-the-trrinnnnn set: Barack Obama, Theresa May. Each identity in the test set contains
vanci httppss//www.bbggggoglll//////// https://www.ssvspoo
at i it news- init intiiiaaiiii ve/ad
15

---

Published as a conference paper at ICLR 2021
Speaker-Agme Agnostic Discriminator
(ccuracy(5)
Speaker Specific Discriminator
Accuracy(18)
speaker within truining set
99.0
style within truining
99.7
real
98.3
real
99.3
fake
99.7
fake
100
speaker training
styll training set
99.7
fake
74.0
fake
100
Table 4: Fake Audio Discriminator: a) Speaker Agnostic Discriminator detects fake audio while agnostic of the speaker. We train real/fake audio classifier on identities from CelebAudio, and test it on both within-set speakers and out-of-set speakers. The training datasets contains (1) Real speech of 6 identities: John F Kennedy, Alexei Efros, Nelson Mandela, Oprah Winfrey, Bill Clinton, (2) Fake speech: generated by turning each sentence (1) into different speaker in those 6 identities. The testing set contains (1) speaker within the training set; (2) speaker of the training set: Barack Obama, Theresa May. Each identity in the testing set contains 100 real sentences and 100 fake ones. For within-set speakers, results show model can predict with very low error rate ( 1%). For out-of-set speakers, our model can still classify real speeches very well, and provide a reasonable prediction for fake detection. b) Speaker Specific Discriminator detects fake audio of specific speaker. We train real/fake audio classifier on a specific style of Barack Obama, and test it on 4 styles of Obama (taken from speeches spanning different ambient environments and stylistic deliveries including presidential press conferences and university commencement speechs). The training set contains 1200 sentences evenly split between real and fake. Each style in the testing set contains 300 sentences evenly split as well. Results show our classifier provides reliable predictions on fake speeches even on out-of-sample styles.
and Takeo Kanade. Each has 200 sentences.
100 real sentences and 100 fake ones. We ensure test speeches are disjoint from training speeches, for the same speaker. Table 4 shows the classifier performs very well detecting fake of within-set speakers, and is able to provide a reasonable reference for out-of-sample speakers.
Speaker-Specifie Discrininator: We restrict training set one identity for specific fake de- tection. We train exemplar autoencoders on 4 different speeches of Barack Obama to get different styles of the same person. The specific fake audio detector is trained on only I style of Obama, and is tested on all the 4 styles. The training set contains 600 sentences for either real or fake. Each style in the testing set contains 150 sentences for either real or fake. Table 4 our classifier provides reliable predictions on fake speeches even on out-of-set styles.
C APPENDIX IMPLEMENTATION DETAILS
We illustrate our method in Fig. 4 in main paper. Below we provide the implementation details of exemplar autoencoder.
C.1 AUDIO CONVERSION
The speech data is sampled 16 kHz. We clip the training speech into clips of 1.6s length, which correspond to 25, 25,600-dimensiona. vectors. We then perform (OPp(hppnnneim et al. 1996) on the raw audio signal with a window size of 800, hop size of 200, and 80 mel-channels. The output of STFT is a complex matrix with size of 80 x 128. We represent the complex matrix in polar form (magnitude and phase), and only keep the magnitude for next steps.
STFT:
The input to the encoder is the magnitude of 80 x 128 Mel-spectrogram, which is repre- sented as a ID 80-channel signal (shown in Figure 4 in main paper). This input is feed-forward to three layers of ID convolutional layers with kernel size of 5, each followed by batch normaliza- tion (Toff & Szegedy, 2015) and ReLU activation (Krizhevsky et al., 2012). The channel these convolutions is 512. The stride is one. There is no time down-sampling up till this step. The output then fed into two layers bidirectional LSTM (Hochreiter & Schmidhuber 1997) layers with both the forward and backward cell dimensions of 32. We then perform a different down- down-sampling for the forward and backward paths with a factor of 32 following (Qian et al.) 2019). The result content embedding is matrix with size of 64 4.
Encoder:
16

---

Published as a conference paper at ICLR 2021
Audio Decoder: The content embedding is up-sampled to the original time resolution of T. The up-sampled embedding is sequentially input to 512-channel LSTM layer and three layers of 512- channel ID convolutional layers with kernel size 5. Each step accompanies batch normalization and ReI.U activation. Finally, the output is fed into two 1024-channel LSTM layers and fully connected layer project into 80 channels. The projection output is regarded as the generated magnitude of Mel-spectrogram m.
Vocoder: We use WaveNet as vocoder (Oord al. 2016) that acts like inverse fourier-transform. but merely use frequency magnitudes. It generates speech signal ã based on the reconstructed mag- nitude of Mel-spectrogram m.
Our model is trained at learning rate of 0.001 and batch size of 8. To train model from scratch, needs about 30 minutes of the target speaker's speech data and around 10k iterations to converge. Although our main structure is straightforward, the vocoder is usually large and complicated network, which needs another 50k iterations to train. However, transfer learning can be beneficial in reducing the number of iterations and necessary data for training purposes. When fine-tuning a new speaker's autoencoder from a pre-trained model, we only need about minutes of speech from new speaker. The entire model, including the vocoder, converges around 10k iterations.
Training Details:
C.2 VIDEO SYNTHESIS
Network Architecture: We keep the voice-conversion framework unchanged and enhance it with an additional audio-to-video decoder. In the voice-conversion network, we have a content encoder that extracts content embedding from speech, and an audio decoder that generates audio output from that embedding. To include video synthesis, we add video decoder which also takes the content embedding as input, but generates video output instead. As shown in Figure 4 (bottom-row) in main paper, we then have an sitiio-tt-uuoovvviiii -iidoo pipeline.
Video Decoder: We borrow architecture the video decoder from (Radford et al., 2016 Zhang et al. 2017). We adapt this image synthesis network by generating the video frame by frame, as well as replacing the 2D-convolutions with 3D-convolutions to enhance temporal coherence. The video decoder takes the unsampled content codes as input. This step is used to align the time resolution with 20-fps videos in our experiments. We nown-saaple it with a ID convolutional layer. This step helps smooth the adjacent video frames. The output is then fed into the synthesis network to get the video result
Training Details: We train an audiovisual model based on a pretrained audio model, with learning rate of 0.001 and a batch size of 8. When fine-tuning from a pre-trained model, the overall model converges around 3k iterations.
D APPENDIX ADDITIONAL EXPERIMENTS
D.1 CELEBAUDIO DATASET
In See 4.1 of main paper, we introduce new in-the-wild CelebAudio dataset for audio translation. Different from well-curated speech corpus such as VCTK, CelebAudio is collected from the In- ternet, and thus has such slight noises applause and cheers in it. This is helpful to validate the effectiveness and robustness of various voice-conversion methods. The distribution of speech length is shown in Fig. 5 identities is:
We select subset of 20 identities for evaluation in our work. The list of 20
Native English Speakers: Alan Kay, Carl Sagan, Claude Shannon, John Oliver, Oprah Winfrey, Richard Hamming, Robert Tiger, David Attenborough, Theresa May, Bill Clin- ton, John Kennedy, Barack Obama, Michelle Obama, Neil Degrasse Tyson, Margaret Thatcher, Nelson Mandela, Richard Feyman
Non-Native English Speakers: Alexei Efros, Takeo Kanade
Computer-generatt Voice: Stephen Hawking
17

---

Published as a conference paper at ICLR 2021
20-
18-
14-
Count
12-
2-
500
1000
1500
2000
3000 Length of Speeches (sces)
2500
3500
4000
4500
5000
5500
6000
6500
7000
Figure 5: Data Distribution of CelebAudio: CelebAudio contains 100 different speeches. The average speech length is 23 mins. Each speech ranges from mins (Hillary Clinton) to 113 mins (Neil Degrasse Tyson).
D.2 VERIFICATION OF REPROIECTION PROPERTY
In Sec 3.2 of main paper, we describe reprojection property (Eq. 6 of our exemplar autoencoder: the output of out-of-sample data i is the projection to the manifold M spanned by the training set.
D(E())) s arg min Error(s, where mEM
= (D((())) R.)
(6)
This is equivalent to two important properties: (a) The output lies the manifold M. (Eq. 7 (b)
The output is closer to the input & than any other point on manifold M. (Eq.
D(E($)) M, where
M (D((())=) Rd).
Error (D((())), 2) < Error(r, .),
Vr E M.
(8)
To verify these, conduct an experiment (Table the parallel corpus of VCTK et al. 2016). We randomly sample 2 sets of words (normal and tongue-twister) spoken by speakers (A
and B) from VCTK dataset. We train exemplar autoencoder A's speech, and input B's words (wB) to get B- To verify Eq. 7 we train a speaker-classifier (as Serra et al. (Serrà et al.) 2019)) on and B's speech. As the fifth column in Table we report the likelihood that wB-A A regarded as A's words. To verify Eq. 8 we calculate Error( D(E())). .t) as the second column in Table and Error(x, .) as the third and Tourth columns in Table 5
The results show that our conjecture about
the reprojection property is reasonable.
D.3 ABLATION ANALYSIS OF EXEMPLAR TRAINING
In the main paper, we posit that is easier disentangle style from content when the entire training dataset consists of single style. To verify this, we train autoencoder with following setup: (1) speeches from 2 people; (2) speeches from 3 people; (3) 4 speeches with different settings from person. We contrast it with exemplar autoencoder trained using speech from I person using A/B testing. Our exemplar results are preferred 71.9% times over (1), 83.3% times over (2), and 90.6% times over (3).
D.4 ABLATION ANALYSIS OF REMOVING SPEAKER EMBEDDING
We point out that generic speaker embeddings struggles to capture stylistic details of in-the-wild speech. Here we provide the ablation analysis of removing such pre-trained embeddings.
18

---

Published as a conference paper at ICLR 2021
Normal words
wB
|| --B-- wal
min WE*x
||or
we
arg min ||w we
Likelihood($))
"many"
18.27
28.33
"many"
100
"complicate"
16.15
20.05
"complicate" "idea""
59.4
"iden"
18.54
26.57
100
"about"
14.20
26.51
"about"
"rainbow"
15.68
27.63
"rainhow" "
100
"form"
13.08
27.46
"form"
"people"
18.94
"people"
"look"
"look"
"find"
15.88
32.87
"find"
"it"
7.54
29.11
"It"
"please"
17.46
23.11
"please"
100
"call"
13.93
13:25
"call"
97.6
"stella"
14.45
15.77
"stella"
82.4
Average
14.43
24.77
(b) Tongue-twister words
min MCA.
||
wnll
meanwwer ||
arg min 1|
(ikelihood(())
WEE
"please"
17.46
23.11
53.93
"please"
100
"tinngs"
21.59
19.87
50.95
"things"
81.8
"theae"
25.22
21.26
82.00
"these"
55.6
"cheese"
20.34
"cheese"
"three"
24.39
22.13
61.07
"three"
"mix"
16.61
55.77
"mix"
100
"six"
20.43
16.75
66.85
"six"
100
"thick"
21.21
18.38
50.16
"thick"
92.3
"kids"
16.98
16.65
48.78
"kids"
100
"is"
16.84
16.25
70.88
"Sc"
78.0
Average
20.23
19.14
59.36
Table 5: Verification of the reprojection property: To verify the reprojection property that Eq. (main paper) describes, we randomly sample sets words (Normal and Tongue-twister) spoken by 2 speakers (A and B) in VCTK dataset. For the autoencoder trained by A, all the words spoken by are sat-lee data. Then for each word WB, we generate WB-A which the projection to A's subspace. We need to verify 2 properties (i) The output wn-A lies in A's subspace; (ii) The autoencoder minimizes the input-output error. To verify (i), we train a speaker classification network speaker A and B, and predict the speaker of WB-AA- We report the softmax output to show how much likely WB-AA is to be classified as A's speech (Likelihood in the table). To verify (i), we calculate (1) distance from Wy WB-+A (2) minimum distance from wn to any sampled words by A. For normal words, the input wp is much closer (sometimes as close as) to the projection WB-AA than any other sampled point A's subspace. Furthermore, such minimum is reached when the content is kept the same. For tongue-twister words, which are more confusable, we additionally calculate the mean distance from A's subspace to Wp. Distance from wp WB close the minimum, and much smaller than the mean. This empirically suggests that nonlinear autoencoder behave similar to their linear counterparts (i.e., they approximately minimize the reconstruction error of the out-of of of-mle input).
19

---

Published as a conference paper at ICLR 2021
VCTR Veaux
2016
Similarity
Consislency
Chou al. Chou al
2019
57.0
491.1
Auto-VC Qaan
2019
with WaveNet
98.5
408.8
Ours
with WaveNet
99.6
420.3
without WaveNet
97.0
420.3
Table Ablation Analysis of WaveNet: We replace the WaveNet vocoder with Griftin-Lim tradi- tional vocoder, and compare method with two zero-shot methods: (1) Au without Wavenet, and (2) Chou et al. (Chou et al., 2019). We measure Voice Similarity by ppeaker-clasiffictte racy (SCA) criterion, where higher is better. The results show approach without WaveNet still outperform other zero-shot approaches not using WaveNet. We also report MCD same as Table 2 in main paper as MCD is only related to Freq. features. For reference, also list the results of "with Wavenet" experiments.
accu-
We perform new experiments that fine-tune a pretrained Auto-VC for each specific speaker (ex- emplar training) but still keeps the pre-trained embedding, and find it preforms similarly to the multi-speaker fine-tuning in Table 3 (main paper). Our outputs still preferred 91.7% times over it.
D.5 ABLATION ANALYSIS OF WAVENET
We adopt neural-net vocoder to convert Mel-spectrograms back to raw audio signal. We prefer WaveNet (Oord et al., 2016) to traditional vocoders like CriffinLLimGrrifiie & Lim 1984). since Wavenet one of the state-of-the-an methods. The existence of WaveNet in our structure makes our method end-to-end trainable and can safely be considered a part of exemplar autoencoder (no special adaptation required). While Auto- VCCoan et al.] 2019) and VQ- NQ-VAB(van den Oord al.] 2017) also use WaveNet, we substantially outperform them on Celeb-Audio dataset.
Here we add the ablation analysis of WaveNet in Table 6 We remove the WaveNet vocoder from our approach and Auto and replace it with Griffin-Limi We also compare with Chou et al. (Chou ct al. 2019), which does not use neural-net vocoders. The results show our model can also generate reasonable outputs with traditional vocoders, and yet outperform other zero-shot approaches not using WaveNet.
D.6 ABLATION ANALYSIS OF VIDEO SYNTHESIS
In main paper, we propose "exemplar autoencoders" and prove the effectiveness of jointly training andio-to-audio-videe pipeline based on pre-trained audio model. To further verify the helpfulness of audio bottleneck features and finetuning from audio model video synthesis, we conduct two ablation experiments on audio-to-vin tovvieeo translation, and compare with ours.
(a) Train only the audio-to-videe translator. (First Baseline in Table
(b) Jointly train audio decoder and video decoder from scratch. (Second Baseline in Table 7
(c) First train an autoeneoder of audio, then train video decoder while finetuning the audio part. (Ours)
From the results in Table 7 ours outperforms (b), which indicates the effectiveness of finetuning from pre-trained audio model. And (b) also outperforms (a). This implies the effectiveness of audio bottleneck features.
D.7 DETAILS OF HUMAN STUDIES
All the users of AMT were chosen have Master Qualification (HIT approval rate more than 98% for more than 1, 000 HITs). We also restricted the users to be from United States to ensure English- speaking audience.
20

---

Published as a conference paper at ICLR 2021
" "end"
"want"
Speech2Vid
LipGAN
Ours
Zoom-in View
Speech2Vid
LipGAN
Ours
Speech2Vid
LipGAN
Ours
Figure 6: Visual Comparisons of Audio-to-Vide Synthesis: We contrast our approach with Speech2vid (Chung et al. (017) (first row) and LipGAN (KR et al. 2019) (second row) using their publicly available codes. While Speech2 synthesize new sequences by morphing mouth shapes, LipGAN pastes modified lip region on original videos. This leads to artifacts (see zoom-in views bottom row) when morphed mouth shapes are very different from the original or in case of dynamic facial movements. Our approach (third row), however, generates full face and does not have these artifacts.
VoxCeleb
Finetune
Audio Decader
MSEJ
PSNR1
77.844 15.612
29.304 0.856
Baselines
77. 12.577
29.315 0.701
76.401 12.590
29.616
Table 7: Ablation Analysis of Video Synthesis: To verify the effectiveness of pre-trained audio model and audio decoder, we construct 2 extra baselines: Train only the audio-to-vice -o-ideo translator (1st Baseline). b. Jointly train audio decoder and video decoder from scratch (2nd Baseline), And we compare these 2 baselines with our method: First train an autoencoder audio, then train video decoder while finetuning the audio part. From the results, we can see the performance clearly drops without the help finetuning and audio decoder.
21


Reviews:
[{"review_id": "x7_Sw1_PWnP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Interesting paper covering a lot of ground; exposition could be tightened/clarified?\n\nSuper interesting topic; strong references to prior work (I add a reference at the very end that I think you'd be interested in, Hani Yehia et al. (2002).)\n\nI found the paper slightly verbose -- I'm not sure i fully \"got\" the central concept of Exemplar Autoencoders. Looking at Section 3, with that in the title, I was waiting to find the canonical definition of E.A.'s, but don't quite see it -- it seems somewhat unclear given the many variants of auto-encoders discussed. The discussion in Section 2, and e.g. Figure 2c, help clarify what the author's mean by EA's. What I'm not fully sure about is, are EA's basically what I would call Auto-encoders? Fig 3c is labeled \"Exemplar Autoencoder\", but then the caption for 3c mentions \"Non-linear autoencoder\", which is confusing. The caption starts with \"Figure 3: Our insights for Exemplar Autoencoders, \" leading me to think that actually all three of these sub-figures are variants of Exemplar Autoencoders. Maybe they could clarify their terminology or re-read the draft from the perspective of someone who is not as \"close\" to the work as they are.\n\nThe contrast with zero-shot conversion is very interesting -- but here too I feel I am somewhat missing the explanation of the essence of these zero-shot methods -- though it is possible most readers do not need any additional explanation. In particular, when I see Fig 2a, I'm wondering how the content and speaker embedding vectors are trained (and on what data), and the text doesn't quite clarify that for me.\n\nMore specific comments follow:\n\n\"(a) Zero-shot conversion that learns a generic low-dimensional embedding from a training set that _are_ designed ...\": \"are\" --> \"is\".\n\n\" In supp material, ...\" --> \"In supplementary material, ...\"\n\nDefine \"retargeting\"?\n\nTable 2: Use up & down errors, as you did in a later table, to indicate whether higher/lower is better.\n\nNo conclusion? It seems you have one, but it's \"Discussion\", as a subsection of the last experiments section.\n\nThe references seem very comprehensive, but I urge you to look at, and cite, Yehia et al. 2002, \n\nYehia, Hani Camille & Kuratate, Takaaki & Vatikiotis-Bateson, Eric. (2002). Linking facial animation, head motion and speech acoustics. Journal of Phonetics. 30. 555-568. 10.1006/jpho.2002.0165. \n\nas a seminal study in this area.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZiG-N9jdFhb", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose a generic system for performing one-shot audiovisual synthesis from only one small sample. The results are impressive for in-the-wild speech synthesis and their approach could have a broader impact in the community. \n\nStrengths:\n + One shot audiovisual synthesis for a target speaker.\n + The publication of a new dataset for AV synthesis evaluation.\n + Comprehensive analysis\n\nWeaknesses: \n - No theoretical novelty. It seems much of the benefits of the approach comes from the extra data and training procedure. \n\nOther comments:\n\nIn Sec 3.2. Autoencoders as projection operators, the authors here make it sound that they are the first ones that noted that autoencoders can capture the data-generating distribution. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "64xXqDOhwP7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper covers a very interesting topic and method to convert any input speech to many audiovisual syntheses via exemplar autoencoders. The manuscript is well written and presented. It is easy to follow the concept. However, there are a few major concerns. \n\nPros: This approach is novel. The presented approach is unsupervised, which makes it more practical. This approach required only a small amount of data to train the exemplar autoencoders when learning specialized models tailored to particular target data.\n\ncons: \n1- After listening to the provided sample demo files. I think this approach is still in an immature status. The generated output speech is highly distorted, without knowing the input speech, it is hard to understand the generated speech. Hence, more work is required to improve on the audio decoder part. \n2- It is not mentioned if the data used for training is all speech of native English speakers or not.\n3- Any reasons authors did not use the x-vector or i-vectors, which are proper for the speaker characterization, and instead used mel features? \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes and investigates an approach for audiovisual synthesis based on the so-called exemplar autoencoders.  The proposed approach is shown to be able to convert an audio input to audiovisual outputs using only very small amount of training data.  All reviewers consider the paper interesting with a lot of potentials in a variety of applications and appreciate the novelty of the work in this domain.  But there are also concerns on the technical presentation and the quality of the samples in the demo.  The authors addressed most of the concerns in the rebuttal but agreed that the quality of the results still had room for further improvements.  Overall, the work presented is interesting. The paper can be accepted. 
Decision: Accept (Poster)