Acoustic Neighbor Embeddings
Authors: Woojay Jeon
Venue:  (2021)

Abstract: This paper proposes a novel acoustic word embedding called Acoustic Neighbor Embeddings where speech or text of arbitrary length are mapped to a vector space of fixed, reduced dimensions by adapting stochastic neighbor embedding (SNE) to sequential inputs. The Euclidean distance between coordinates in the embedding space reflects the phonetic confusability between their corresponding sequences. Two encoder neural networks are trained: an acoustic encoder that accepts speech signals in the form of frame-wise subword posterior probabilities obtained from an acoustic model and a text encoder that accepts text in the form of subword transcriptions. Compared to a triplet loss criterion, the proposed method is shown to have more effective gradients for neural network training. Experimentally, it also gives more accurate results with low-dimensional embeddings when the two encoder networks are used in tandem in a word (name) recognition task, and when the text encoder network is used standalone in an approximate phonetic matching task. In particular, in an isolated name recognition task depending solely on Euclidean nearest-neighbor search between the proposed embedding vectors, the recognition accuracy is identical to that of conventional finite state transducer(FST)-based decoding using test data with up to 1 million names in the vocabulary and 40 dimensions in the embeddings.

Under review as a conference paper at ICLR 2021
ACoUSTIC NEIGHBOR EMBEDDINGS
Anonymous authors
Paper under double-blind review
ABSTRACT
This paper proposes a novel acoustic word embedding called Acoustic Neigh- bor Embeddings where speech or text of arbitrary length are mapped to a vector space of fixed, reduced dimensions by adapting stochastic neighbor embedding (SNE) to sequential inputs. The Euclidean distance between coordinates in the embedding space reflects the phonetic confusability between their corresponding sequences. Two encoder neural networks are trained: acoustic encoder that accepts speech signals in the form of frame-wise subword posterior probabilities obtained from an acoustic model and text encoder that accepts text in the form of subword transcriptions. Compared to triplet loss criterion, the proposed method is shown to have more effective gradients for neural network training. Experi- mentally, it also gives more accurate results with low-dimensional embeddings when the two encoder networks are used in tandem a word (name) recognition task, and when the text encoder network is used standalone in an approximate phonetic matching task. In particular, in an isolated name recognition task de- pending solely on Euclidean nearest-neighbor search between the proposed em- bedding vectors, the recognition accuracy is identical to that of conventional finite state transducer(FSTT) -based decoding using test data with up to million names in the vocabulary and 40 dimensions in the embeddings.
INTRODUCTION
Acoustic word embeddings (Levin et al. 2013] Maas et al. 2012 are vector representations of words that capture information on how the words sound, as opposed to word embeddings that capture information on what the words mean.
A number of acoustic word embedding methods have been proposed, applied to word discrimina- tion (He et al. 2017 Jung et al.) 2019), lattice rescoring in automatic speech recognition (ASR) keyword search (Settle al., 2017) or detec- tion (Chen et al. 2015). Previous works have also applied multilingual acoustic word embeddings (Kamper et al. 2020 Hu al. (020) zero-resource languages, and acoustic word embeddings or acwusticallyyyae.... -rounndd word embeddings to improve acoustic-to-word (A2W) speech recognition
(Bengio & Heigold 2014), and query-by-exampe
Settle al. (2019): Shi et al. (2020).
Recently, triplet loss functions (He al. 2017 Settle et al. 2019 Jung et al. 2019) have been used train two neural networks simultaneously: an acoustic encoder (()) network" that accepts speech, and text encoder 9(-) network that accepts text as the input. By training the two to transform their inputs into common space where matching speech and text get mapped to the same coordinates, and can be used in tandem in applications where speech utterance is compared against a database of text, or vice versa. They can also each be used standalone, general-purposs word embedding network that maps similar-sounding speech (in the case of f) or text (in the case of g) to similar locations in the embedding space.
Note that in stricter choices of terminology, one may use the term acoustic word embedding to refer only to the output of g. and speech embedding (Algayres et al. 2020 to refer to the output of f.
An obvious application of such embeddings is highly-scalabe isolated word (or name) recognition (c.g. in a music player app where the user can tap the search bar to say a song or album title), where
It is interesting to note that the basic notion of "memorizing" audio signals in fixed dimensions can be traced back early as Longuet-Higgnt (1968)

---

Under review as a conference paper at ICLR 2021
given speech input is mapped via f to embedding vector f, which is then compared against a database of embedding vectors (gi.... ...... ). prepared via g, that represents a vocabulary of N words, to classify the speech. Using the Euclidean distance, the classification rule is:
arg min |f Byl1?
(1)
ISSSN
where ||. is the L2 norm. Vector distances have been used in the past for other matching problems (e.g. [Schroff et al. 2015). For speech recognition, a rule like (T) is interesting because it can be eas- ily parallelized for fast evaluation modern hardware (e.g. Garcia et al. (2008), and the vocabulary can also be trivially updated since each entry is a vector. If proven to work well in isolated word recognition, the embedding vectors could be used in continuous speech recognition where speech segments hypothesized to contain named entities can be recognized via nearest-neighbor search, par- ticularly when we want an entity vocabulary that both large and dynamically updatable. However, none of the aforementioned papers have reported results in isolated word recognition.
The main contribution in this paper is a new training method for f and g which adapts stochastic neighbor embedding (SNE) (Hinton & Roweis 2003) to sequential data. It will be shown by anal- ysis of the gradients of the proposed loss function that it is more effective than the triplet loss for mapping similar-sounding inputs to similar locations in vector space. It will also be shown that the low-dimensional embeddings produced by the proposed method, called Acoustic Neighbor Embed- dings (ANE), work better than embeddings produced by triplet loss in isolated name recognition and approximate phonetic matching experiments. This paper also claims to be the first work that successfully uses simple La distance between vectors to directly perform isolated word recognition that can match the accuracy of conventional FST-based recognition over large vocabularies.
One design choice in this work for the acoustic encoder f is that instead of directly reading acoustic features, a separate acoustic model is used to preprocess them into (framewise) subword posterior probability estimates. "Posteriorgrams" have been used in past studies (e.g. Haazen al. (2009) for speech information retrieval. In the proposed work, they allow the f network to be much smaller, since the task of resolving channel and speaker variability can be delegated to a state-of-the ooet----r----tt-tte ASR acoustic model. One can still use acoustic features with the proposed method, but in many practical scenarios an acoustic model is already available for ASR that computes subword scores for every speech input, so it is feasible to reuse those scores.
As inputs to the text encoder g. this study experiments with two types of text: phone sequences and grapheme sequences. In the former case, a prapheme-to-phoneee converter (G2P) is used to convert each text input to one or more phoneme sequences, and each phoneme sequence treated as separate "word." This approach reduces ambiguities caused by words that could be pronounced multiple ways, such as "..",,.A", which could be pronounced as (in ARPABET phones) [aa r p aa], or [ey p iy ey]. or [ey d aa t d aa p iy d aa t ey] (pronouncing every """ "dot"), In the latter case using graphemes, more errors can occur in word recognition because a single embedding vector may not capture all the variations in how a word may sound. On the other hand, such a system can be more feasible because it does not require a separate G2P.
Also note that while we use the term "word emhedding" following known terminology in the litera- ture, a "word" in this work can actually be a sequence of multiple words, such as "John_ W_ W_Smith" or "The Hilton Hotell" in the name recognition experiments in Section 5
REVIEW OF STOCHASTIC NEIGHBOR EMBEDDING
In short, stochastic neighbor embedding (SNE) (Hinton & Roweis 2003 is a method of reducing dimensions in vectors while preserving relative distances, and is popular method for data visual- ization. Given a set of N coordinates {xi..... xxv.. J. SNE provides a way to train a function f(-) that maps each coordinate x; another coordinate of lower dimensionality f, where the relative distances among the x; 's are preserved among the corresponding ft's.
The distance between two points x; and x, in the input space is defined as the squared Euclidean
distance with some scale factor o;:
||x;
x 111
de
(2)
20
2

---

Under review as a conference paper at ICLR 2021
This distance is used to define the probability of x, choosing x, as its neighbor in the input space:
exp (-
Pij Enee exp (
(3)
The corresponding "induced" probability in the embedding space is
exp (-1 f. f)
4 Enees exp (- [, 12)
(4)
The loss function for training f is the Kullback-Leibler divergence between the two distributions:
Ly Σ Pij
log Pij
(5)
( gij
which can be differentiated obtain oC, [ - f,) 1 - Dj + Pys 9jz)-
0f
Hinton & Roweis (2003)'s 's cogent interpretation of the above equation "a sum of forces pulling f, toward f,, or pushing away depending on whether j is observed to be a neighbor more or less often than desired" is seed for other arguments that will be made in the present paper.
3 PROPOSED EMBEDDING METHOD
3.1 METHOD DESCRIPTION
Consider a training set of N speech utterances. The n n'th utterance characterized by (Sn, Xns Yn) where S, is the audio signal containing or more words of speech, X,, is sequence of subword posterior probability vectors for S,, and Y, is sequence subwords pertaining to the reference transcription of Sn.
X, X1 Xg. xr] is obtained from an acoustic model such as a DNN-HMM (Deep Neural Network-Hiddenn Markov Model) system, where the d'th element of x, is an estimate of the poste- rior probability of subword Wa occurring at frame t given the speech signal S,, There is a subword posterior vector for every speech frame, and T is the number of frames in the utterance. While monophone posteriors are used in this study, other subword posteriors could be used, such as HMM state grapheme posteriors. Also note that raw acoustic features could be used for X, (see Ap- pendix A where we used MFCCs), but posteriors were used here for reasons mentioned in Section
The subword reference transcription of cach utterance is also obtained, such as the phone sequence [k ao ih ng], or the grapheme sequence r sin g] for the word "crossing," by force-aligning the utterance with its manual word transcription using ASR with a pronunciation dictionary. This sequence represented as sequence of 1-hot vectors Y, = [y y2. ... .Yml where M is the number of subwords in the sequence. Each subword usually occupies least one frame of speech, so T for every utterance.
Note that the "subwords" used for the posteriors in X,, need not be the same as the "subwords" used for the transcriptions in Y, (for ANE-g in Section 5 X,, a monophone posterior vector sequence, whereas Y, is grapheme transcription).
The idea in this paper is to train an acoustic encoder neural network f(-) that will transform each posterior vector sequence to single fixed-dimension embedding vector f. f(X,) such that predefined relative distances between data samples the space X,, will be preserved the space of f, in manner similar SNE.
First, since each X; is a sequence of vectors, the Fuclidean distance in 2 does not make sense in input space. Instead, we define the distance between X, and X, based on whether their transcriptions are an exact match in the space of Y:
0
Y; Y,
dis { x
clse
3

---

Under review as a conference paper at ICLR 2021
Previous work in "supervised Shenn et al. 2013) also used input distance metric incor- porating label information. However, they were applied to ixxe-dimension point inputs where a sequence of vectors would be mapped to another sequence of n vectors, not variable-length vector sequences where a sequence of n vectors is mapped to a single embedding vector as in this paper.
It is worth noting that alternate forms of dy based dynamic time warping between X, and X; (in a manner similar to Hazen et al. )209)) or Y; and Y, with heuristic insertion, deletion, and substitution costs were also tried, but the binary distance above gave much better accuracy. One may expect that smooth "degrees" of dissimilarity in the input space may work better than a hard binary dissimilarity, but there an intuitive way to understand how the model can learn such "degrees" from binary labelings as long as there is a sufficient amount of training data. See Appendix B for further discussion on the edit distance as well as justification for the binary distance in 77)
The distance in results in the following probability for 33:
1/c,
Y
Pij
(8)
0
else
where oj is the number of utterances (other than the i'th) that have the same subword sequence Y,
We use the same induced probability in (4) for the embedding space. Since this probability is ex- plicitly based on the L2 norm, all comparisons we do in the embedding space (e.g. word recognition using | or phonetic distance computations in Table [ are done using the Lg norm.
The loss function for training the acoustic encoder f is Y described in (5). Once we have fully trained f. we simply train the text encoder g such that its output for every subword sequence Y, will match closely as possible the output of f for the corresponding subword posterior sequence X., where we keep f fixed. A simple mean square error loss proves to be sufficient for this purpose:
Cgy Σ llo(Y.)) f(X.))1
(9)
3.2 TRAINING STRATEGY FOR THE I ACOUSTIC ENCODER
In practice, the training data for the proposed system is much larger (millions of utterances) than the original data (thousands of data points) for which SNE (Hinton Roweis 2003) was proposed. Hence, we sample the data into equally-sized "microbatches" for computing the loss in (5) Each "mierobatch" consists of N data samples, each sample characterized by a subword posterior se- quence and subword transcription (X,Y) Y). A fixed number of microbatches then form a minibatch (as in "minibatch training"), ) and the minibatch loss is the average loss of the microbatches therein. Because the utterances are extremely diverse, forming each microbatch via purely random sampling often results in all the subword sequences being different from each other and pij = 0 everywhere in (8. In such case, the loss in (5) would be O, so all gradients would be O, and the microbatch would have no effect on training. To avoid wasting training time on defunct microbatches, for every mi- crobatch designate (Xo. Yo) the "pivot," and artificially search for at least one sample (X,, Y,) in the training data that satisfies Y, = Yn and insert it in the microbatch. The other samples (with a different subword transcription from Yo) in the microbatch are chosen purely randomly. For further simplicity, we compute the loss over only subset of i, pairings in (5). by fixing i to 0:
C = Σ Poj log Poj
(10)
quj
3.3 PHONETIC CONFUSABILITY REFLECTED IN THE g VECTORS
In the proposed method, clear intuition exists on how the Euclidean distance between two embed- ding vectors gi and g, can directly reflect the acoustic confusability of their corresponding (purely text) subword sequences Y, and YG. Since the text encoder g directly mirrors via ( the embeddings generated by the acoustic encoder f. the g vectors generated purely from text will mirror the knowledge learned by on how the text actually sounds.
Consider the two vowels [ae] (as in "bat") and [ch] as in ("bet"). Since they sound similar, it is likely that their posteriors will behave similarly; if one scores high, the other will also score high. On the
4

---

Under review as a conference paper at ICLR 2021
Table I: Euclidean distance between g vectors (from ANE-p-40 in Section 5 for different pairs of words, computed by the text encoder from pure text. The differences in distances are consistent with the intuitive phonetic confusability between the words.
Words
Phone sequences fed to
Distance
g network
between g's
Jackson
[] ae k s ah n]
0.0478
Jeckson
eh k ah
Jackson
ae s ah n]
1.331
Sackson
[s ae s ah n]
game of thrones game of drones
[g ey
f ow z] [g ey m ax f r ow z]
0.
game of thrones fame of thrones
Ig ey m ax f th ow z] [f ey m ax f r ow n
1.077
other hand, the consonants [j] and [s] sound distinctly different, and therefore their scores will tend to be opposite of each other; if one scores high, the other will score low, and vice versa.
Now imagine acoustic instances of "Jackson" ((j. ae, k, s, ah, nl). "Jeckson" (lj. eh, k, s, ah, n)), and "Sackson" " ([s, ae, k, s, ah, n|) appearing as training samples for f. Both the second and third word are only one phoneme away from "Jackson", but the phone posteriorgram sequence for "Jeckson" will be close to that of "Jackson" due to the scores for [ae] and [eh] trending similarly, so their embeddings will also be inevitably similar. On the other hand, the posteriorgram sequence for "Sackson" will be different from that for "Jackson" because the scores for [j] and [s] tend to be mutually exclusive, so their embeddings will be different.
Now consider the embeddings obtained from pure text inputs "Jackson". "Jeckson", and "Sack- son". As we can see Ta "Jackson" much closer to "Jeckson" than it to "Sackson" in the embedding space, which is consistent with what actually sounds more similar. Similarly, "game of thrones" is closer to "game of drones" than "fame of thrones."
3.4 THE IMPORTANCE OF NORMALIZZTION
It is interresting
what would happen if no normalization were done in (3) and ( The scores in g would be trivial (1 or 0) instead of scaled (1/c, or 0) binary values, and the computation would be significantly reduced with denominator in Although this simplicity may be tempting, the
resulting gradient is
of, Of,
4
Σ (f. fy)Pij.
(11)
Comparing this with (6) (or later with 12 the gradient, meaning the similarity in the embedding space no longer plays a role in weighting the gradient. Only the similarity in the input space (Pij) influences the gradient weight, the fundamental notion of preserving relative distances is no longer being enforced.
can that di has disappeared entirely from
COMPARISON WITH THE TRIPLET LOSS
Some insight into how the proposed method compares with the basic triplet loss criterion, which has been implemented in various forms (Kamper et al. [0166] Settle & Livescu 2016; He et al.) (0117), can be found by inspecting the gradients the two methods loss functions used in backpropagation.
For ANE's f network, we can apply 8 to c to obtain the gradient of the loss function. The summation can be split into the sum over the samples with the same subword sequence as i.e.
J = (j: Y, Y,), and the rest of the samples, i.e., J Y Y,}:
0C, 2 Σ (f.
f,) (1
9ís
2 Σ (f. f2) qj.).
(12)
of, JEJT
C
JEJ

---

Under review as a conference paper at ICLR 2021
Now, let consider the triplet loss, placed in the same context ANE. For every given triplet of
posterior sequences (Xo: Xm. X,) with a corresponding triplet of subword sequences (Yo Ym. Yn).
where Yo_ Ym and Yo # Y, we have the following basic hinge loss function (He et al. 2017
Settle et al. 2019 Jung et al. 2
Lsingl--rrp max {0, Sim(fo. gm) + Sim(fo, gx)):
(13)
where a some constant and Sim(f,, gj) is some pre-defined similarity function between f, and gi- The idea is to make the embeddings attract each other if their utterances have identical subword sequences, and repel each other if the subword sequences are different.
Note that the state-of-the-art multi-view triplet loss (He et al. 2017 Jung et al. 2019) is actually composed of two loss functions as in 2T the scope of this paper (see Appendix A for further discussion).
We will not explore this case, however, as this is beyond
In (13), one can see that the max operator is essentially a "data selection" mechanism. For "light offender" triplets, whose Sim(fo, gm) is high and/or Sim(fo, gn) is low, the loss is fixed to 0, mean- ing they do not give rise to any gradients in the training, and are in effect dropped from the training data. Instead, the "heavy offender" triplets, whose Sim (fo, gm) is low and/or Sim(fo, gn) is high, are the ones kept for training. The margin a controls the border drawn between "light" and "heavy."
To examine what happens with the heavy offenders, can ignore the max operator and set in (13). Let us consider a "batch" triplet loss that is summed over all the samples in our set of N speech utterances:
Lup =
Σ f
si Sim (f,, (f..f.) gy):
(14)
where we have defined
if, YY,
8ij
(15)
+1
else
Also, since the training tries to make f, as close as possible to gy for every make the approxi- mation g; 2 f,, resulting in
Cuip Σ
stggy
(16)
in
where %/ & ff)
A variety of possibilities exist for Sim(f, gj) in 13 If we assume a Euclidean-distanc--bsdd form similar to that is bounded between 0 to 1,
Sim(f, gj) exp {-1f6, g, |)]]
(17)
it can be determined that the gradient of the loss in 16) is
0mmp
Σ (f. - 13)4ky - 4 Σ (f. - f5)dg-
(18)
dr,
jEJ!
JEJ;
Comparing 18 with 12), we notice a key difference in the contribution of the samples in J to the gradient. In the case of the triplet loss, the contribution to the gradient is amplified as the similarity Ky increases, causing more perturbation
the model parameters during backpropagation. This behavior is counterintuitive because high & means the embedding for X, and X, are already almost the same, which is exactly what we're trying to achieve for j E J so there no need to change them further. In contrast, in the ANE loss in (12) one can see in the summation over J+ that as dú gets higher, we amplify the gradient less and therefore perturb the model parameters less, which is consistent with intuition.
Alternate similarity functions for [77 can also be tried. If Sim g,) A 3 |f. - g,11 the gradient in (88) becomes -4 E, Sty (f; - f)), and if Sim f., 8,) 2 E //gs/([IGG) ||g,]]) (cosine similarity), the gradient is - Σ, sij (f.g/y/ || E/[ ||E, ID}. and we can make similar arguments as above that they are less effective than the proposed method.
Note that other examples of decomposing loss functions into attractive and repulsive weights can be found in the literature (e.g. Carreira-Perpintt (2010)).

---

Under review as a conference paper at ICLR 2021
In summary, both the triplet-loss-basee method and the proposed method ANE seem to share the same basic notion of pulling vectors closer if they represent the same word and pushing them apart if not. However, ANE has the added subtlety of pushing pulling with more "measured strength" based on how good the embeddings currently are. It is possible that part of this effect could be achieved with the triplet loss as well by adjusting the margin a in (3), to change the range of "heavy offender" triplets we train on. Even then, note that the counterintuitive weighting discussed above still holds for the heavy offenders, so the method is rather crude. Previous studies like (Kamper et al. 2016) searched over range of values to find the best margin. In this regard, it is possible that ANE is more elegant than the triplet loss in that it can find optimal ways to push and pull with less need for additional manual intervention. Data sampling strategies (Riad et al., (018)) have also been proposed to further refine the accuracy of tripler-based embeddings. Whether such strategies would also benefit ANE is a question we leave to future investigation.
5 NAME RECOGNITION EXPERIMENTS
Three l-g encoder pairs were trained, then tested using the Euclidean nearest-neighhor matching rule in (T). ANE-p is the proposed system using monophone posteriorgram sequences for every X,, and monophone sequences for every Yn in Section and ANE-g is the same but using graphemes for Y, Trip-p uses the same inputs and outputs as ANE-p but is trained by a triplet distance per Section 2 using the hinge triplet loss in (13) and the bounded form of the L2 distance in (17). For all systems, both f and g were Bi-LSTM networks, with 51 input dimensions (1 dimension for each monophone, including a "silence" monophone) for f and 50 input dimensions (excluding the "silence") for 9. All if networks had 2 layers and 100 states in each direction, and all g networks had layer and 200 states in each direction (with the exception of ANE-p-40, which had 300 states), with an additional linear layer applied to the last output of every sequence to produce the embedding vector. Training was done using Adam (Kingma & Ba 2015) optimization for all encoders, with an initial learning rate of 0.001.
For training data, speech utterances of varying lengths were extracted from large set of proprietary data, and their monophone posterior vector sequences and phonetic transcriptions as described in Section 3.1 were obtained using an ASR. The lengths of the utterances were randomly chosen so that the overall distribution of the number of phones per utterance roughly matched that of a database of music titles and person names. The process resulted in training data of 4.2M utterances and a development data set (used to stop training) of 1.4M utterances. The average utterance length in the training data was 885 ms (so the mean of T in Section 3.1 was 88.5 frames), and the average number of phones per utterance (the mean of M in Section 3.1 was 8.56. From the training set, 678K microbatches (160 utterances per microbatch, and 32 microbatches per minibatch) were generated for training the f for ANE-p, and 783K microbatches for training ANE-g. From the development set, 130K microbatches were generated for ANE-p, and 141K microbatches for ANE-g. For the triplet-loss-basee encoder Trip-p, 17.9M triplets were generated from the training set, and 5.2M triplets from the development set.
The ASR "hybrid" DNN-HMM system (Bourlard & Morgan 1994) with state-of-the-an convolutional neural network acoustic model (Abdel- et al. (014]]Hunn et al.) (020)) accept- ing 40 filterbank outputs and emitting posteriors for 8,419 HMM states. When training f networks, the 8,419 state posteriors were mapped down to 51 monophone posteriors via a simple uniform map- ping based on state cluster membership. The FST decoder was based on Kaldi-OpenFSTT (Povey et al.) 2011: Allauzen et al.) 2007) built from a general language model with 900K vocabulary. Note that this general continuous speech recognizer was used for data preparation and the phonetic matching experiment in Table 4 but not for the FST-based recognition in Tables 2 [and 3 For the latter, separate whole-word FSTs were built, but the same acoustic model was used.
For the isolated name recognition experiment, 19,646 audio utterances of spoken names were pre- pared as evaluation data. Each utterance had a corresponding user-dependent list of possible names (i.e., the speaker's "phonebook") of varying size, with an average 1,055 names per phonebook. The phonebook was used to build a whole word FST recognizer for every utterance, using pronunciations obtained from G2P. The same list of pronunciations was used to generate the g embeddings. The transducer designed to directly consume subword posterior
FST recognizer was a subword-to-woe sequences instead of audio signals, to ensure that the same inputs were used for both the FST and
7

---

Under review as a conference paper at ICLR 2021
Table 2: Results of name recognition based nearest-neighbor search in 1 using the acoustic encoder f and the text encoder 9 in tandem, compared with the result from a whole-word FST recognizer. ANE-p used f and g trained by the proposed method using monophone transcriptions for each Y, (see Section 3.1 while ANE-g used grapheme transcriptions for Yn. Trip-p used a triplet distance with the same data as ANE-p. The numeric suffixes indicate the number of dimensions in the embedding vectors.
Method
Accuracy (%)
FST
97.5
Trip-p-20
79.6
Trip-p-30
84.7
ANE-g-20 (proposed method)
96.0
ANE-g-30 (proposed method)
96.3
ANE-p-20 (proposed method)
96.9
ANE-p-30 (proposed method)
97.3
ANE-p-40 (proposed method)
97.4
Table 3: Name recognition results for increasing phonebook size using f and in tandem. See caption for for description of methods. Because the phonebooks are much larger here than in Table the accuracies are lower overall. When the number of dimensions reaches 40, the Euclidean nearest-neighhor search between Acoustic Neighbor Embedding vectors (ANE-p-40) is accurate as FST decoding.
Method
Accuracy (%) for phonebook size
20K
100K
500K
1M
FST
88.1
84.4
76.8
72.1
ANE-g-20
79.7
75.7
67.5
62.0
ANE-g-30
81.5
78.0
70.3
64.8
ANE-p-20
86.5
81.7
73.1
68.2
ANE-p-30
88.1
84.1
76.4
71.9
ANE-p-40
88.2
84.4
77.0
72.7
the ff networks. A single acoustic model was used generate all training and testing inputs for the If networks, as well as the inputs to the FST wherever needed.
Table (able shows the accuracy of each method in the name recognition task, for varying dimensions in the embedding vectors. For ANE-p and Trip-p, the result of the rule (I) was a best-matching phone deemed correct when it was exact match with any pronunciation in
sequence, and the match the phonebook for the reference imanually-transcribeee transcribed) name. For ANE-g, crude normalization was first done on all names (removing special characters and converting to lowercase) for both training and testing data. The result of the rule in ( was a best-matching normalized name, and the match was deemed correct when it was an exact match with the normalized reference name. ANE-p was more accurate than ANE-g because ambiguities in word pronunciation were better resolved.
With the triplet distance, competitive accuracy could not be achieved in word recognition. Note, however, that previous studies using the triplet distance generally reported the best accuracy using the cosine distance, not the exponential T-2 distance that we used (in somewhat artificial manner) in (17) to parallel ANE's L2 distance. In Appendix A explore further comparisons with the triplet loss in experiments that are more grounded with the existing literature.
In second experiment, accuracies were measured for user-independent phonebooks of increasing size and are shown in Table 3 Because the phonebook sizes were much larger than in Table 2 accuracy differences can be observed more clearly. For simplicity, unified phonebook was used for all test samples, where the list contained all the reference names and was increasingly padded with other random names attain the different sizes in Table 3. There was increasing confusion
8

---

Under review as a conference paper at ICLR 2021
Table 4: Approximate phonetic matching accuracy for isolated names using g. The FST performs poorly because it uses only a general vocabulary where many of the phonebook names are missing. For ANE-p and Trip-p. the 9 network was used transform the FST's 1-best phone sequence into an embedding vector, then a nearest neighbor search was done on the actual phonebook names. See caption for Table 2 for further details on the methods shown.
Method
Accuracy (%)
FST using default vocabulary
67.8
Trip-p-20
88.8
Trip-p-30
89.6
ANE-p-20 (proposed method)
93.5
ANE-p-30 (proposed method)
93.9
between the names as the phonebook became larger, especially because most of the names were short (e.g. "Lian" and "Layan").
As seen in both Tables 2 and 3 the accuracy of ANE-p increases the number of dimensions in- crease because less discriminative information is discarded. For every utterance, the average number of dimensions in the original input used by the FST was 604K (8419 the length of the utterance, where the mean length is 71.8 frames). This was reduced to 20, 30, or 40 in Table 3 so the dimen- sion reduction was significant. However, the name recognition accuracy of ANE-p essentially identical to that of the FST when the number of embeddings is 40.
third experiment uses the phonebooks in the first experiment to perform recognition via approx- imate phonetic matching using only the y networks. First, general large-vocabulary (900K) con- tinuous ASR was run. Since many of the spoken names did not exist in the vocabulary, accuracy is low for the FST in Table 4 Next, from the FST's 1-best phone sequence, a gg vector computed, followed by a nearest neighbor search over the g vectors of the corresponding phonebook. Accuracy was computed based how often the correct name chosen, and ANE-p gave the best result.
The phonetic matching task is "casier" than the word recognition task in the sense that we already obtain fairly accurate phonetic transcription of the speech from the ASR (as opposed to having to recognize the speech from "scratch")). which is probably why the accuracy of the triplet-based embeddings improve in Table over Table 2] On the other hand, there also exists "accuracy ceiling" because of some -vocab-uyyyyylaryy words being transcribed far too wrong by the ASR, which is probably why the high accuracy of ANE in Table 2 drops in Table
CONCLUSION AND DISCUSSION
This paper has proposed a method of adapting stochastic neighbor embedding (SNE) to sequential data so that spoken words of arbitrary length be represented fixed-dimension vectors, called Acoustic Neighbor Embeddings, where the acoustic similarity between words is represented by the Euclidean distance between the vectors.
The proposed training criterion seems to share the same idea as the triplet loss criterion of pulling together vectors in the embedding space that represent the same phonetic sequence and pushing apart vectors that represent different phonetic sequences. To see how the two training criteria in essence differ, the gradients of the two loss functions were inspected to show that the proposed method can be more effective than the triplet loss. The efficacy of nearest-neighbor search using dimension-reducee ANE vectors in isolated name recognition and phonetic matching experiments was also demonstrated. In particular, experiments demonstrated that its accuracy in isolated word recognition can be identical to that of conventional FST-based decoding for vocabulary sizes up to 1 million.
is possible that creating only one embedding for each phone sequence does not sufficiently cap- ture all the acoustic variabilities in its pronunciation. SNE (Hinton & Roweis 2003) already has extensions that allow multiple embeddings per input in the dimension-reducced space, which we may be able to leverage in future work to make more robust matches.

---

Under review as a conference paper at ICLR 2021
REFERENCES
O. Abdel-Hamid, A. Mohamed, H. Jiang, L. Deng, G. Penn, and D. Yu. Convolutional neural networks for speech recognition. JEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 22(10)) 2014.
Robin Algayres, Mohamed Salah Zaiem, Benoit Sagot, and Emmanuel Dupoux. Evaluating the Reliability of Acoustic Speech Embeddings. In Proc. Interspeech 2020, pp. 4621-4625, 2020.
Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mchryar Mohri. Openfst: A general and efficient weighted finite-state transducer library. In Proceedings of the 12th Interna- tional Conference on Implementation and Application of Automata, CIAA'07, pp. 11-23, Berlin, Heidelberg, 2007. Springer- Springer-Verlag.
K. Audhkhasi and A. Verma. Keyword search using modified minimum edit distance measure. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP '07, volume 4, 2007.
Samy Bengio and Georg Heigold. Word embeddings for speech recognition. In Proceedings of the 15th Conference of the International Speech Communication Association, Interspeech, 2014.
Herve Bourlard and Nelson Morgan. Connectionist Speech Recognition: A Hybrid Approach. 01 1994.
Miguel Á. Carreira- Perpiia-Perpiian. The elastic embedding algorithm for dimensionality reduction. Pro- ceedings of the 27th International Conference on International Conference on Machine Learning, ICML' 10, pp. 167-174, Madison, WI, USA, 2010. Omnipress.
G. Chen, C. Parada, and T. N. Sainath. Query-by-example keyword spotting using long short-term memory networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5236-5240, 2015.
J. Cheng, H. Liu, F. Wang. H. Li, and C. Zhu. Silhouette analysis for human action recognition based supervised temporal t-SNE and incremental learning. IEEE Transactions Image Processing, 24(10):3203-3211 2015.
V. Garcia, E. Debreuve, and M. Barlaud. Fast k nearest neighbor search using GPU. In 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 1-6, 2008.
T. J. Hazen, W. Shen, and C. White. exxaple spoken term detection using phonetic pos- teriorgram templates. 2009 IEEE Workshop on Automatic Speech Recognition Understanding, pp. 421-426, 2009.
Wanjia He, Weiran Wang, and Karen Livescu. Multi-view recurrent neural acoustic word embed- dings. In International Conference on Learning Representations (ICLR), 2017.
Geoffrey Hinton and Sam T. Roweis. Stochastic neighbor embedding. In S. Becker, S. Thrun, and K. Obermayer (eds.), Advances in Neural Information Processing Systems 15, pp. 857-864 MIT Press, 2003.
Yushi Hu, Shane Settle, and Karen Livescu. Multilingual Jointly Trained Acoustic and Written Word Embeddings. In Proc. Interspeech 2020, pp. 1052-1056. 2020.
Z. Huang, T. Ng, L. Liu, H. Mason, X. Zhuang, and D. Liu. SNDCNN: Self-normalizing deep CNNs with scaled exponential linear units for speech recognition. ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1854-6858. 2020.
M. Jung, H. Lim, J. Goo, Y. Jung, and H. Kim. Additional shared decoder Siamese multi-view encoders for learning acoustic word embeddings. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 629-636, 2019.
10

---

Under review as a conference paper at ICLR 2021
H. Kamper, W. Wang, and K. Livescu. Deep convolutional acoustic word embeddings using word- pair side information. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 9950-495,, 2016.
H. Kamper, Y. Matusevych, and S. Goldwater. Multilingual acoustic word embedding models for processing zero-resource languages. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6414-641 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
K. Levin, K. Henry, A. Jansen, and K. Livescu. Fixed-dimensional acoustic embeddings of variable- length segments in low-resource settings. In 2013 IEEE Workshop on Automatic Speech Recog- nition and Understanding, pp. 410-415, 2013.
Hugh Christopher Longuet-Higgins The non-local storage of temporal information. In Proceedings of the Royal Society of London B, 1968.
C. Ma and W. Jeon. Efficient speech indexing and search for embedded devices using uniterms. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1297- 1300, 2009.
Andrew L. Maas, Stephen D. Miller, Tyler M. O'Neil, Andrew Y. Ng, and Patrick Nguyen. Word- level acoustic modeling with convolutional vector regression. In ICML 2012 Representation Learning Workshop. 2012.
Daniel Povey, Arnb Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek. Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, Jan Silovsky, Georg Stemmer, and Karel Vesely. The kaldi speech recognition toolkit. IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. IEFE Signal Processing Society, December 2011.
Rachid Riad, Corentin Dancette, Julien Karadayi, Neil Zeghidour, Thomas Schatz, and Emmanuel Dupoux. Sampling strategies in siamese networks for unsupervised speech representation learn- ing. In Proc. Interspeech 2018, pp. 2658-2666 2018.
Schroff, D. Kalenichenko, and J. Philbin. FaceNet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 815-823, June 2015.
Settle and K. Livescu. Discriminative acoustic word embeddings: recurrent neural network-based approaches. In 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 503-510, 2016.
S. Settle, K. Levin, H. Kamper, and K. Liveseu. Query-by-example search with diseriminative neural acoustic word embeddings. In INTERSPEECH 2017, 2017.
S. Settle, K. Audhkhasi, K. Livescu, and M. Picheny. Acoustically grounded word embeddings for improved acoustics-to-woot speech recognition. In ICASSP 2019 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5641-5645 2019,
Bowen Shi, Shane Settle, and Karen Livescu. Whole-word segmental speech recognition with acous- tic word embeddings. In arXiv 2007,00183, 2020.

---

Under review as a conference paper at ICLR 2021
A FURTHER COMPARISONS WITH THE TRIPLET LOSS USING MFCC INPUTS
One goal of this paper has been to examine some differences in the essence of ANE's training criterion in (5) and the triplet distance criterion in 13 not do a rigorous head-to-head comparison between specific known embodiments or implementations.
However, given that there exists rich body of work on the triplet distance, it is still meaningful to quantitatively compare ANE with known applications of the triplet loss in experimental settings that are more compatible with the existing literature than Section 5
One point of interest is that previous methods directly used acoustic feature vectors, rather than posteriorgrams, as the inputs to the f-encoder. Another characteristic that the best results were reported using the cosine distance rather than the L2 distance.
Furthermore, previous methods did not separately train g to explicitly match the output of f. as ANE does in 99. It would be interesting to incorporate this idea into previous triplet-based methods.
While word recognition the main application of interest in Section 5 it also useful to experiment with the acoustic word discrimination (Kamper al. 2016) and cross-view word discrimination (He ct al. 2017) tasks that have been the main focus of previous literature.
Based on these notions, for this appendix we trained an entirely new set of encoder networks as follows, all using MFCCs for the acoustic inputs and monophones for the text inputs:
1. mfc-ane-f: An encoder trained for MFCC inputs using the same method as described in Section 3.1
2. mfc-ane-g: A 9 encoder trained using the MSE distance in to mirror the outputs of mfc-ane-f.
3. mfc-tripfff: An encoder trained using single-view triplet distance with the cosine dis- tance on acoustic data only, based on (Kamper et al. 2016 Settle & Livescu 2016);
Ssingle. max {0, a Sim(fo fm Sim f,))
(19)
where f. has the same phone sequence as fg, f,, has different phone sequence from fo- and a is set to 0.15 as in (Kamper et al.) 2016)..
Note that this is probably a more direct parallel of mfc-ane-f than the multi-view approach
of (He et al. (017) or the triplet-based embeddings used Section 5 since mfc-ane-f is also trained independently of mfc-ane-g.
Since the cosine similarity is stated to give the best result in (Kamper et al. 2016),
use the cosine similarity in (19). 4. mfc-tripf-cos-g: Similar to mfc-ane-g, this is a g network that is trained to directly match mfc-tripf-f, where the f network is kept constant and only the network is trained. Note that previous studies have not explored this possibility, but it seems natural to do so in the current context.
Since mfc-tripf-f is based on the cosine distance, mfc-tripf-coo-- is also trained to minimize
the mean of the cosine distances over minibatch of "same-wer word" (in case, "same- pronunciation") pairs as follows:
g f
Cg
(20)
2N
gn
1=
5. mfc-tripfs-f: f network obtained using the multi-view triplet distance of [3 based on (He et al. 2017) using the cosine distance and MFCC inputs.
6. mfc-tripfs-g: g: g network that is jointly trained with mfc-tripfe-f via the multi-view triplet distance.
This is a "fine-tuned" version of mnf-tripfs- g. where we start with mfc- 7. tripfg-8 g and apply the training criterion in (20) to bring its outputs further closer to those mfc-tripfs-f for "same-word" pairs. Like mfc-tripf-coss previous studies have not ex- plored this possibility, but this seems be natural extension to try.
12

---

Under review as a conference paper at ICLR 2021
Note that the state-of-the- multi-view triplet- bipsett-based method is to actually consider two (or possibly more) different multi-view triplet losses (He et al. 2017: Jung et al. 2019), of the form
max {0, a Sim for gm) Sim(fo, gn))
(21)
max {0, Sim fm Bo) + Sim go)}
and to use the sum of the two corresponding batch losses. We do not compare with this loss function in our study, however, because it extends to realm of multi-view training that is beyond the scope of this paper. It may be possible, for example, to construct a multi-view version of ANE's loss similar above, which gives ANE the same enhancement as it does the triplet loss. For example,
L = L1 + L2
(22)
where L1 uses
cxp( (0|-- f; gg 1)11²)
Pij Expil (-1|f; gh l
(23)
and L2 uses
exp(- [g:
fg 1(9
d- Σesi I|g: 12
(24)
This is well beyond the scope of this paper, and we leave as a potential direction in future work.
The data used for this appendix is as follows:
1. Since purpose is compare the different embeddings listed above, not maximize accuracy (i.e., match FST recognition rates in Table 3), we used a significantly reduced data set than that in Section for faster turnaround. The training data and development data consisted of 941K and 335K utterances, respectively. For ANE training, 448K and 186K microbatches (each of size 160) were extracted from the training and development utterances, respectively. For triplet loss training, 4.6M and 1.7M triplet examples were extracted from the training and development utterances, respectively.
2. The inputs to encoders were MFCCs with 39 features (including delta and delta-delta) per frame, with a frame rate of 100 frames/s and a window length of 25ms. Mean and variance normalization was done based on global statistics from the training data.
3. For testing, from our proprietary data prepared set of test data with similar configu- ration as that used in previous work (Kamper et al. 2016 He et al. 2017). From our eval- uation data of random utterances, 662ms, of which 97K pairs were "same-pronunce" pronunciation" pairs and the other were "different- pronunciation" pairs. Note that the previous studies actually used "same-orthogruphy" or tdifferent-orlhogrruhhh pairs, but this difference would not detract from our goal of com- paring different encoders.
extracted 60M pairs of utterances, with mean length
The embedding dimensions were fixed to 30 for all f and g networks in this appendix. All networks were BiLSTM encoders with 2 layers and 400 states per direction per layer, with 39- dimensional input (for MFCCs) and a final linear layer applied to the final state of the last layer to produce 30-dimensional embedding. All networks were BiLSTM encoders with layer and 200 states per direction per layer, accepting 50-dimensional one-hot monophone vector sequences and a final linear layer producing 50-dimensional embedding. Training done using Adam (Kingma & Ba 2015) optimization for all encoders, with an initial learning rate of 0.001.
Three experiments were conducted:
1. Acoustic word discrimination: Similar to (Kamper al., 2016 Settle & Livescu 2016 He et al. 2017), the task was to decide whether two given acoustic sequences represent the same pronunciation not based on the distance between their f vectors.
The g encoders are not used in this task; only the f encoders.
In the case of mfc-ane-f. the Lg distance was used. In the case of mfc-tripe-f. the cosine distance was used. The overall accuracy is measured by the average precision (AP), which is between and and represents the area under the precision- recall curve (the higher the better).
13

---

Under review as a conference paper at ICLR 2021
Table shows the results, where see that mfc-ane-f has the highest accuracy among the three embeddings compared.
Among the two triplet-based methods, mfc-tripf-f shows a small improvement over mfc- tripfg-f. Note that the loss functions used by the two triplet- based methods are essentially the same when we make the approximation that f, = gj in our analysis in Section 4 The difference is that the loss function for mffetrip/g-f incurs the added "burden" of having to train both the f and g at the same time, and is therefore harder train than mfc- tripf-f. Hence, when all other conditions are held equal (data organization, neural network configuration, optimizer used for training, etc.), it is consistent with our expectations that mfc-tripf-f is more accurate than mfc-tripfg-f.
2. Cross-view word discrimination: Similar to (He et al. 2017). the task is to use f and in tandem to decide whether given acoustic sequence and a given text phone sequence represent the same pronunciation not.
As with the acoustic word discrimination task, the L2 distance used with mfc-ane-* and the cosine distance is used with mfc-tripe, and the overall accuracy is represented by the AP.
Table shows the results, where mfc-ane-f and mfc-ane-g provide the highest accuracy. Among the triplet-based methods, mfc-tripf-f provides the highest accuracy, consistent with the results in the acoustic word discrimination task.
With mfc-tripfg-f. we can observe a small increase in accuracy by the additional "fine- tuning" step applied in mff-tripf8-coo-- However, it still scores lower than mfc-tripf-f mic-tripf-cos-g probably because the latter has a better f encoder.
3. Name recognition: This is the same task as Table and the 20K task Table 3 but using MFCCs instead of posteriorgrams. The Euclidean nearest neighbor match in (I is used for mfc-ane-s embeddings, whereas the cosine nearest neighbor match used for mfc-trip* embeddings:
8
arg min ISSEN
(25)
g
[[]
Table shows the results, and we can see that the relative accuracy differences are generally similar those seen in the cross-view word discrimination task in Table 6 These differ- ences get further amplified when we increase the size of the phonebooks to 20K. Again, note that we have used less training data for our encoder networks than those in Tables 2 3 since our purpose here is to compare between the different MFCC-based encoders rather than match the accuracy of FSTs.
Table 5: Average precision in acoustic word discrimination task using 30 embedding dimensions
Method
AP
mfc-ane-f
0.622
mfc-tripf-f
0.575
mfc-tripfs-f
0.550
Table 6: Average precision in cross-view word discrimination task using 30 embedding dimensions
AP (%)
mfc-ane-f mfc-tripf-f mfc-tripfe-f mfc-tripfg-f
mfc-ane-g
0.829
mfc-tripf-cos-g
0.798
mfc-tripfg-8 g
0.772
mfc-tripfg-coss
0.790
14

---

Under review as a conference paper at ICLR 2021
Table 7: Name recognition accuracy (%) using 30 embedding dimensions on user-dependent phone- books (mean size 1,055) and a static 20K phonebook
Variable
20K
mfc-ane-f mfc-tripf-ff mfc-tripfg-f mfc-tripfg-f
m
88.78
64.95
mfc-tripf-cos--
87.28
61.11
mfc-tripf8-8
82.06
51.03
mfc-tripfg-cos--
85.30
56.64
B FURTHER DISCUSSION ON THE HARD BINARY DISTANCE IN EQUATION
There is rich history of computing distances between phonetic sequences or phonetic posteri- orgrams using dynamic programming (see, for example, (Hazen et al. 2009 Ma & Jeon, 2009. Audhkhasi & Verma 2007)). However, the edit distance tends to invite ad-hoc design decisions that make the process essentially heuristic and brittle. In particular, the substitution, deletion, and insertion costs are difficult to assign in a context-dependent manner. Data-driven phone confusion matrices, for example, can be used to compute substitution costs in (Audhkhasi & Verma 2007), but confusion matrices only provide a context- innrexttiideeeneent static cost for any given phone pair that does not account for coarticulation. Additional heuristics are also needed if, for example, one wishes to have a nonlinear length mismatch penalty, as it is hard to achieve this with insertion and deletion costs alone.
As such, we found that using a simple binary distance in 7 results in far more superior models (in terms of recognition accuracy) than an edit-distance-based distance measure.
can be puzzling that a hard binary distance can result in an f encoder that learns to produce smooth phonetic distances. Note that the triplet distance also shares the same characteristie; in (13), the signs are binary (+ or -) based on "same" or "different" labelings, with no "degree" of similarity imposed.
Here, we provide some intuition on how the binary distance works.
Consider pair of audio (X;, where Y, + Yj and therefore we assign dij co. If the words actually have similar pronunciation, and we have large training data, there will be many other identical pairs in the training data (pairs that are from different audio recordings, but with content almost the same (X;, Xj)) which have the "same" labeling and are assigned di = 0. This would have the same effect as (X,, Xj) being presented to the model multiple times, sometimes with the "same" labeling and sometimes with the "different" labeling. The "same"-labcled pairs will counteract the "different" llaabes pairs and prevent qú from becoming
For example, consider the words "cryin'," "crying," and "shouting."
Since "cryin"" and "crying" are very similar, as a toy example we could imagine 2n near-identical audio samples where half are labeled "cryin"" and the other half are labeled "crying." The total possible pairs that will get = oa from our harsh binary labeling is n², whereas the total possible pairs that will get = is 2n (n- 1). Asymptotically, the latter is twice the former, for any given pair from the 2n samples, the expected Øi would be somewhere closer to than
Now consider "cryin"" and "shouting." Because they would be extremely hard to find another pair in the training data that is acoustically identical to (X,, Xj) that also labeled "same". The distance d dominates for (X,, Xj) and the expected li would be somewhere close to O.
different, for any given Xy) it
intuit that as long large amount of training data is used where the proportion of "same""" "different" labelings for a given pair of audio is likely to represent the pair's phonetic confusability, ANE will learn smoothed distances that reflects these proportions.
Hence,
More rigorous mathematical analysis could probably be done in this direction, but in this work we suffice with the above intuitions and leave further quantitative analysis to future work.
15


Reviews:
[{"review_id": "nXSKvxLbGlU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approach to learn acoustic word embedding by adapting stochastic neighbor embedding (SNE) to sequential inputs. It learns both the acoustic embeddings and text embeddings from two neural encoders. The acoustic word embeddings of two acoustic sequences are learned to be close in Euclidean distance if their transcriptions are the same, or far apart otherwise. The text word embeddings, based on either phoneme or grapheme sequences, are learned to be close to their acoustic word embeddings. The experiment results on an isolated word (name) recognition task show that using nearest-neighbor search alone based on the proposed acoustic and text embeddings in tandem can achieve the same performance as a standard ASR model.\n\nStrong points:\n1. Well written with clear presentation.\n2. Well motivated by the extension from SNE. And it is technically sound.\n3. Interesting idea: Acoustic word embeddings are typically learned with a triplet loss in the literature. The proposed acoustic neighbor embeddings in this paper is a different way but shares similar motivation with the triplet loss, while showing more effective gradients.\n4. Promising experimental results: It shows using just the L2 distance between the embedding vector can perform as well as a standard ASR model (without a language model) over large vocabularies in an isolated word recognition task. The paper claims it is the first work to achieve this.\n\nWeak points:\n1. The isolated word recognition task requires known word boundaries.  It would be even more useful to show the effectiveness of the proposed approach in continuous speech recognition applications, as in some previous work on acoustic word embeddings, e.g. in lattice rescoring in (Bengio & Heigold, 2014), or in acoustic-to-word ASR in (Settle et al., 2019).\n2. Even though it might not affect the isolated word recognition task much, it would be good to also use a language model to build the stronger baseline FST-based ASR model. It would also be interesting to explore how well to combine the proposed embeddings with an LM in an ASR model, especially for continuous speech recognition.\n3. Why does the triplet loss perform much worse in Table 2 than in Table 4? More explanation and analysis would be very helpful.\n4. Not required, but would be great to have more analyses in the experiments, e.g.:\n  - How important is it to use posteriorgrams vs. acoustic features (maybe with a deeper embedding encoder) as the inputs for training the acoustic word embeddings?\n  - Analyze individual examples to show the better and worse cases compared to the baselines.\n  - Visualization of the proposed embeddings, and possibly with comparison to that of the triplet-loss-trained embeddings.\n\nIn general, the paper looks good and I would recommend it for acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bLQhUNk6YM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose a new training method to learn acoustic embeddings by simultaneously training two encoder networks, one for speech (f) and one for text (g), such that the resulting embeddings from the two networks are in a common subspace. At test time, the embedding for a given speech input using the network f could be compared against a text database of embedding vectors derived using the network g and vice-versa. The proposed technique is shown to be more accurate than a standard triplet loss-based retrieval method and it performs at par with an FST-based speech recognition system.\n\nWhile the proposed embedding method is a direct adaptation of the stochastic neighbor embedding (SNE) technique by Hinton & Roweis ('03), the modifications have been described very clearly in this draft. The authors also do a nice job of describing why the proposed technique might be more effective than a triplet loss-based method by inspecting the gradients of both losses.\n\nAs it stands, the experimental section in the draft is a bit thin:\n\n* For a more direct comparison with prior work on learning acoustic embeddings, it would be useful to see how the proposed embeddings fare on an acoustic word discrimination task (as described in He et al. 2017) where the task is to predict whether a pair of acoustic sequences correspond to the same word or not. Cross-view word discrimination (where the pairs of inputs consist of both written words and spoken words) is another task proposed by He et al. 2017 which could also be easily set up for the proposed method.\n\n* Since embeddings from both f and g are simultaneously learned, could the authors also show experiments on the standard task of spoken term detection where one has to search for a given text query within spoken utterances.\n\n* The authors mention that the use of a hard binary distance as defined in Equation (7) was the best choice, rather than using softer definitions of distance. Some supporting experiments to show how different definitions of the distance function affect performance would be useful. (This could go into an appendix.)\n\n* Given that SNE lends itself well to visualizations, it would be nice for the reader to see visualizations showing how acoustic and textual embeddings of similar words cluster together.\n\n* A relatively minor point. In Section 5, the authors mention that the phonetic transcriptions were obtained using an FST-based ASR with a DNN-HMM acoustic model but don't provide any further details. Please elaborate on the ASR system that was used or include an appendix with more implementation details.\n\nAlso, given that the proposed technique performs at par with an FST-based recognizer, the motivation for why the proposed technique is needed should be brought out more clearly. (For example, when using phonebooks of larger sizes, the nearest neighbour search using the proposed embedding method might be more computationally efficient than an FST-based recognizer.)\n\n----------------------\n\nUpdate after author response:\n\nThanks to the authors for addressing some of my concerns by conducting additional experiments that are listed in Appendix A. I have now increased my rating from 5 to 6. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "b7dA9eXjf9M", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Overview:\n\nThis paper proposes a new acoustic word embedding approach, where acoustic and text embeddings are jointly learned. The text encoder takes either phonemes or characters as input. The novelty of the paper lies in a new loss, which is based on stochastic neighbour embeddings (SNE). The acoustic embedding network is first trained with this loss, after which the text network is trained to produce similar embeddings for matched (acoustic, text) input. The proposed model is evaluated in a word recognition task, where an isolated spoken word's acoustic embedding is compared to the text embeddings and the nearest neighbour is used to classify the spoken word.\n\n**Note the edit at the bottom of this review, based on the authors' feedback.**\n\n### Strengths:\n\n- I do not believe that existing acoustic embedding methods have considered the idea of including a SNE-like objective, especially using the text-view to identify \"neigboring\" items.\n- The use of posteriorgrams as input is well-motivated, in the context of what the paper tries to accomplish.\n- The paper is generally easy to follow.\n\n\n### Weaknesses:\n\n- The models aren't sufficiently compared to previous models on established tasks. (I make this more concrete below.)\n- The experiments don't show the effect of using e.g. the posteriorgrams over standard features (like MFFCs). Posteriorgrams means that this approach is essentially reliant on a first-pass ASR model.\n- As a minor weakness, some recent related work aren't cited (references given at the bottom).\n\n\n### Detailed questions and suggestions:\n\nA number of studies in the acoustic word embedding literature have used the same-different task to evaluate performance, e.g. in (He et al., 2017) and [1] and [2]. Given the (somewhat surprising) poor performance of the triplet-based model, I would suggest that the paper does a comparison on the same data and task, to confirm the validity of their triplet-based model. This will also make the work more valuable, in that it can be directly compared to previous studies. One potential issue with the triplet model in this paper is that I believe the model of (He et al., 2017) makes use of cosine similarity, instead of the Euclidean distance. Since labelled examples are available, it would also have been good to compare to a direct classification model, as in [3].\n\nI did not state this as a weakness, since I am worried that I am missing some details, but I am concerned by the overall analysis of Section 4. First, by defining the neighbourhood as in equations (7) and (8), it seems that this model essentially optimises the loss based on whether two acoustic realisations are from the same or different words, and no finer-grained neighbourhood information is included. The only \"strength\" that is imposed comes from $c_i$, which is essentially linked to the word frequency. The section concludes that \"ANE has the added subtlety of pushing or pulling with more 'measured strength' based on how good the embeddings currently are,\" but if the loss is purely based on a weighing according to the word frequency, could something similar be accomplished by having a type-specific margin for the triplet loss in equation (13)? If I read the analysis correctly the margin is actually completely ignored: \"we can ignore the max operator and set $\\alpha = 0$ in (13), since they are merely for data selection.\"  I do not believe that this last statement is correct.\n\nOne further suggestion is to look at more advanced sampling strategies in the triplet model, as e.g. in [1] or [6].\n\n\n### Overall assessment:\n\nGiven the shortcomings in the experimental investigation, I do not believe the paper can be accepted as it is. I would recommend that the authors include the above-mentioned additional experiments; this would be a non-trivial extension, and I, therefore, recommend the paper then be submitted a future conference. I award an \"Ok but not good enough - rejection\".\n\n\n### Typos, grammar and style:\n\n- \"approximate phonetic match task\" -> \"approximate phonetic matching task\"\n- \"As we can see in Table. 1\" -> \"As we can see in Table 1\"\n- \"this beautiful equation\". I would suggest that the authors remove subjective words such as \"beautiful\".  (I agree it's a beautiful equation, but I don't think this type of language is appropriate for such a paper.)\n- \"it is not a good idea\". Similar to the above.\n- Make sure to write phoneme sequences correctly. See [4].\n\n\n### Missing references:\n\n1. https://arxiv.org/abs/2006.02295\n2. https://arxiv.org/pdf/2006.14007\n3. http://arxiv.org/pdf/1510.01032\n4. https://arxiv.org/abs/1907.11640\n5. https://arxiv.org/abs/2007.00183\n6. https://arxiv.org/abs/1804.11297\n7. https://arxiv.org/abs/2007.13542\n\n\n### Edit based on the authors' response\n\nI believe the author(s) were able to address many of the major concerns that I and the other reviewers had.  One issue is that much of this is placed in an appendix, so it doesn't form a core part of the main thread of the paper;  I also disagree about one small point (see my separate comment to the Part 2 message below), but this is minor.  Based on their more extensive investigation, I am changing my rating from \"4: Ok but not good enough - rejection\" to \"6: Marginally above acceptance threshold\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "GEkVxFAt9g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While word embedding are very popular and useful for NLP tasks, in speech recognition acoustic embedding plays the same important role (for example to perform query search with the audio recordings). In this paper authors propose a novel idea of acoustic embedding learning, called Acoustic Neighbor Embedding. Using stochastic neighbor embedding (SNE) idea of preserving the relative distances and optimizing Kullback-Leibler divergence (KL), proposed in the paper embedding is constructed optimizing KL divergence (for acoustic) and $L_2$ distance between acoustic embedding and correspondent text embedding (for text). With experiments authors show that their approach is better than triplet loss approach used in the previous works. Also with $L_2$ distance between embedding vectors this paper is the first one which has a competitive isolated word recognition quality compared to the FST-based approaches. Experiments are done for several different scenarios: search for personal phonebook, search in global phonebook of different size and search by closest text embedding after ASR model.\n\nPros of the paper:\n- A new idea for acoustic embedding learning with SNE idea behind.\n- Analysis of the proposed loss with the triplet loss.\n- Analysis of the phonetic confusability demonstrating how the method behaves for the most complicated cases of recognition.\n\nCons:\n- General application for entity recognition with the proposed approach will be not super easy: besides ASR system there should be also force-aligning system which should extract segments where entities are presented. And the most complicated task is obtaining these force-aligned segments.\n- Experiments are done on the private datasets, so results in the paper are not reproducible and not comparable with the previous and future works. \n- There is no analysis what is the force-aligning error presented in the data.\n- There is no comparison with the other acoustic features instead of \"postreriograms\" (\"posteriograms\" could have errors of ASR model itself and possibly not be ideal for acoustic embedding construction) both for triplet loss and proposed method. Previous work He et al., 2017 used MFCC for the triplet loss. Thus to demonstrate that proposed approach is well suited for the task it should be also shown that it outperforms the triplet loss on MFCC input.\n\nComments:\n- \"However, none of the aforementioned papers have reported results in isolated word recognition.\":  Jung et al., 2019 reports average precision of isolated word recognition (if audio and text are given for the same word). Otherwise, \"isolated word recognition\" should be clarified in the paper.\n- Would like to have an ablation when instead of \"posteriograms\" raw wave or MFSC/MFCC features are used to investigate 1) what is the effect of ASR system errors on the embedding learning 2) what performance we loose if switch to the \"posteriograms\" with small network $f(x)$. Experiments for triplet loss should be provided too for this case (it could be that triplet loss behaves better with not \"postriograms\" input).\n- What are the mean and std of the values for $T$ and $M$ in the data (for both monophone and grapheme cases)?\n- It is better to move section 3.3 into experiments section 5 otherwise it is not clear with which model $L_2$ distances are computed for examples (Table 1). This is more analysis of a model trained on particular dataset.\n- In formulas (13) and (14) the same notation for loss is used $\\mathcal{L}_\\text{trip}$: the first one is for one sample and the second - for all. Usage of different notation will improve readability.\n- Why is silence excluded from the $g(y)$ training? It gives the word boundaries notion so the confusion for the text could occur in the silence absence.\n- Could authors clarify what they mean with the \"regression layer\" (just linear layer?)?\n- Comparison with FST baseline is not clear. In case of FST we can generate any sequence of words (say, we have \"Katarina F.\" and \"John S.\" in the phonebook, then FST potentially can infer also \"John F.\" or \"Katarina S.\", while ANE cannot generate this as we have text embeddings only for original \"Katarina F.\" and \"John S.\", thus accuracy for FST will be underestimated in this comparison). Could authors clarify this?\n- It would be interesting to see accuracy on in-vocabulary and out-of-vocabulary in Table 4 (with the number of such examples) to have the better comparison with FST (showing that in-vocab has the same performance as FST and what is peformance for pure out-of-vocab).\n\nThe idea of the paper is very interesting and based on well known and effective SNE approach. The paper itself very well written, with enough clarification to understand the main idea. However, the main concerns of the paper are:\n- usage of private data\n- lack of the experiments with MFCC input for both triplet loss and proposed loss to show the new method effectiveness (one of the paper claims is that the method is better than triplet loss)\n- clarification on the FST evaluation is needed (one of the paper claims that the methods performs similar to FST).\n\n### Edit based on the authors' response\n\nI believe the authors have addressed part of the major concerns that I and the other reviewers had. Comparison with FST approach and triplet loss is clarified and supported by the extensive experiments now, however, most important things are in Appendix only. Based on the updated paper's version I change my rating from \"5: Marginally below acceptance threshold\" to \"6: Marginally above acceptance threshold\".\n\nOne of the reviewers mentioned about comparison with language model (LM): here we could use character/phoneme based LM and vocabulary which can help to solve ambiguity too. So this could be considered as a good experiment for future work to show the great potential of ANE if it outperforms LM usage. \n\nStill I have several concerns, probably more philosophical from some point of view:\n- Rely on the force-aligning data for practical applications (in experiments it was ideal segmentation)\n- Usage of private data in experiments.\n\nAuthors state \"The goal of our paper is to introduce ANE and highlight some interesting things about it.\". I don't see any points in the paper and author's comments why this is helpful/applicable/better than some another ideas. Authors mentioned in the comments that they cannot state that ANE is faster than the FST while having the same performance (this could be one good point that we have speed up using small embeddings + simple L2 distance computation as pointed by one the reviewers). About applications for continuous speech authors gave the comment \"However, we are not even sure if speech recognition per se is the best application for ANE. We are hoping that the community will find other interesting uses, either with the embeddings themselves or the distances between embeddings.\". Thus, I am feeling that the paper is not finished in that respect. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "-LOA5yZg3vM", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work adapts stochastic neighbor embedding (SNE) to acoustic segments to learn “acoustic neighbor embeddings” (ANE), which are fixed dimensional embeddings of variable-length speech. They also learn embeddings of phone or grapheme sequences of words corresponding to these segments. They compare the form of this loss function as well as its gradient with popular multi-view triplet loss approaches. Performance is then given on an isolated word recognition task, shown to outperform triplet losses as well as an FST baseline. Overall, I think there is a slight misalignment between ANE and multi-view training (vs acoustic-only triplet loss), which could be addressed in additional experiments as well as concerns that second stage $L_2$-distance training introduces unfair advantages to ANE over multi-view training for comparison on isolated word recognition.\n\nPros:\n- Interesting analysis of phonetic confusability between words reflected in the $g$ vectors.\n- I appreciate the straightforward nature of the extension, as well as comparison and motivations versus (multi-view) triplet losses in Section 4.\n- This approach is strong for the isolated word recognition task versus the provided baselines (FST and triplet).\n\nComments:\n- It seems like the main comparison between loss functions is being done between ANE and a multi-view triplet loss, which is okay, but I think the better comparison would be between triplet losses that only operate on acoustic segments (e.g. Kamper et al https://arxiv.org/pdf/1510.01032.pdf). It looks like you’re training a new acoustic embedding function (similar more to acoustic-only triplet losses than to multi-view triplet losses) and then incorporating the second view $g$ through a secondary $L_2$ loss applied after first training ANE. You could compare the pipelines of acoustic-only triplet loss versus ANE to learn f followed by either an $L_2$-distance or multi-view training approach to learn g (with or without f fixed). Also, since you’re using nearest neighbors to perform your evaluation, fine-tuning after training the multi-view approach with a second stage $L_2$-distance training would be fairer given that that’s what’s being used through nearest neighbor search at evaluation time I believe. It may detract from the overall aesthetic of moving away from triplet loss, but I think these would be reasonable comparisons. ANE shows a well-motivated method for learning an acoustic embedding space, but incorporation of the second view through $L_2$-distance versus a triplet loss isn’t compared. Additionally, an argument can be made for establishing $g_j$  an $f_j$ as roughly the same from Equation 14, but in practice it appears that the differences in these values actually help stabilize training as g’s are consistent for a particular word rather than varying for every acoustic instance. I admit this hasn’t been adequately/explicitly shown in prior work, but I wanted to mention it as another benefit to comparison directly with acoustic-only triplet losses which would remove this approximation in your derivations.\n\n- It would be nice to see additional results on some common acoustic word embedding datasets, or evaluations that include the word discrimination task using average precision (AP) in addition to word accuracy. Also, there are many applications of these acoustic embedding approaches to low-resource domains in which word-labels may not be available but clusterings of similar examples may be. It could be interesting to give a word discrimination evaluation (in AP) for just the acoustic embedding outputs (from $f$) for this reason.\n\n- Additional evaluation using typical acoustic features would be nice for curiosity reasons as well as due to the above concern about common applications to low-resource domains where high-performing ASR systems may not exist a priori.\n\nOverall:\nI find application of SNE to acoustic segments interesting, but I think the presentation it could benefit from more direct comparisons. In particular the three sets of experiments I would have in mind would be:\n\n- acoustic-only triplet loss\n    - train $f$ with acoustic-only triplet loss, then train $g$ with $L_2$-distance as given in Eq 9\n    - train $f$ with acoustic-only triplet loss, then train $g$ with multi-view approach as given in Eq 13 (with $f$ fixed)\n    - train $f$ with acoustic-only triplet loss, then train $g$ with multi-view approach as given in Eq 13 (without $f$ fixed)\n- ANE experiments\n    - train $f$ with ANE loss, then train $g$ with $L_2$-distance as given in Eq 9\n    - train $f$ with ANE loss, then train $g$ with multi-view approach as given in Eq 13 (with $f$ fixed)\n    - train $f$ with ANE loss, then train $g$ with multi-view approach as given in Eq 13 (without $f$ fixed)\n- Multi-view experiments\n    - train $f$ and $g$ with multi-view approach as given in Eq 13\n    - train $f$ and $g$ with multi-view approach as given in Eq 13, then fine-tune $g$ with $L_2$-distance as given in Eq 9\n\nIn addition, an AP evaluation (and, to a lesser extent given that time is limited, more common acoustic embedding datasets or domains) could be nice to include.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper propose to learn the embedding of audio segment in the framework of stochastic neighbor embedding (SNE), where the embeddings of the same word shall be close to each other. The method was initially demonstrated for name recognition. The use of SNE for acoustic embedding is novel and this is recognized by all reviewers. There has been quite some discussions between the authors and reviewers/AC, and the papers has got improved since. To summarize:
1. The discussion of the properties of SNE (the reduction from SNE to weighted least squares terms) was not accurate and confused multiple reviewers. The authors have made some clarifications and added citations. As the author claim this to be a contribution, I feel this part of the main text can be further strengthened and made more accurate.
2. For the experiment in main paper, the comparison between proposed method and prior work was not fair since the proposed method use outputs from an ASR system to obtain phone posteriors. The authors then added more results for the word discrimination task in the appendix. But reviewers are still concerned that the authors are not comparing with the strongest variant of He et al, 2017, and that comparisons are shown for embeddings with low dimensions. The reviewers believe this set of experiments shall be more illuminating, and be moved to the main text.
3. The biggest concern at the intuition level is whether it is the best choice to make the affinity binary, which does not take into account the more fine-grained similarity between different words. Quoting the comment from Reviewer 6:  "The argument of trying to be as simple as possible is reasonable, but we would have liked to see it motivated by some experiments. Something along the lines of a method which introduced rudimentary edit distance-based affinities and then presented their version with hard affinities, and then show some results comparing them. These edit distance-based versions could be as simple or complicated as they felt necessary, but it would be nice for some comparison to be made in order to then dismiss them."

Overall, we think the paper is borderline in the current stage, and the paper can be further improved if the above concerns are properly addressed.

Decision: Reject