Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This work adapts stochastic neighbor embedding (SNE) to acoustic segments to learn “acoustic neighbor embeddings” (ANE), which are fixed dimensional embeddings of variable-length speech. They also learn embeddings of phone or grapheme sequences of words corresponding to these segments. They compare the form of this loss function as well as its gradient with popular multi-view triplet loss approaches. Performance is then given on an isolated word recognition task, shown to outperform triplet losses as well as an FST baseline. Overall, I think there is a slight misalignment between ANE and multi-view training (vs acoustic-only triplet loss), which could be addressed in additional experiments as well as concerns that second stage $L_2$-distance training introduces unfair advantages to ANE over multi-view training for comparison on isolated word recognition.

Pros:
- Interesting analysis of phonetic confusability between words reflected in the $g$ vectors.
- I appreciate the straightforward nature of the extension, as well as comparison and motivations versus (multi-view) triplet losses in Section 4.
- This approach is strong for the isolated word recognition task versus the provided baselines (FST and triplet).

Comments:
- It seems like the main comparison between loss functions is being done between ANE and a multi-view triplet loss, which is okay, but I think the better comparison would be between triplet losses that only operate on acoustic segments (e.g. Kamper et al https://arxiv.org/pdf/1510.01032.pdf). It looks like you’re training a new acoustic embedding function (similar more to acoustic-only triplet losses than to multi-view triplet losses) and then incorporating the second view $g$ through a secondary $L_2$ loss applied after first training ANE. You could compare the pipelines of acoustic-only triplet loss versus ANE to learn f followed by either an $L_2$-distance or multi-view training approach to learn g (with or without f fixed). Also, since you’re using nearest neighbors to perform your evaluation, fine-tuning after training the multi-view approach with a second stage $L_2$-distance training would be fairer given that that’s what’s being used through nearest neighbor search at evaluation time I believe. It may detract from the overall aesthetic of moving away from triplet loss, but I think these would be reasonable comparisons. ANE shows a well-motivated method for learning an acoustic embedding space, but incorporation of the second view through $L_2$-distance versus a triplet loss isn’t compared. Additionally, an argument can be made for establishing $g_j$  an $f_j$ as roughly the same from Equation 14, but in practice it appears that the differences in these values actually help stabilize training as g’s are consistent for a particular word rather than varying for every acoustic instance. I admit this hasn’t been adequately/explicitly shown in prior work, but I wanted to mention it as another benefit to comparison directly with acoustic-only triplet losses which would remove this approximation in your derivations.

- It would be nice to see additional results on some common acoustic word embedding datasets, or evaluations that include the word discrimination task using average precision (AP) in addition to word accuracy. Also, there are many applications of these acoustic embedding approaches to low-resource domains in which word-labels may not be available but clusterings of similar examples may be. It could be interesting to give a word discrimination evaluation (in AP) for just the acoustic embedding outputs (from $f$) for this reason.

- Additional evaluation using typical acoustic features would be nice for curiosity reasons as well as due to the above concern about common applications to low-resource domains where high-performing ASR systems may not exist a priori.

Overall:
I find application of SNE to acoustic segments interesting, but I think the presentation it could benefit from more direct comparisons. In particular the three sets of experiments I would have in mind would be:

- acoustic-only triplet loss
    - train $f$ with acoustic-only triplet loss, then train $g$ with $L_2$-distance as given in Eq 9
    - train $f$ with acoustic-only triplet loss, then train $g$ with multi-view approach as given in Eq 13 (with $f$ fixed)
    - train $f$ with acoustic-only triplet loss, then train $g$ with multi-view approach as given in Eq 13 (without $f$ fixed)
- ANE experiments
    - train $f$ with ANE loss, then train $g$ with $L_2$-distance as given in Eq 9
    - train $f$ with ANE loss, then train $g$ with multi-view approach as given in Eq 13 (with $f$ fixed)
    - train $f$ with ANE loss, then train $g$ with multi-view approach as given in Eq 13 (without $f$ fixed)
- Multi-view experiments
    - train $f$ and $g$ with multi-view approach as given in Eq 13
    - train $f$ and $g$ with multi-view approach as given in Eq 13, then fine-tune $g$ with $L_2$-distance as given in Eq 9

In addition, an AP evaluation (and, to a lesser extent given that time is limited, more common acoustic embedding datasets or domains) could be nice to include.
review_title: I find the application of SNE to acoustic segments interesting and well-motivated, but I think the presentation would benefit from more direct comparisons as I believe there is misalignment between ANE and multi-view training (vs acoustic-only triplet loss), which could be addressed in additional experiments. I also have concerns that second stage L2-distance training introduces unfair advantages to ANE over multi-view training for comparison on isolated word recognition.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new approach to learn acoustic word embedding by adapting stochastic neighbor embedding (SNE) to sequential inputs. It learns both the acoustic embeddings and text embeddings from two neural encoders. The acoustic word embeddings of two acoustic sequences are learned to be close in Euclidean distance if their transcriptions are the same, or far apart otherwise. The text word embeddings, based on either phoneme or grapheme sequences, are learned to be close to their acoustic word embeddings. The experiment results on an isolated word (name) recognition task show that using nearest-neighbor search alone based on the proposed acoustic and text embeddings in tandem can achieve the same performance as a standard ASR model.

Strong points:
1. Well written with clear presentation.
2. Well motivated by the extension from SNE. And it is technically sound.
3. Interesting idea: Acoustic word embeddings are typically learned with a triplet loss in the literature. The proposed acoustic neighbor embeddings in this paper is a different way but shares similar motivation with the triplet loss, while showing more effective gradients.
4. Promising experimental results: It shows using just the L2 distance between the embedding vector can perform as well as a standard ASR model (without a language model) over large vocabularies in an isolated word recognition task. The paper claims it is the first work to achieve this.

Weak points:
1. The isolated word recognition task requires known word boundaries.  It would be even more useful to show the effectiveness of the proposed approach in continuous speech recognition applications, as in some previous work on acoustic word embeddings, e.g. in lattice rescoring in (Bengio & Heigold, 2014), or in acoustic-to-word ASR in (Settle et al., 2019).
2. Even though it might not affect the isolated word recognition task much, it would be good to also use a language model to build the stronger baseline FST-based ASR model. It would also be interesting to explore how well to combine the proposed embeddings with an LM in an ASR model, especially for continuous speech recognition.
3. Why does the triplet loss perform much worse in Table 2 than in Table 4? More explanation and analysis would be very helpful.
4. Not required, but would be great to have more analyses in the experiments, e.g.:
  - How important is it to use posteriorgrams vs. acoustic features (maybe with a deeper embedding encoder) as the inputs for training the acoustic word embeddings?
  - Analyze individual examples to show the better and worse cases compared to the baselines.
  - Visualization of the proposed embeddings, and possibly with comparison to that of the triplet-loss-trained embeddings.

In general, the paper looks good and I would recommend it for acceptance.
review_title: Interesting idea with promising results

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Overview:

This paper proposes a new acoustic word embedding approach, where acoustic and text embeddings are jointly learned. The text encoder takes either phonemes or characters as input. The novelty of the paper lies in a new loss, which is based on stochastic neighbour embeddings (SNE). The acoustic embedding network is first trained with this loss, after which the text network is trained to produce similar embeddings for matched (acoustic, text) input. The proposed model is evaluated in a word recognition task, where an isolated spoken word's acoustic embedding is compared to the text embeddings and the nearest neighbour is used to classify the spoken word.

**Note the edit at the bottom of this review, based on the authors' feedback.**

### Strengths:

- I do not believe that existing acoustic embedding methods have considered the idea of including a SNE-like objective, especially using the text-view to identify "neigboring" items.
- The use of posteriorgrams as input is well-motivated, in the context of what the paper tries to accomplish.
- The paper is generally easy to follow.


### Weaknesses:

- The models aren't sufficiently compared to previous models on established tasks. (I make this more concrete below.)
- The experiments don't show the effect of using e.g. the posteriorgrams over standard features (like MFFCs). Posteriorgrams means that this approach is essentially reliant on a first-pass ASR model.
- As a minor weakness, some recent related work aren't cited (references given at the bottom).


### Detailed questions and suggestions:

A number of studies in the acoustic word embedding literature have used the same-different task to evaluate performance, e.g. in (He et al., 2017) and [1] and [2]. Given the (somewhat surprising) poor performance of the triplet-based model, I would suggest that the paper does a comparison on the same data and task, to confirm the validity of their triplet-based model. This will also make the work more valuable, in that it can be directly compared to previous studies. One potential issue with the triplet model in this paper is that I believe the model of (He et al., 2017) makes use of cosine similarity, instead of the Euclidean distance. Since labelled examples are available, it would also have been good to compare to a direct classification model, as in [3].

I did not state this as a weakness, since I am worried that I am missing some details, but I am concerned by the overall analysis of Section 4. First, by defining the neighbourhood as in equations (7) and (8), it seems that this model essentially optimises the loss based on whether two acoustic realisations are from the same or different words, and no finer-grained neighbourhood information is included. The only "strength" that is imposed comes from $c_i$, which is essentially linked to the word frequency. The section concludes that "ANE has the added subtlety of pushing or pulling with more 'measured strength' based on how good the embeddings currently are," but if the loss is purely based on a weighing according to the word frequency, could something similar be accomplished by having a type-specific margin for the triplet loss in equation (13)? If I read the analysis correctly the margin is actually completely ignored: "we can ignore the max operator and set $\alpha = 0$ in (13), since they are merely for data selection."  I do not believe that this last statement is correct.

One further suggestion is to look at more advanced sampling strategies in the triplet model, as e.g. in [1] or [6].


### Overall assessment:

Given the shortcomings in the experimental investigation, I do not believe the paper can be accepted as it is. I would recommend that the authors include the above-mentioned additional experiments; this would be a non-trivial extension, and I, therefore, recommend the paper then be submitted a future conference. I award an "Ok but not good enough - rejection".


### Typos, grammar and style:

- "approximate phonetic match task" -> "approximate phonetic matching task"
- "As we can see in Table. 1" -> "As we can see in Table 1"
- "this beautiful equation". I would suggest that the authors remove subjective words such as "beautiful".  (I agree it's a beautiful equation, but I don't think this type of language is appropriate for such a paper.)
- "it is not a good idea". Similar to the above.
- Make sure to write phoneme sequences correctly. See [4].


### Missing references:

1. https://arxiv.org/abs/2006.02295
2. https://arxiv.org/pdf/2006.14007
3. http://arxiv.org/pdf/1510.01032
4. https://arxiv.org/abs/1907.11640
5. https://arxiv.org/abs/2007.00183
6. https://arxiv.org/abs/1804.11297
7. https://arxiv.org/abs/2007.13542


### Edit based on the authors' response

I believe the author(s) were able to address many of the major concerns that I and the other reviewers had.  One issue is that much of this is placed in an appendix, so it doesn't form a core part of the main thread of the paper;  I also disagree about one small point (see my separate comment to the Part 2 message below), but this is minor.  Based on their more extensive investigation, I am changing my rating from "4: Ok but not good enough - rejection" to "6: Marginally above acceptance threshold".
review_title: Interesting idea, experimental validation falls short

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this work, the authors propose a new training method to learn acoustic embeddings by simultaneously training two encoder networks, one for speech (f) and one for text (g), such that the resulting embeddings from the two networks are in a common subspace. At test time, the embedding for a given speech input using the network f could be compared against a text database of embedding vectors derived using the network g and vice-versa. The proposed technique is shown to be more accurate than a standard triplet loss-based retrieval method and it performs at par with an FST-based speech recognition system.

While the proposed embedding method is a direct adaptation of the stochastic neighbor embedding (SNE) technique by Hinton & Roweis ('03), the modifications have been described very clearly in this draft. The authors also do a nice job of describing why the proposed technique might be more effective than a triplet loss-based method by inspecting the gradients of both losses.

As it stands, the experimental section in the draft is a bit thin:

* For a more direct comparison with prior work on learning acoustic embeddings, it would be useful to see how the proposed embeddings fare on an acoustic word discrimination task (as described in He et al. 2017) where the task is to predict whether a pair of acoustic sequences correspond to the same word or not. Cross-view word discrimination (where the pairs of inputs consist of both written words and spoken words) is another task proposed by He et al. 2017 which could also be easily set up for the proposed method.

* Since embeddings from both f and g are simultaneously learned, could the authors also show experiments on the standard task of spoken term detection where one has to search for a given text query within spoken utterances.

* The authors mention that the use of a hard binary distance as defined in Equation (7) was the best choice, rather than using softer definitions of distance. Some supporting experiments to show how different definitions of the distance function affect performance would be useful. (This could go into an appendix.)

* Given that SNE lends itself well to visualizations, it would be nice for the reader to see visualizations showing how acoustic and textual embeddings of similar words cluster together.

* A relatively minor point. In Section 5, the authors mention that the phonetic transcriptions were obtained using an FST-based ASR with a DNN-HMM acoustic model but don't provide any further details. Please elaborate on the ASR system that was used or include an appendix with more implementation details.

Also, given that the proposed technique performs at par with an FST-based recognizer, the motivation for why the proposed technique is needed should be brought out more clearly. (For example, when using phonebooks of larger sizes, the nearest neighbour search using the proposed embedding method might be more computationally efficient than an FST-based recognizer.)

----------------------

Update after author response:

Thanks to the authors for addressing some of my concerns by conducting additional experiments that are listed in Appendix A. I have now increased my rating from 5 to 6. 
review_title: Adapting the SNE technique to learn acoustic embeddings; could be more thorough in its experiments.

final_score: rating: 6: Marginally above acceptance threshold
review_content: While word embedding are very popular and useful for NLP tasks, in speech recognition acoustic embedding plays the same important role (for example to perform query search with the audio recordings). In this paper authors propose a novel idea of acoustic embedding learning, called Acoustic Neighbor Embedding. Using stochastic neighbor embedding (SNE) idea of preserving the relative distances and optimizing Kullback-Leibler divergence (KL), proposed in the paper embedding is constructed optimizing KL divergence (for acoustic) and $L_2$ distance between acoustic embedding and correspondent text embedding (for text). With experiments authors show that their approach is better than triplet loss approach used in the previous works. Also with $L_2$ distance between embedding vectors this paper is the first one which has a competitive isolated word recognition quality compared to the FST-based approaches. Experiments are done for several different scenarios: search for personal phonebook, search in global phonebook of different size and search by closest text embedding after ASR model.

Pros of the paper:
- A new idea for acoustic embedding learning with SNE idea behind.
- Analysis of the proposed loss with the triplet loss.
- Analysis of the phonetic confusability demonstrating how the method behaves for the most complicated cases of recognition.

Cons:
- General application for entity recognition with the proposed approach will be not super easy: besides ASR system there should be also force-aligning system which should extract segments where entities are presented. And the most complicated task is obtaining these force-aligned segments.
- Experiments are done on the private datasets, so results in the paper are not reproducible and not comparable with the previous and future works. 
- There is no analysis what is the force-aligning error presented in the data.
- There is no comparison with the other acoustic features instead of "postreriograms" ("posteriograms" could have errors of ASR model itself and possibly not be ideal for acoustic embedding construction) both for triplet loss and proposed method. Previous work He et al., 2017 used MFCC for the triplet loss. Thus to demonstrate that proposed approach is well suited for the task it should be also shown that it outperforms the triplet loss on MFCC input.

Comments:
- "However, none of the aforementioned papers have reported results in isolated word recognition.":  Jung et al., 2019 reports average precision of isolated word recognition (if audio and text are given for the same word). Otherwise, "isolated word recognition" should be clarified in the paper.
- Would like to have an ablation when instead of "posteriograms" raw wave or MFSC/MFCC features are used to investigate 1) what is the effect of ASR system errors on the embedding learning 2) what performance we loose if switch to the "posteriograms" with small network $f(x)$. Experiments for triplet loss should be provided too for this case (it could be that triplet loss behaves better with not "postriograms" input).
- What are the mean and std of the values for $T$ and $M$ in the data (for both monophone and grapheme cases)?
- It is better to move section 3.3 into experiments section 5 otherwise it is not clear with which model $L_2$ distances are computed for examples (Table 1). This is more analysis of a model trained on particular dataset.
- In formulas (13) and (14) the same notation for loss is used $\mathcal{L}_\text{trip}$: the first one is for one sample and the second - for all. Usage of different notation will improve readability.
- Why is silence excluded from the $g(y)$ training? It gives the word boundaries notion so the confusion for the text could occur in the silence absence.
- Could authors clarify what they mean with the "regression layer" (just linear layer?)?
- Comparison with FST baseline is not clear. In case of FST we can generate any sequence of words (say, we have "Katarina F." and "John S." in the phonebook, then FST potentially can infer also "John F." or "Katarina S.", while ANE cannot generate this as we have text embeddings only for original "Katarina F." and "John S.", thus accuracy for FST will be underestimated in this comparison). Could authors clarify this?
- It would be interesting to see accuracy on in-vocabulary and out-of-vocabulary in Table 4 (with the number of such examples) to have the better comparison with FST (showing that in-vocab has the same performance as FST and what is peformance for pure out-of-vocab).

The idea of the paper is very interesting and based on well known and effective SNE approach. The paper itself very well written, with enough clarification to understand the main idea. However, the main concerns of the paper are:
- usage of private data
- lack of the experiments with MFCC input for both triplet loss and proposed loss to show the new method effectiveness (one of the paper claims is that the method is better than triplet loss)
- clarification on the FST evaluation is needed (one of the paper claims that the methods performs similar to FST).

### Edit based on the authors' response

I believe the authors have addressed part of the major concerns that I and the other reviewers had. Comparison with FST approach and triplet loss is clarified and supported by the extensive experiments now, however, most important things are in Appendix only. Based on the updated paper's version I change my rating from "5: Marginally below acceptance threshold" to "6: Marginally above acceptance threshold".

One of the reviewers mentioned about comparison with language model (LM): here we could use character/phoneme based LM and vocabulary which can help to solve ambiguity too. So this could be considered as a good experiment for future work to show the great potential of ANE if it outperforms LM usage. 

Still I have several concerns, probably more philosophical from some point of view:
- Rely on the force-aligning data for practical applications (in experiments it was ideal segmentation)
- Usage of private data in experiments.

Authors state "The goal of our paper is to introduce ANE and highlight some interesting things about it.". I don't see any points in the paper and author's comments why this is helpful/applicable/better than some another ideas. Authors mentioned in the comments that they cannot state that ANE is faster than the FST while having the same performance (this could be one good point that we have speed up using small embeddings + simple L2 distance computation as pointed by one the reviewers). About applications for continuous speech authors gave the comment "However, we are not even sure if speech recognition per se is the best application for ANE. We are hoping that the community will find other interesting uses, either with the embeddings themselves or the distances between embeddings.". Thus, I am feeling that the paper is not finished in that respect. 

review_title: Novel approach to construct acoustic embeddings with SNE idea behind. 

Meta-review: This paper propose to learn the embedding of audio segment in the framework of stochastic neighbor embedding (SNE), where the embeddings of the same word shall be close to each other. The method was initially demonstrated for name recognition. The use of SNE for acoustic embedding is novel and this is recognized by all reviewers. There has been quite some discussions between the authors and reviewers/AC, and the papers has got improved since. To summarize:
1. The discussion of the properties of SNE (the reduction from SNE to weighted least squares terms) was not accurate and confused multiple reviewers. The authors have made some clarifications and added citations. As the author claim this to be a contribution, I feel this part of the main text can be further strengthened and made more accurate.
2. For the experiment in main paper, the comparison between proposed method and prior work was not fair since the proposed method use outputs from an ASR system to obtain phone posteriors. The authors then added more results for the word discrimination task in the appendix. But reviewers are still concerned that the authors are not comparing with the strongest variant of He et al, 2017, and that comparisons are shown for embeddings with low dimensions. The reviewers believe this set of experiments shall be more illuminating, and be moved to the main text.
3. The biggest concern at the intuition level is whether it is the best choice to make the affinity binary, which does not take into account the more fine-grained similarity between different words. Quoting the comment from Reviewer 6:  "The argument of trying to be as simple as possible is reasonable, but we would have liked to see it motivated by some experiments. Something along the lines of a method which introduced rudimentary edit distance-based affinities and then presented their version with hard affinities, and then show some results comparing them. These edit distance-based versions could be as simple or complicated as they felt necessary, but it would be nice for some comparison to be made in order to then dismiss them."

Overall, we think the paper is borderline in the current stage, and the paper can be further improved if the above concerns are properly addressed.

Decision: Reject