Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an approach to learning word vector representations for character sequences and acoustic spans jointly. The paper is clearly written and both the approach and experiments seem reasonable in terms of execution. The motivation and tasks feel a bit synthetic as it requires acoustics spans for words that have already been segmented from continuous speech - - a major assumption. The evaluation tasks feel a bit synthetic overall and in particular when evaluating character based comparisons it seems there should also be phoneme based comparisons.

There's a lot of discussion of character edit distance relative to acoustic span similarity. It seems very natural to also include phoneme string edit distance in this discussion and experiments. This is especially true of the word similarity test. Rather than only looking at levenshtein edit distance of characters you should evaluate edit distance of the phone strings relative to the acoustic embedding distances. Beyond the evaluation task the paper would be more interesting if you compared character embeddings with phone string embeddings. I believe the last function could remain identical it's just swapping out characters for phones as the symbol set.  finally in this topic the discussion and experiments should look at homophones As if not obvious what the network would learn to handle these.

 the vocabulary size and training data amount make this really a toy problem. although there are many pairs constructed most of those pairs will be easy distinctions. the experiments and conclusions would be far stronger with a larger vocabulary and word segment data set with subsampling all pairs perhaps biased towards more difficult or similar pairs.

 it seems this approach is unable to address the task of keyword spotting in longer spoken utterances. If that's the case please add some discussion as to why you are solving the problem of word embeddings given existing word segmentations. The motivating example of using this approach to retrieve words seems flawed if a recognizer must be used to segment words beforehand 
review_title: null

final_score: rating: 6: Marginally above acceptance threshold
review_content: this proposes a multi-view learning approach for learning representations for acoustic sequences. they investigate the use of bidirectional LSTM with contrastive losses. experiments show improvement over the previous work.

although I have no expertise in speech processing, I am in favor of accepting this paper because of following contributions:
- investigating the use of fairly known architecture on a new domain.
- providing novel objectives specific to the domain
- setting up new benchmarks designed for evaluating multi-view models

I hope authors open-source their implementation so that people can replicate results, compare their work, and improve on this work.
review_title: well-done domain adaptation

final_score: rating: 5: Marginally below acceptance threshold
review_content: Pros:
  Interesting training criterion.
Cons:
  Missing proper ASR technique based baselines.

Comments:
  The dataset is quite small.
  ROC curves for detection, and more measurements, e.g. EER would probably be helpful besides AP.
  More detailed analysis of the results would be necessary, e.g. precision of words seen during training compared to the detection
  performance of out-of-vocabulary words.
  It would be interesting to show scatter plots for embedding vs. orthographic distances.

review_title: The paper investigates jointly trained acoustic and character level word embeddings, but only on a very small task.

final_score: rating: null
review_content: How big are the training/dev/test sets in seconds?
Having 60M test word pair samples, how many samples have the answer yes, how many no? (Prior probabilities of the 2 classes)
Saying always "no" (different words) seems to be a good strategy.

Why have the authors not considered proper ASR techniques as baselines:
  a, Cross-view word discrimination: Measuring score difference between a background HMM model and a word model, a la keyword spotting.
  b, Acoustic word discrimination: Training phone/grapheme based HMM neural network acoustic model, phone posteriors could be extracted.
     Using DTW in posterior feature space with KL-divergence would definitelly result in a better DTW baseline, aka. posterior based features in template matching.

review_title: questions

final_score: rating: null
review_content: - have you tried using all combination of losses and doing ablation for each one of them? that would be informative.
- have you tried pretraining characterLSTM on a large text corpus to initialize your model? I think your dataset is too small to learn a good character model.

MINOR issues:
- in figure-1 half of $f_2(x)$ is missing.
review_title: combination of losses & pretraining

Meta-review: The paper explores a model that performs joint embedding of acoustic sequences and character sequences. The reviewers agree the paper is well-written, the proposed loss function is interesting, and the experimental evaluation is sufficient. Having said that, there are also concerns that the proxy tasks used in the experiments are somewhat artificial.
Decision: Accept (Poster)