Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Pretty interesting paper attempting to learn discrete linguistic units via vector quantization of visually grounded, speech related features. I think this is a worthwhile contribution. My main complaint is that the exposition is a bit diffuse and fails to crystallize the essence of the work. In particular, the claim is that the novelty is from the use of a "discriminative, multi-modal grounding objective". Reading the work, this seems to be the triplet loss described in Section 3.5. Is that the novel objective? In my my the really interesting aspect that should be stressed is the visual grounding -- I encourage the authors to highlight that aspect more directly. I fear that essential and interesting point is somewhat diluted in the detailed exposition of results.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper attempts to learn discrete speech units in a hierarchical (phone and word) fashion by incorporating multiple vector quantization layers into the audio encoder branch of a model that visually grounds speech segments with accompanying images.

The model has been tested and compared against two algorithms and implementations that set the SOTA on the Zero Speech 2019 challenge (further improving one of them in the process, it seems), and outperforms these significantly using the ABX metric, so the proposed method seems to perform well (the model is using additional supervision, though). In addition, this is an interesting and timely research problem with implications far beyond the core machine learning setup. The hierarchical setup, and the finding that successful learning here depends on the curriculum, is intriguing indeed. The paper is a pleasure to read and provides a rich set of results and analyses. 

A few remarks:
- It is probably worth explaining how the ABX test is performed, i.e. that features are extracted from some layer of a model, and then a time alignment is performed to get the score - this is in the text somehow, but i had to read it multiple times.
- Did you try other architectures like 5 layers (rather than 4) in Figure 2
- Figure 2 is a bit hard to interpret. Maybe plot log of ABX error rate or something, to pull apart the different layers?
- Could you explain the difference between cold-start and warm-start? One is adding the discretization to a pre-trained model, the other is training from the start?
- When you measure ABX at layer 2 and 3, in a model trained with quantization, do you measure ABX on the features before or after quantization? does it make a difference?
- Table 7: some of the top word hypothesis pairs make sense acoustically (building-buildings, red-bed, ...), some could be neighboring words (large-car, ...), but some are just weird (people-computer) - any intuition as to what is going on?
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Overview:

The paper proposes a method to learn discrete linguistic units in a low-resource setting using speech paired with images (no labels). The visual grounding signal is different from other recent work, where a reconstruction objective was used to learn discrete representations in unsupervised neural networks. In contrast to other work, a hierarchy of discretization layers are also considered, and the paper shows that, with appropriate initialization, higher discrete layers capture word-like units while lower layers capture phoneme-like units.

Strengths:

The paper is extremely well-written with a clear motivation (Section 1). The approach is novel. But I think the paper's biggest strength is in its very thorough experimental investigation. Their approach is compared to other very recent speech discretization methods on the same data using the same (ABX) evaluation metric. But the work goes further in that it systematically attempts to actually understand what types of structures are captured in the intermediate discrete layers, and it is able to answer this question convincingly. Finally, very good results on standard benchmarks are achieved.

Weaknesses:

Although I think the paper is very well-motivated, my first criticism is that discretization itself is not motivated: why is it necessary to have a model with discrete intermediate layers? Does this give us something other than interpretability (which we obtain due to the sparse bottleneck)? In the detailed questions below, I also specifically ask whether, for instance, the downstream speech-image task actually benefits from including discrete layers.

My second point is that it is unclear why word-like units only appear when the higher-level discrete layers are trained from scratch; as soon as warm-starting is used, the higher level layers capture phoneme-like units (Table 1). Is it possible to answer/speculate why this is the case?

Overall assessment:

The paper presents a new approach with a thorough experimental investigation. I therefore assign an "accept". The weaknesses above asks for additional motivation and some speculation.

Questions, suggestions, typos, grammar and style:

- Section 3.3: It maybe makes less sense for the end-task, but did the authors consider discretization on the image side of the network? This could maybe lead to parts of objects being composed to form larger objects (in analogy to the speech network).
- Section 3.3, par. 3: "with the intention that they should capture discrete word-like and sub-word-like units" -> "with the intention that they should capture discrete *sub-word-like and word-like units*" (easier to read with first part of sentence)
- Section 3.3: The more standard VQ-VAE adds a commitment loss and a loss for updating the embeddings; was this used or considered at all, or is this all captured through the exponential moving average method?
- Section 3.4: "with same VQ layers" -> "with *the* same VQ layers"
- Section 3.5: Can you briefly outline the motivation for adding the two losses (so that it is not required to read the previous work).
- Section 4.1: Following from the first weakness listed above, the caption under Figure 2 states that the non-discrete model achieves a speech-image retrieval R@10 of 0.735. This is lower than some of the best scores achieved in Table 1. Can this be taken as evidence that discretization actually improves the downstream task? If so, it would be worth highlighting the point more; if there is some other reason, that would also be worth knowing.
- Figure 1: Did the authors ever consider putting discrete layers right at the top of the speech component, just before the pooling layer? Would this more consistently lead to word-like units?

review_title: Official Blind Review #1

Meta-review: The paper is extremely well-written with a clear motivation (Section 1). The approach is novel. But I think the paper's biggest strength is in its very thorough experimental investigation. Their approach is compared to other very recent speech discretization methods on the same data using the same (ABX) evaluation metric. But the work goes further in that it systematically attempts to actually understand what types of structures are captured in the intermediate discrete layers, and it is able to answer this question convincingly. Finally, very good results on standard benchmarks are achieved.

To authors: Please do include the additional discussions and results in the final paper. 
Decision: Accept (Talk)