Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a new method to segment pseudo-phoneme boundaries by combining multi-level CPC and boundary predictor agent. The reward for the boundary predictor agent is the high-level CPC objetive, thereby producing sparse boundary flag. The learned representations show nice performance on phoneme prediction and boundary detection tasks.
strengths_and_weaknesses: strengths

- The idea of training boundary predictor using reinforcemnent learning is a clever idea.

weakness

- The performance of the training algorithm significantly reduces when there is no average frame sample rate regularization (Eq. (7)). This implicitly demonstrates that the agent fail to find the boundaries without the help of regulaizration and show that the reward function for the agent may not be an appropriate one.
- Maybe the proposed training scheme cannot really gurantee to find the optimal boundary if there is no endeavour to reconstuct the original signal.
- It'd have been nicer if there were some qualitative analysis.
questions: - line 92, conditioned on **c** → doesn’t make sense judging from Figure 1. I guess it’s **z** not **c**?
- line 92, and and ?
- line 132, again, c → z
- what are **p**s in equation (1) and (2)? what nn architecture was used to produce this?
- why does the policy trained to minimize the expected value of L_H makes the agent predict the phoneme boundaries? Is there any guarantee for this?
limitations: It is written in the disccussion part.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed to achieve unsupervised acoustic unit discovery in speech using a specially designed variable-rate hierarchical CPC method. The low-level CPC is applied at a frame-level while the high-level CPC is applied by non-uniformly downsampling the low-level CPC outputs. The non-uniformly downsampling is performed based on high-level unit segmentations estimated by a boundary predictor, which is trained using REINFORCE. 
strengths_and_weaknesses: Strengths:
The paper is easy to follow. The proposed method is novel and does technically sound from the speech perspective.  

Weaknesses:
1. Figure 1 should be redesigned to be more descriptive and better looking. The resolution should be considerably improved and better to use a vector image. 
2. The formats of the references are not consistent. 
3. The results may not be strong enough to demonstrate the superiority of the proposed method. For instance:
(1) In Table 1, the improvement in phone accuracy is small over existing method. I am not sure how sensible it is to evaluate the frame accuracy of phonemes here (as the phoneme boundaries are known to be noisy).
(2) In Table 2, although the proposed method achieved the best phone accuracy among the downsampling-based methods, the result is still quite a bit worse than the no-downsampling method. 
(3) In Table 3, the phone boundary detection results are quite a lot behind previous numbers. 
questions: 1. "We consider that an interesting objective for future work is characterizing the patterns of the discovered pseudo-units sequences, and perhaps investigate their correlation with models of human speech perception."

This seems to be a very interesting direction, but it's very possible that human and machines have different solutions. I wonder how do the authors think to make the findings useful rather than "inventing" a different phonology. 

In particular, there exists very-well performed algorithms to achieve automatic frame-to-subword-unit alignment without requiring any phone labels. This could easily achieve much better phone accuracy and possibly a better averaged sampling rate. 

2. What if some collar (e.g. 1-2 frames) is allowed in the phone boundary detection experiments? 
limitations: 1. This research direction is certainly very interesting from a scientific perspective, but I would still hope the authors to discuss more about the potential use of such automatically-discovered acoustic units. 

2. Hope to see more statistics and analysis about the automatically discovered high-level acoustic units. It's unclear to me how many of them were found, how well they correspond to the phones, and how about their durations etc.?

3. Acoustic unit generation, clustering and discovery have been long-standing research problems in the speech community. Although the related work tried to cover some recent influential works related in the NLP community, the pioneering and more directly related works from the speech communities are mostly ignored. Actually it would be quite interesting to digging into those works and find the connections and relations. At least, even if the current framework of related work is kept, this paper should be cited as it is the original paper for the widely used wordpiece models:
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/37842.pdf
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies unsupervised acoustic unit discovery using a hierarchical CPC training criterion. In particular, the CPC loss is computed at two levels: frame-level and segment-level. Segmentations are produced by a jointly trained boundary detector, which can be understood as a policy network that minimizes the top-level CPC loss optimized using the REINFORCE algorithm. 

The authors further introduce two crucial designs, without which the secondary CPC does not improve the performance from single-level CPC. The first design is how distractors of the top-level CPC are selected. The authors show in ablation studies that it is important to select distractors from the adjacent frames of the target one to encourage contrastiveness. The second is to inject prior knowledge of average phone duration in the boundary prediction module. 

The proposed model achieves better frame-level phone accuracy, segment-level phone accuracy, and phone segmentation compared to several baselines. 

strengths_and_weaknesses: Strengths:
1. The paper is easy to follow, with all the design choices well motivated and clearly described.
2. The experiment section also contains a good set of ablation studies and baselines to justify the importance of design choices. 
3. Strong results compared to the baselines

Weaknesses
1. The authors could have included ABX results on the frame-level feature, segment-level features, and quantized frame/segment features. This would be more directly related to “unit discovery.” The evaluation included in the paper only concerns boundary detection.
2. More ablation studies can be added. 
    - It is shown that selecting adjacent representations of the target one is beneficial for high-level CPC. Would it also be helpful for training low-level CPC?
    - In line 202, the author states “Multi-level models are trained for 50 additional epochs after appending to the pre-trained frame-level modules the high-level network and downsampler, where applicable.” How important is it to pre-train the frame-level modules? It would be nice to also show the results of training everything from scratch.
    - Add boundary detection results to Table 4
3. (minor) The following statement is wrong: “In a similar spirit Lakhotia et al. [2021] trained language models on raw speech. However, in these approaches the second-level models still operated at a frequency synchronous with the input one.” In fact, Lakhotia et al. [2021] and Kharitonov et al. [2021] both use deduplicated unit sequences ([1, 1, 2, 3, 3, 3] -> [1, 2, 3]) for training LMs, so the LM operate on the segment level instead of on the frame level.

Kharitonov, Eugene, et al. "Text-free prosody-aware generative spoken language modeling." arXiv preprint arXiv:2109.03264 (2021).

questions: 1. Why do the authors modify CTC to not predict blank symbols for evaluation if phone accuracy is computed at the sequence level?
2. Following the previous question, do phone accuracy in Table 1 and Table 2 both represent the same metric and are both computed at the sequence level? If they are computed at the frame level / segment level instead, how are the segment level ground truth phones determined if a segment contains frames from more than one phones?

See weaknesses for other questions

limitations: Yes, the authors have addressed limitations.
review_title: null

Meta-review: metareview: In this paper the authors propose to use multiple levels of contrastive predictive coding (CPC) in self-supervised learning to discover acoustic units.  The lower level of CPC learning is a conventional one while the upper level of CPC is carried out based on non-uniform downsampling with boundary prediction. To facilitate the training, the authors introduce various loss functions for hierarchical CPCs, quantization, boundary predictor and average sampling rate regularization.  The authors show that the acoustic units discovered by this framework give better phone accuracies in both frame and segment levels compared to existing models.   The idea of variable-rate hierarchical CPC learning is interesting and experimental results are supportive.  The paper is well written and easy to follow.  Most of the raised concerns by the reviewers have been addressed in the rebuttal.  To further improve the paper,  the authors may want to show the effectiveness of this model to generate better acoustic representations in more downstream speech domain and tasks. 
confidence: Certain
Decision: Accept