Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper describes a deep neural network method for discrete tokenization of audio signals for retrieval problems. The model is trained from pairs of audio that correspond to each other. The tokenization has two losses, one is similar to a contrastive version of a VQ-VAE loss and the other is a loss that encourages correspondence of representation between two pairs of audio. This second loss uses CTC which correlates with a low edit distance between pair token sequences.

The method is applied to query by humming and spoken term detection tasks and it is shown to perform better than other alternatives including using large self-supervised representations (wav2vec 2.0) etc.
strength_and_weaknesses: Strengths:
1. A novel method to extract token sequences from audio signals, trainable with pairs of audio signals that should correspond to each other.
2. Comparison of the method with competitive methods on two different audio retrieval tasks.

Weaknesses:
1. Uses BiLSTM models and shows that it is better than using a transformer architecture. For audio, U-net style models are also used in token generation which could be compared.
2. Similarity and difference between the first part of the loss and VQ-VAE loss would be a nice addition to the paper. 
3. Details about how the CTC loss is back-propagated to the encoders is not clear. Does the CTC model use blanks? What happens when the label sequence is longer than the number of frames?
clarity,_quality,_novelty_and_reproducibility: The paper is written mostly clearly. The results seem reproducible. It would help to provide the software.
summary_of_the_review: The method seems novel and seems worth publishing.

Some specific comments are given below:

1. Is the contrastive loss in (3) back-propagated to both p and e?  In VQ-VAE, it is not done to avoid collapse. But here, due to the contrastive nature of the loss, we do not have to worry about that? Some explanations would be nice.
2. In Table 1, what are the columns V, TS and PS? Validation, test and something else?
3. In Table 2, there is no mention of Wav2Tok+Cos which is mentioned in the text. Is it equivalent to Wav2Tok+NoRel in the table which is not mentioned in the text.
4. Do the losses apply to both of the inputs in a pair? Maybe this can be stated more clearly.
5. If the token sequence is longer than the number of frames in the other audio, how does the CTC loss work? Typically, CTC loss assumes the label sequence is shorter than the number of frames and uses blank symbols to fill in the middle.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an unsupervised method of computing discrete token sequences from audio, for the purpose of audio retrieval.
Previous works either:
- Use supervised token sequence labels during training, which requires domain expertise and access to labelled data.
- Train an unsupervised autoencoder on only a single sequence at a time, and are therefore not contrastive in nature.

The proposal in this paper addresses both of these issues, by contrastively discovering a token codebook. A local minimum, where all codebook entries converge to the same value, is prevented by the joint use of a softmax contrastive criterion and a regular K-means re-clustering.

strength_and_weaknesses: Strengths:
- The problem that the proposal in this paper is trying to address is well motivated.
- The proposal is tested on multiple diverse tasks.
- The proposal is explicitly compared against multiple baselines.

Weaknesses:
- In the paper, it is emphasised that the CTC loss is used for the purpose of aligning two audio sequences.  The use of the word "align" in this context may lead to misinterpretation, as it differs from the standard use of the word.  The standard meaning of "align" is to determine which input audio frames correspond to each token in the output.  This is not what CTC is being used for in this paper.  If this is what CTC is being used for in this paper, then this use comes into conflict with the BLSTM topology of the model, because prior works have shown that the temporal flexibility of a BLSTM allows it to learn any arbitrary alignment between the input and output, without needing any realistic correspondence.  The actual use of the CTC loss in this paper is to encourage the model to output similar token sequences for both inputs, which is not what "align" typically means.  Perhaps the authors may consider re-phrasing the description of the motivation for the CTC loss, to abate potential confusion.
- Wav2Tok+Cos is described in the text, and the text seems to imply that the performance results can be found somewhere.  However, these results seem to be missing from table 2.
- During audio retrieval, the query is compared against the database using token sequence edit distances.  This assumes that all non-similar tokens are equidistant from each other.  This equidistance assumption may potentially limit the system performance.  This assumption is not discussed in the paper.
- The unit of measurement in many of the experiment tables is not stated in the table, and a reader would need to search through the text to figure out what the numbers in the tables represent.

clarity,_quality,_novelty_and_reproducibility: The use of English is good.  There are only a few typo errors (e.g. "of of").

The paper comprises multiple complicated ideas, and the authors do an adequate job of explaining them within the constraints of the page limit.

The description of the approach, together with some educated guessing, should allow for replication of the experimental procedure.  However, the modified datasets that were synthesised from existing public datasets have not been released by the authors yet.

summary_of_the_review: The proposed approach seems sufficiently novel.  The proposal seems sufficiently useful in addressing limitations of previous methods.  I do not find any major flaws in the paper.  The experiments seem sufficiently comprehensive.

The Ithenticate similarity is 3%, which is good.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a discrete representation learning strategy for audio sequences designed primarily for retrieval. The learning strategy encourages representations to be similar for pairs of similar inputs, where pairs are created by data augmentation of the input audio. The authors compare their approach on two retrieval tasks: musical query-by-humming and spoken term detection.
strength_and_weaknesses: A strength of this paper is that it focuses on learning discrete representations of audio that are useful for edit distance-based text retrieval, in contrast to other work on discrete audio representations which does not consider text retrieval as a primary goal.

The primary weaknesses of this paper are (1) a lackluster evaluation which makes straw man comparisons against weak baselines and omits comparisons to domain-specific methods, and (2) an unclear description of critical aspects of the proposed methodology.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

This paper is fairly clear overall, however the descriptions of key aspects of the proposed pre-training approach are quite vague. Specifically, the CTC-based alignment loss seems to be of critical importance to the success of this approach, however the explanation of this aspect in Section 5 leaves many questions unanswered. For instance, how is this loss function differentiated if it uses the discrete token sequence \tilde{T}’? Are there extra parameters that are trained to map the embedding vocabulary from Z to alphabet A U {CTC blank token}?

Another aspect which is unclear in this paper is how the training data is used in the query-by-humming example. Is the model pre-trained both on humming recordings and the full music recordings? Also, it seems a bit unconventional to train this representation on such a small / domain-specific dataset. Why not pre-train it on a large corpus of music recordings and then evaluate it on MIR-QbSH?

**Quality**

Perhaps the biggest issue with this work in its current form is the evaluation. The introduction of the paper appeals to the convenience of one-size-fits-all approaches over domain-specific strategies - as such, I expect to see a comparison where the proposed approach comes close to (or possibly even exceeds) the performance of domain-specific approaches.

Table 1 contains the “Triplet” baseline which appears to be a domain-specific approach, but it’s not stated whether this is representative of state-of-the-art for domain-specific query-by-humming. Additionally, the primary comparison seems to be against ED-based search with Wav2Vec2 tokens, but this is a straw man argument as Wav2Vec2 was not designed with ED-based search in mind (unlike Wav2Tok).

**Novelty**: The methods proposed here are reasonably novel - I am unaware of any other work which seeks to learn variable-rate discrete representations of audio that are useful for retrieval via text-based search.

**Reproducibility**: There is likely not enough information in this paper to reproduce the results, however the authors have promised to share code.
summary_of_the_review: Overall, I think this paper has some interesting ideas for a novel application of discrete representation learning for audio. However, a weak evaluation prevents me from understanding the strength of the proposed approach. I think the authors should continue working on this, but I don’t think it’s ready for ICLR at this time.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors address the problem of tokenizing audio data for improving the audio based content retrieval. They propose yet another approach for audio tokenization based on clustering the representation in Vector-Quantized space and forcing the representations to get closer their nearest centroid. The authors evaluate on smaller music and speech datasets for the query by humming and spoken term detection tasks and show better results compared to some of the existing approaches. 
strength_and_weaknesses: Strengths:
1. The paper is easy to understand and the proposed approach seems to be interesting

Weakness
1. There is no clear explanation of why different aspects of the proposed approach are necessary. 
2. The ablation studies presented are not good enough to understand the proposed approach thoroughly. 
3. The authors test the proposed work on a few smaller datasets and claim their approach is better than popular approaches trained on large datasets. This conclusions seems a bit premature as the authors have not tested their approach on large datasets. 
4. The paper is not really an easy read and the presentation can be improved. 
clarity,_quality,_novelty_and_reproducibility: 1. Slightly difficult read. 
2. I am not at all certain that his work can be reproduced without the authors releasing their code base. The descriptions in the paper do not seem to suffice. 
summary_of_the_review: Overall, I feel this is yet another solution proposed for audio tokenization without clear articulation of what problems in the previous approaches are being solved. While this in itself is fine, the paper also does not contain any thorough justification for this design and seems like author's may have a good intuition about this problem which may have gone into the design but as a reviewer some aspects of the design are not at all clear.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper describes a system that learns to encode audio files into sequences of discrete tokens for the purposes of performing sequence matching to a database of such sequences. The approach is evaluated in terms of matching query by humming queries to one another on a dataset of 48 songs each hummed about 10 times as well as a word search experiment with 34 keywords spoken in 337 utterances queried by 1289 queries. In comparison with a number of neural baselines, the proposed approach provided more accurate matches.
strength_and_weaknesses: Strengths: 
* The proposed approach seems to work well on the proposed tasks and datasets
* The idea of using the CTC from one sequence with the labels of a different sequence (and vice versa) is clever and interesting.

Weaknesses:
* Scalability. Fundamentally, the proposed approach takes time linear in the size of the database being searched to perform a single query. In general, it is difficult to achieve sub-linear matching speed in the size of a database for sequence matching.
* Need for sequence matching. Given questions about the speed of sequence matching, I wonder about the need for it. A clear comparison with a system that does not require sequence matching demonstrating that the sequence-based approach is superior would resolve this. For example, the proposed system could be compared with a similar system that does matching based on a fixed length representation. 
* Need for discrete tokens. It is also not clear why discrete tokens are necessary. wav2vec 2.0, which uses discrete tokens has evolved into data2vec, which uses continuous hidden representations as targets of prediction. The reason that discrete tokens are preferable to a continuous representation in this case are not clear. 
* Summarization of tokens. The idea of the replacement of a sequence of the same token with a single instance of it was introduced, but it is not clear why it is necessary. It is not shown in figure 1 and page 8 states "We observe a drop in performance for all audio tokenizers when we apply sequence compression to sequences T and Z." So is it used? Is it necessary?
* Baselines could be better. In addition to the fixed-length encoders above, it would be good to compare with DTW on mel spectrograms, and perhaps one with edit distance-based matching on k-means quantized mel spectrograms.
* No citations are provided in the introduction. While many citations are provided later in the paper, they would be very helpful to the reader in the introduction to support various claims.
clarity,_quality,_novelty_and_reproducibility: Clarity:

* Are the MIP and triplet baselines the systems described by Arcas et al (2017) and Chang et al (2020) or are there differences? This should be clarified.
* While page 3 states that neural network fingerprinter approaches are unsuited to comparing fully produced tracks with user query hums in query by humming, this does not appear to be the task that is evaluated in the experiments. The experiments focus on the more straightforward task of matching two hums together, for which NNFP may be well suited.
* Please use \citep and \citet to cite papers either in parentheses (Arcas et al., 2017) or textually as Arcas et al. (2017) say...
* Data augmentations are shown in figure 2, but it is never stated which augmentations are used in the main body of the paper.
* Approximate string matching is mentioned on page 6, but no specific algorithm or citation is provided.
* The description of the wav2vec baselines should be clarified to highlight what exactly is different between them. Presumably it is mainly the data used to train them, but this is not clearly stated.
* The dataset for the speech queries is not clearly explained. How were the target words selected? How were the sentences selected from TIMIT? How were they put together? Were there artifacts from this combination process? Does each sentence in the database just have a single target word or could any word in it be targeted? How does this task compare to the others mentioned in the introduction?
* Wav2Tok+Cos is mentioned on page 8, but is not present in table 2
* "Locally sensitive hashing" -> "Locality sensitive hashing"

Novelty:
* The approach appears to be novel

Reproducibility:
* The authors state that they will release the code after the review process, although there is no way to ensure this.
summary_of_the_review: Overall, this approach is not especially well motivated, but does appear to work better than several baselines (although these are perhaps not the best baselines) on two experiments involving rather small datasets. Several important details of the approach are not entirely clear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper the authors propose a robust and fast audio retrieval approach that is based on concise discrete representation of an audio sequence.  The audio retreival problem is then converted to a matching problem between the tokenized query and target sequences.  The learning of the discrete representations is conducted using the paired signals of the original audio and its data augmented version based on CTC and quantization.  Experiments on query by humming (QbH) and spoken term detection (STD) tasks show supportive results.   The authors cleared most of the concerns raised by the reviewers in their rebuttal, especially on the motivation, justification and implementation details of CTC which plays a crucial role in the learning.  The authors also added new experiments to verify the performance of Wav2Tok on additional datasets and domain-specific baselines.  All reviewers consider the idea interesting and the work technically novel.  The performance is good on the two investigated tasks although there are lingering minor issues that need to be taken care of in the final version.  Last but not least, the paper has a formatting issue where the fonts are too small in the tables.   
justification_for_why_not_higher_score: There are still some issues with the work, although they are considered relatively minor. 

1. The proposed Wav2Tok is supposed to be robust (through quantization) and fast (through shortened tokenization sequences leveraging CTC).  Although the speed advantages have been demonstrated in the experiments, the robustness is only partially verified. 

2. As a generic technique that can be used for QbH and STD,  how does it compare with domain-specific techniques (e.g.  OOV issues in STD, generalization capability to other domains, etc..) is still not very clear.

justification_for_why_not_lower_score: The idea is interesting.  The work is technically novel.  The experimental results are good.
Decision: Accept: poster