Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: This seems to be a task in which the data doesn't have a gold standard label, you mention that "experts have not yet decided how to chunk the signals of species". During your evaluation, you say that your approach shows promise because a bioacoustic expert analyzed your results and validated the proposed representation, but it is not clear what the baseline is? How does your approach compare to current approaches in the field?
review_title: Regarding evaluation

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper is a novel application for the sticky HDP-HMM, focused on correctly identifying the number of components in bird and whale song across a variety of datasets. It's nice to see the model applied to an interesting dataset. My main issues with the paper have to do with structure and the choice of representation used in the model. Namely:

The organization of the paper could be significantly improved. There is a lot of repetitive introduction that adds little to the paper. The first and last two sentences of the abstract could be cut. Many other parts of the abstract basically repeat the introduction. The second paragraph of section 2.3 also repeats your introduction - by now we know what you're doing. I think most people reading this will have no idea what Kershenbaum (2014) is. The description of the data should go in the experiments section. "Different hypotheses for the songs were emitted" in the introduction is odd phrasing. Figure 4 should be the first figure and go in the introduction. Figure 5 should be in the methods section. A summary of Table 1 should be in the experiments section. Generally the writing could be tightened quite a bit, which would make space for these figures. The description of the HDP-HMM, which mostly follows the existing literature, is well done.

Some general questions about the methods used:

If you're interested in scalable inference, why use Gibbs sampling? Why not the beam sampler (van Gael 2008), which at least recently was the state of the art for MCMC inference in the HDP-HMM? More generally, why use MCMC at all? For very large datasets, most of the Bayesian ML community has converged on stochastic variational inference as the most practical method (eg Wang, Paisley and Blei 2011).

If your interest is mainly in the number of clusters, how would you address the fact that DP mixture models are known not to be consistent for estimating the true number of clusters (Miller and Harrison 2013)?

MFCC features are calibrated to the human auditory system, not bird or whale auditory systems. In your data, do you calibrate the MFCC scale to be closer to the auditory systems of the animals that generated the song?

And a final suggestion for future work, which could use the results presented here as a baseline:

Given the success of LSTMs in speech recognition in recent years, it may be the case that deep learned representations are superior to linear features (like the means of each cluster in an HDP-HMM) for animal song as well. Have you considered a hybrid model, similar to recent work combining autoencoders and graphical models (Johnson, Duvenaud, Wiltschko, Datta and Adams 2016)?
review_title: A nice application of a well-established model to a novel dataset

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presented an unsupervised approach for the automatic segmentation of bioacoustic data. The authors applied an existing approach (Hierarchical Dirichlet Process Hidden Markov Models) to their task. The originality of their work is the investigation of this approach on a new task, which they argue is more difficult, namely bioacoustic segmentation. They provide evidence that this is a difficult task by explaining that there doesn't exist a consensus among human experts on how this should be done. However, they do not provide convincing results that their approach is successful, as it fails in many cases to replicate the correct segmentations as defined by their baseline: human experts. In addition, the clarity of the writing is extremely poor, including many grammatical errors and awkward sentences. 
review_title: Application of existing approach to a not particularly clear problem

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper applies HDP-HMM to challenging bioacoustics segmentation problems including humpback whole sound and bird sound segmentation. Although the technique itself is not novel, the application of this data-driven method to bioacoustics segmentation is quite challenging, and may yield some scientific findings, and this is a valuable contribution to the bioacoustics field. My concern for this paper is that it does not have fair comparison of the other simple methods including BIC and AIC, and it is better to provide such comparisons. Especially, as the authors pointed out, the computational cost of HDP-HMM is a big issue, and the other simple methods may solve this issue.
review_title: null

final_score: rating: null
review_content: What is a novel aspect of this technique? It seems that the technique itself is not novel (but application to bioacoustic segmentation is novel)?
Also, I'm curious about how the algorithm is sensitive to an initialization and hyper-parameter. Please describe these in more details, and possibly provide further experimental results by changing the initialization and hyper-parameters.
review_title: novelty and initialization

Meta-review: The paper studies an unusual and (apparently) challenging application -- segmentation of whale and bird songs. Though the authors claim that the applications is much more challenging than previous applications of the proposed techniques (in speech processing), the evaluation is very questionable (as there is no gold standard), there is no convincing comparison with other (potentially simpler techniques). Overall, the reviewers believe the work is not mature enough to be accepted at ICLR.
 
 + interesting dataset / task 
 
 - novelty is limited
 - evaluation is weak
 - writing is poor
Decision: Reject