Kati8/ICLR_2023_2025 huggingface dataset:
authors: Dongchao Yang
authors: Jinchuan Tian
authors: Xu Tan
authors: Rongjie Huang
authors: Songxiang Liu
authors: Xuankai Chang
authors: Jiatong Shi
authors: sheng zhao
authors: Jiang Bian
authors: Xixin Wu
authors: Zhou Zhao
authors: Shinji Watanabe
authors: Helen M. Meng
reviews: 
Comments: In this paper, the authors train a single multi-scale transformer to perform multiple different audio tasks. They demonstrate that, by learning these tasks together, they get a performance boost for this model, and achieve state of the art results on multiple tasks (against specialist models). They show that the existing model can be fine tuned to perform well on novel tasks. In addition, they describe how to translate multiple input modalities into a universal data space with a neural codec to allow this to happen, and discuss the merits and detriments of this system against other approaches.
Strengths: * All data used is public, so results can be reproduced
* Compute required is not out of the realms of most academic institutions (16 x MI200-64G GPUs ~ $64k) - particularly if, as is suggested, this model is taken and simply fine tuned upon
* I am not aware of another model which has covered all of these tasks nor comprehensively demonstrated that combining different input modalities benefits each task; the authors show the benefit of training on multiple modalities by training their model on just the one task, and then again on all task, and by comparison with the current sota (generally single-task) model
* Care and attention has been taken to compare with recent sota models in each domain, and to perform both objective and subjective evaluations. In addition, an anonymised website of audio generations was provided
Weaknesses: I would like to emphasise that all the following are suggestions for discussion / future work - IMHO this contribution is more than sufficient for publication in its current form.
* Reliant on the input representation - any losses incurred by input representation cannot be modelled, and if a new representation were to be used, a new model must be trained
* As noted in the limitation section, there's no demonstration of fine tuning to an unknown modality (which would require a new special modality token to be included in the vocabulary)
* The evaluation is so wide ranging it's difficult to parse in tables. Perhaps a bar chart or plot could improve the interpretation of relative performance with sota benchmarks?
* Only amazon turk was used for evaluation - a broader human evaluation with subject experts would be very interesting
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper presents a LM-like audio model, that is able to conduct multiple (11) audio generation tasks. Audio is tokenized with a EnCodec-like quantizer to fit for LM. Multi-scale Transformer is used for handling multiple tokens per audio frames. Experiment was conducted by training on 165K hours audio data.
Strengths: The overall approach is sound. It's great to see a single model can handle so many different audio generation tasks.
Weaknesses: 1. The novelty of this paper is very limited. All the components used in the paper are previously existing. The main contribution of this paper is to train a multi-task model with a mixture of multiple datasets. 
2. The overall contribution to the research community seems very limited. There is no much insights can be drawn from this paper to benefit the community. There is no ablation study conducted. It looks more of an engineering work than a research work.
3. The presentation of this paper needs improvement. It seems presented in an eye-catching, but misleading way. For example, it's improper to put Table 1 as the first thing in the content of this paper, because it lacks context and is confusing. Even worse, it's not a fair comparison -- the cited papers didn't include experiments on specific tasks doesn't mean that the methods presented in those papers are incapable of those tasks.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 3 good
Clarity: 2 fair