Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new method to do domain adaptation without textual data from target domain for Transducer based ASR. Text-only adaptation for ASR enables a compact model for target domain without relying on external LMs. The idea proposed in this paper is also novel and interesting. Experiments on a small and medium size ASR data shows the improvement over multiple related baselines while maintains the performance on source domain.
strength_and_weaknesses: Strength
1. Novelty; The idea proposed in this paper indeed meet the requirements mentioned in this paper for text-only domain adaptation, which are high accuracy, no retraining, no impact on inference speed and no deterioration on source domain. As far as I know, previous studies fail to meet all four requirements. 
2. Intensive experiments and ablation studies show the effectiveness of the design choices.

Weakness
1. The dataset used in this paper is not large enough. There exists gigaspeech with 10k hours training data. With large dataset, it could relieve the overfitting on source domain data and provide better generalization ability, thus making the results more convincing.
2. Transducer based models are specific to ASR domain, thus it's hard to apply the method in this paper to other fields, such as speech translation/machine translation domain, making the contribution of this paper less accessible. If similar idea could be applied to LAS based ASR, it could be better. However, the current model design can only be used to transducer based ASR.
clarity,_quality,_novelty_and_reproducibility: Clarity: clear enough
Novelty: good.
Reproducibility: easy if the author can make the code available.
summary_of_the_review: Given the strength and weakness, the reviewer likes the novelty and still intend to have this paper accepted.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel text-only adaptation method for RNNT-based ASR systems. Unlike previous approaches (shallow fusion, TTS, fine-tuning), the proposed method called TOLSTOI is lightweight with a cheap training cost. The key idea of TOLSTOI is to train an “imputation model" that mimics the speech module outputs from the language module outputs. The imputation model is trained first, and then the language and joint models are fine-tuned. Experimental results show that the proposed method achieves the lowest WER when the source and target domains are different.
strength_and_weaknesses: Strengths:

- The paper tackles important and practical problems with a simple approach. The motivation of the paper is meaningful, and the related works are studied well and clearly demonstrated in a general framework (Figure 1).
- The success of FixedGram is interesting that a fixed number of blanks are sufficient for pseudo-generating the speech features.
- Measuring not only WER but also RTF clearly shows the advantage of the method. Also, it is good to consider catastrophic forgetting.

Weaknesses:
- The model architectures are slightly old-fashioned. How about trying Transformer-based acoustic models instead of LSTM-based ones? Also, I guess that in the case of ContextNet or Conformer, convolution operations would distribute the acoustic features and FixedGram may not work well.
- As I understand, the imputation model is removed after the adaptation during inference. It would be better if authors investigate the difference between learned IMP output and actual output from speech module (for both source and target domains).
- Just a question; training takes multiple steps; is “end-to-end” an appropriate explanation? It seems that there are many manual steps, for example, extracting alignments and training different modules with different hyperparameters.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to understand. The idea is novel, and I believe this work can motivate many others. The paper supports the claim with key experiments. It seems that the training and architecture details are explained sufficiently, but I am not fully sure if there are some missing details.
summary_of_the_review: Overall, the paper is novel and tackles an important problem. The strengths are strong, and the weaknesses are minor. Although the scalability to other models needs further validation, I recommend this paper be accepted.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a text-only adaptation method for RNN-Ts, allowing to make these models better for a new domain than the one represented in the training data. While existing methods require a change in the RNN-T architecture, a full retraining, or introduce some latency, the proposed approach allows to only fine-tune the language and joint network, similarly to methods relying on TTS. Here, instead of generating audio for domain-specific texts using TTS, the output representations of the speech model are generated from the text directly. 

The model that generates these intermediate representations is trained using the forced alignments of the labeled training dataset. A small recurrent neural network is trained to predict the next hidden speech vector from the current one and current language hidden vector. To adapt to a new domain, alignments are first generated from the text, and the small network is used to predict a sequence of hidden speech vectors. These vectors are in turn used to fine-tune the language and joint networks.

Through experiments with two base training sets and 3 adaptation domains, the paper shows that the proposed method with a 56M-parameter model outperforms the existing adaptation approaches without a significant increase of latency or degradation of performance on the original domain.
strength_and_weaknesses: Strength: the proposed method is very interesting, does not require to retrain the whole network, gives good results, and seems quite simple to implement and to test. Among the different design choices, the simplest ones (L1 loss, fixed gram for alignment generation, imputation network architecture) seem to work best, which is also a nice result. Finally, a pretty small imputation network looks sufficient.

Weakness: The evaluated RNN-Ts predict characters, while is looks more common to use subwords. It would be nice to see if the results tranfer also to subword modeling. Although the focus here is not really rare words or named entities, I would also be curious to see the results for this challenging task, compared to biasing techniques.

clarity,_quality,_novelty_and_reproducibility: The paper is very well written, easy to follow and to read and describe a compelling idea. It is well structured and contains the right level of details. 

The idea seem novel, although it could be seen as an evolution of the textogram method or something between TTS generation and textogram. It addresses a very relevant problem for end-to-end ASR and its adaptability to real-life industrial scenarios. 

It contains a lot of details that should help reproduce the results, as stated by the authors, although making the code or a recipe available would be even better.
summary_of_the_review: The paper is good and the idea worth sharing with the community. The ability to adapt an end-to-end ASR model is important and represents a challenging task, and the presented method allows to do so without changing the RNN-T architecture and without requiring a complete fine-tuning of the model, which not only saves energy but also allows to use that method without requiring a lot of computational power. 

For these reasons, I think the paper is interesting for ICLR.

Some clarifications and additional experiments could nevertheless add more value to this paper:

   - is teacher forcing used for training imputation model?
   - to measure catastrophic forgetting, WER is computed on a source-target mixture dataset: why not measure on source and target separately?
   - is the alignment sampling method (and the overall adaptation technique) also good when the model outputs subwords instead of characters?
   - the "Real Time Factor(RTF) which measures the average time (in seconds) to decode 50 audio frames": 50 audio frames is indeed, in the described setup, 1 second of audio: it would be clearer and more consistent with the literature to define RTF as the total processing time divided by the actual audio duration maybe?
   - more analysis of the representations learned by the imputation model, and maybe a focus on rare words would be very interesting for an ICLR paper
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies RNN-T ASR model adaptation using unpaired text in the target domain. The authors proposed to use a module called “Imputation Model” which predicts speech representation of the speech encoder ($M_s$) given text, such that text encoder $M_L$ and the joint network $M_j$ in RNN-T can be fine-tuned using text and pseudo speech embedding.

The use of the imputation model is compatible with any existing RNN-T model, does not increase the total number parameters during inference, suffer milder catastrophic forgetting, and yields better performance compared to the included baselines (shallow fusion, $M_L$ as LM, textogram).

strength_and_weaknesses: Strengths
* The design of the imputation model is well motivated (keeping inference time short, drop-in replacement for existing models, good performance on both target and general domain)
* It is very easy to follow the paper and all details are included.
* Performance is strong compared to the baselines. Ablation studies are informative, showing a) the impact of loss function, b) simple alignment generation is sufficient, c) reusing $M_L$ embedding simplifies the imputation model.

Weakness
* The idea of generating internal representations of an end-to-end model to fine-tune a subset of parameters has been studied before [1,2]. I can see that the design of the “imputation models” (or so-called text-to-embedding (TTE) models in prior studies) are different. However, the authors should discuss these studies and present experiments to compare with prior models.

[1] Hayashi, Tomoki, et al. "Back-translation-style data augmentation for end-to-end ASR." 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018.

[2] Hori, Takaaki, et al. "Cycle-consistency training for end-to-end speech recognition." ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.

clarity,_quality,_novelty_and_reproducibility:  Quality and presentation are good. Details are provided and hence it would not be hard to reproduce. However, similar ideas have been presented in prior studies.
summary_of_the_review: This paper presents a practical solution to RNN-T ASR model adaptation using unpaired text. Experiments and presentations are good. I would encourage the author to improve the comparison with existing works.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: the paper presents a novel text-only adaptation method for RNN-T based ASR system. It is a lightweight adaptation technique. Experimental results show that the proposed method achieves better quality to address domain differences. 

Strengths: The idea is novel, detailed experiments and ablation studies are conducted to justify various design choices.

Weaknesses: The current algorithm design and experimental justification are only for the transducer based ASR. 
justification_for_why_not_higher_score: The scope of the current paper is limited to transducer based ASR. 
justification_for_why_not_lower_score: The proposed method is novel and has been thoroughly justified.
Decision: Accept: poster