SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CoDA is evaluated on three domains including natural language processing (NLP), computer vision and speech processing, and on a range of applications such as classification, question answering, summarization and speech recognition. The experiments are organized as follows: We first demonstrate the effectiveness of CoDA conduct analyses on its design choices using the publicly available T5 models ( $\S 5$ ). In our final results ( $\S 6$ ), we pretrain Transformer models from scratch and extend our evaluation to vision and speech domains.
Datasets We use the C4 corpus [Raffel et al., 2020] for pretraining text models. For speech models, we use the LibriLight corpus [Kahn et al., 2020] for pretraining. Our vision Transformer models use the same data and training procedure in Pix2Struct [Lee et al., 2022]. Our finetuning datasets for text models include the MNLI [Williams et al., 2018], RTE [Dagan et al., 2005, Haim et al., 2006, Giampiccolo et al., 2007, Bentivogli et al., 2009], BoolQ [Clark et al., 2019], SQuAD [Rajpurkar et al., 2016] and XSum [Narayan et al., 2018] datasets. The speech models are evaluated on the speech recognition task using the LibriSpeech dataset [Panayotov et al., 2015]. Finally, we use the OCR-VQA [Mishra et al., 2019], DocVQA [Mathew et al., 2021], and Screen2Words [Wang et al., 2021] datasets for vision models.
Model	Reduction r	Base	Large	XL	Δ Avg
MNLI	RTE	BoolQ	MNLI	RTE	BoolQ	MNLI	RTE	BoolQ
Soft top-k	86.3	72.2	76.2	89.8	87.0	83.7	91.4	89.5	88.2	±0.0
Sigmoid gate as f(s)	3	85.7	70.8	72.8	89.2	82.3	81.0	90.6	88.1	86.2	-2.0
Truncation – selecting first k	81.1	70.8	72.7	84.9	77.3	82.3	85.6	84.5	85.4	-4.4
Soft top-k	82.5	70.8	75.4	88.1	87.0	81.8	89.9	87.7	84.8	±0.0
Sigmoid gate as f(s)	5	82.9	71.5	72.1	86.7	82.3	80.1	88.3	87.0	82.4	-1.6
Truncation – selecting first k	62.2	64.6	71.1	64.9	70.4	75.4	66.6	76.2	81.1	-12.9
Table 3: Ablation study on routing methods. We use CoDA  $k$ -to-  $k$  variant for a fair comparison with the truncation method. Better routing method delivers better accuracy on various tasks and model sizes tested. We use soft top-  $k$  as our default method.
5 Understanding and Analyzing CoDA
Setup We present several analyses to validate the design choices of CoDA in this section. We initialize CoDA using the version 1.1 release of T5 checkpoints², and perform CoDA pretraining using the same setting as the T5 models. During pretraining, we set routing capacity to  $k = 192$  given input sequence length  $n = 512$ . We do not tune the value of  $k$  for pretraining, but will report the results of using different  $k$  values in finetuning. We perform 100K gradient steps, which is  $10\%$  of the total number of steps used to train the T5 dense models. The overall computational cost is over 20x less than the full training of dense models, since CoDA only applies heavy computation on less than half of the tokens.
For simplicity, we evaluate on classification tasks for various ablation studies of CoDA. Specifically, we report results on the MNLI, RTE and BoolQ datasets, and test three different model sizes including the Base, Large and XL size of T5. We will extend our evaluation to generation tasks such as question answering in the full result section (§6).
Can CoDA be fast and accurate? Table 2 presents the finetuning results of CoDA. As a comparison, we also report the results of Parallel Adapter, which is similar to CoDA except that it applies the expensive Transformer layers to all input tokens. This constitutes an upper-bound, and is a strong baseline that has been reported as the best among a range of adapter and prompt tuning methods [He et al., 2021]. As shown in Table 2, CoDA can achieve 3-5x computation reduction  $(r = 3, 5)$  in the Transformer layers at a cost of less than 1.0 point drop on average accuracy. As expected, our  $k$ -to-all attention variant achieves consistently better accuracy than the  $k$ -to- $k$  variant, since it can access the full attention context. On the other hand, the  $k$ -to- $k$  attention variant runs faster in practice, which can be beneficial for tasks with very long inputs. We select the  $k$ -to-all version in the final result section ( $\S 6$ ).
How many pretraining steps are needed? Figure 3 plots the finetuning accuracy by varying the number of pretraining steps for CoDA. Because CoDA can be initialized using pretrained dense models, it requires as few as 20K steps to obtain competitive finetuning results. Of course, using more pretraining steps can improve the downstream accuracy. The fact that CoDA can be quickly updated without repeating the expensive pretraining will be very beneficial in real-world applications.
Does learned routing matter? We analyze the impact of learned routing in Table 3 by comparing our soft top- $k$  router with other router implementations. We implement a variant that replaces soft top- $k$  with the sigmoid activation function, so the selection weight of each token activates on its own (without considering the capacity constraint). As shown in the table, this variant
Figure 3: Finetuning accuracy (y-axis) as a function of CoDA pretraining steps (x-axis). We show results using 0, 20K, 50K and 100K pretraining steps, and for reduction factor  $r = 3$  and  $r = 5$  respectively. CoDA requires as few as 20K steps to obtain competitive finetuning accuracy.
$^{2}$ https://github.com/google-research/text-to-text-transfer-transformer/blob/main/ released_checkpoints.md#t511
Model	Trainable Params	Reduction r	MNLI Acc.	RTE Acc.	BoolQ Acc.	SQuAD F1	ReCord F1	XSum R2	ΔAvg
Parallel Adapter	10M	-	91.5	91.0	88.5	94.8	91.4	21.9	±0.0
CoDA	10M	3	91.2	90.3	87.5	94.1	89.3	20.6	-1.0
CoDA	10M	5	90.7	89.5	87.3	93.5	87.6	20.2	-1.7
Prefix Tuning [Li and Liang, 2021]†	15M (2M)	-	(86.3)	-	-	-	-	20.5
Sequential Adapter [Houlsby et al., 2019]†	10M (2M)	-	(87.2)	-	-	-	-	20.0
Parallel Adapter [He et al., 2021]†	10M	-	-	-	-	-	-	20.7
Figure 4: Average finetuning scores of Parallel Adapter and CoDA at different model sizes.
Table 4: Comparison of CoDA and parallel adapter on 6 language tasks. We report results on the test set of XSum, and on the development set of other tasks.  $\dagger$  indicates results taken from He et al. [2021], and referenced results in bracket correspond to using 2M adapter parameters. Note that our Parallel Adapter numbers are stronger as our pretrained Transformer backbone uses more parameters than the model used in He et al. [2021].
Figure 5: The scaling of CoDA on the XSum and LibriSpeech dataset. Left: CoDA achieves better speed-quality trade-off than finetuning adapters with smaller models, on the XSum dataset. Middle: larger CoDA model achieves higher speed-ups. Right: CoDA achieves better speed-quality trade-off than the dense baseline on the LibriSpeech dataset.
achieves worse accuracy on almost all tasks and model sizes tested, getting 2.0 point worse on average. We also implement a "no-learning" baseline that simply selects the first  $k$  tokens, which is equivalent to truncating the input sequence. $^3$  This baseline performs much worse, resulting in more than 10 point decrease in accuracy for small  $k$  (and equivalently large  $r$ ). This analysis confirms the importance of learning a good routing in order to retain strong model performance.
6 Full Results
Setup In this section, we apply our best training recipe to all tasks and application domains. We first pretrain dense Transformer models, followed by the CoDA training procedure in §3.2. Our speech models are pretrained using a masked language modeling (MLM) objective similar to BERT [Devlin et al., 2019], and random quantized output label space [Chiu et al., 2022]. Our vision and text models use an encoder-decoder architecture similar to T5 but incorporate a few changes. Following PaLM [Chowdhery et al., 2022], we use multi-query attention [Shazeer, 2019] that shares the same key and value projection for multiple query heads. We only use 6 decoder layers and increase the feed forward hidden size (to compensate for the decrease in the number of layers). These modifications have a neutral effect on model quality, but speed up auto-regressive decoding significantly. We will show CoDA is compatible with these changes and can further speed up inference by a considerably large factor. We provide more details of our experimental setup in Appendix A.
NLP results In addition to the classification datasets used in Section 5, we also evaluate our final models on the SQuAD, ReCord and XSum datasets which require generating an answer or a summary
3We always include the question text for BoolQ, to achieve higher accuracy.
Model	r	Base	Large	XL
clean	other	clean	other	clean	other
w2v-BERT	-	1.8	3.6	1.5	2.9	1.5	2.9
BEST-RQ	-	1.7	3.5	1.6	2.9	1.4	2.7
P-Adapter	-	1.6	3.5	1.4	3.0	1.4	2.7
CoDA	2	1.6	3.5	1.4	3.0	1.4	2.8
CoDA	4	1.6	3.6	1.5	3.1	1.4	2.8
Table 5: Comparison of CoDA and the parallel adapter baselines on Librispeech. We report the WER results on test-clean and test-other. More results can be found in §B.2.
Model	r	OCRVQA	DocVQA	Screen2Words
EM	Speedup	ANLS	Speedup	CIDEr	Speedup
Parallel Adapter	-	67.5	1×	70.8	1×	110.2	1×
CoDA	4	68.2	4.6×	71.8	4.6×	111.6	4.6×
CoDA	8	67.6	8.0×	66.6	8.0×	108.1	8.0×
CoDA	16	66.9	13.5×	56.6	12.1×	109.0	12.5×
CoDA	32	64.4	19.4×	42.5	16.7×	104.2	17.8×
Table 6: Comparison of CoDA and the parallel adapter applied to a pretrained Pix2Struct model [Lee et al., 2022] on 3 visually-situated language understanding tasks.
Original Image
Figure 6: Visualization of routing preferences for a CoDA model applied to the OCR-VQA task. Warmer and cooler colors represent higher and lower scores respectively. Router prefers diverse coverage in early layers, but converges to selecting sparse and representative patches in later layers.
Layer 17
given the input. Table 4 contains the finetuning results of XL models. Compared to the parallel adapter baseline that uses full computation, CODA achieves  $3\mathrm{x}$  and  $5\mathrm{x}$  computation reduction with only 1.0 and 1.7 point loss in average score.
Figure 4 and 5 highlight the scaling trend of CoDA. CoDA runs much faster with slightly worse quality than the parallel adapter baseline. This is expected because the baseline processes all tokens in every layer, whereas CoDA only selects  $\frac{1}{r}$  of tokens for heavy processing. Importantly, this quality gap reduces as the model size increases (as shown in Figure 4), making CoDA a computationally efficient choice for large models. Indeed, CoDA can trade off quality for speed by varying the number of selected tokens. Figure 5 (left) demonstrates that CoDA achieves much stronger speed-quality trade-off compared to dense models without conditional computation. The black line indicates the results of Parallel Adapter when the model size grows from Small to XL, and each blue line represents the speed-quality trade-off of CoDA using  $r = 1, 3, 5$ . Moreover, Figure 5 (middle) shows that larger CoDA models exhibit higher inference speed-ups. These observations are consistent on other tasks. We provide additional results in Appendix §B.
Speech recognition results We further validate the performance of CoDA in the speech domain. Our model uses a Transformer encoder and a 2-layer LSTM Transducer [Graves, 2012]. Similar to NLP setups, we test the performance of the speech model on 3 scales - Base, Large and XL (see Appendix A for details). Table 5 demonstrates that with sizable reduction ratios  $(r = 2,4)$ , the change on word error rate (WER) is consistently minimal on the test-clean and test-other sets of LibriSpeech across different model sizes (and on other sets in §B.2). Moreover, our results are comparable to the top-performing models, such as w2v-BERT [Chung et al., 2021] and BEST-RQ [Chiu et al., 2022], that are fully finetuned by updating all parameters. Figure 5 (right) highlights again that applying conditional computation leads to better speed-quality trade-off compared to dense models.
Vision results We extend our experiments to visual tasks that involves natural language within the image, such as documents and user interfaces. Our experiments are based on Pix2Struct [Lee et al., 2022], where an image-encoder-text-decoder is pretrained by learning to predict simplified HTML from webpage screenshots. Table 6 shows the results on three tasks that were also evaluated in the
original Pix2Struct paper. In OCRVQA and Screen2Words, we observe relatively small drops in performance when reducing the number of routed tokens (i.e. patches). When the capacity is 1/16th of the original sequence length, leading to around  $13 \times$  speedup, we only lose about 1 point. We speculate that this is due to the high-level sparsity in the inputs for these two tasks. For DocVQA, where there is comparatively more textual information, we observe a steeper performance-speed trade-off but still achieve a  $8 \times$  speedup with a 4-point drop.
To provide a more intuitive understanding why CoDA works, we visualize the router behavior for the OCR-VQA model in Figure 6. We show which patches the routers prefer the most (warmest colors) and least (coolest colors), for several layers. The first, immediately obvious, observation is that router avoids low-frequency patches, i.e. patches likely to be "whitespace", since they can be adequately handled by the cheap adapter layers. The second, more subtle, observation is that the router progressively converges on a small number of key patches that we hypothesize serve as representations for larger regions. The visualization confirms that CoDA is able to select meaningful and representative patches that are useful for the prediction task.
7 Conclusion and Limitation
We present CoDA, a parameter-efficient adapter method that enables fast inference. CoDA relies on conditional computation to selectively activate model computation on important input units, providing a novel way to balance model expressivity and efficiency.
In this work, we focus on encoder-heavy applications such as summarization, speech recognition and visual question answering, by applying our method to the encoder. One limitation of CoDA is that the current routing mechanism (i.e. token selection in a given sequence) is not directly applicable to decoder-only models for auto-regressive token generation. Enabling fast token generation using conditional activation in decoder layers is an interesting direction we plan to explore in future work.
We would like to thank Rama Pasumarthi, Hongkun Yu, Kelvin Guu, Zhuyun Dai, Timothy Dozat, Raphael Hoffmann, Tao Wang, Tal Schuster, Ziwei Ji, Frederick Liu and Slav Petrov for helpful advice and discussion.
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https://www.tensorflow.org/. Software available from tensorflow.org.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. Colt5: Faster long-range transformers with conditional computation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magnusson, Hannaneh Hajishirzi, and Ludwig Schmidt. Exploring the landscape of distributional robustness for question answering models. arXiv preprint arXiv:2210.12517, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Ankur Bapna, Naveen Arivazhagan, and Orhan First. Controlling computation versus quality for neural sequence models. arXiv preprint arXiv:2002.07106, 2020.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Computational Linguistics (ACL), 2022.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698, 2020.
Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-ViT: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10809-10818, June 2022.
Yu Zhang, Daniel S Park, Wei Han, James Qin, Anmol Gulati, Joel Shor, Aren Jansen, Yuanzhong Xu, Yanping Huang, Shibo Wang, et al. Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition. IEEE Journal of Selected Topics in Signal Processing, 16(6):1519-1532, 2022.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. Mixture-of-experts with expert choice routing, 2022. URL https://arxiv.org/abs/2202.09368.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
Model implementation For our text and vision experiments, we implement our models using JAX [Bradbury et al., 2018]. Specifically, our training and model modules are built on top of the T5X, Flax and Flaxformer framework [Roberts et al., 2022, Heek et al., 2020]. Following the T5 v1.1 implementation and PaLM [Chowdhery et al., 2022], our Transformer models use the GLU variant [Shazeer, 2020] as the feed forward network and multi-query-attention [Shazeer, 2019] as the attention block. These modifications are shown to enhance modeling capacity and speed up decoding respectively.
For the speech experiments, we use TensorFlow [Abadi et al., 2015] and the Lingvo framework [Shen et al., 2019]. The state-of-the-art Transformer variant for speech recognition is the Conformer architecture [Gulati et al., 2020] which additionally uses depth-wise convolution in each layer. Since the convolution operation is applied to consecutive inputs and does not immediately support routing, we use the standard Transformer architecture [Vaswani et al., 2017] instead. Swish activation is used in the feed forward blocks, following Gulati et al. [2020]. We provide the model configuration details in Table 7.
Model	Num of params	Layers	Num of heads	\(d_{\text{model}}\)	\(d_{\text{fn}}\)	\(d_{\text{head}}\)	\(d_{\text{adpt}}\)
Text Base	0.1B	12, 6	12	768	3072	128	64
Text Large	0.5B	24, 6	16	1024	4096	128	64
Text XL	2.1B	24, 6	32	2048	8192	128	64
Speech Base	0.2B	31, 2	8	768	3072	96	256
Speech Large	0.6B	32, 2	8	1280	5120	160	256
Speech XL	1.1B	32, 2	8	1664	6656	208	256
Vision	0.7B	18, 6	24	1536	3968	128	256
Table 7: Configuration of Transformer models used in §6. We show the total number of parameters (in billions).  $d_{\mathrm{model}}$  is the model hidden dimension,  $d_{\mathrm{fn}}$  is the intermediate FFN hidden dimension,  $d_{\mathrm{head}}$  is the attention head dimension and  $d_{\mathrm{adpt}}$  is the adapter hidden dimension.
Dataset	Input length	Batch size	Steps	Optimizer	Learning rate
MNLI	128
RTE	256
BoolQ	384	128	300K	Adafactor	0.001, constant
SQuAD	512
ReCord	512
XSum	1664
LibriSpeech	3200	256	150K	Adafactor	0.001, inverse decay
OCR-VQA	4096	256
DocVQA	4096	256	20K	Adafactor	0.01, cosine decay
Screen2Words	4096	32
Table 8: Fine-tuning hyperparmaeters. We use a maximum of 300K steps for NLP tasks following T5 [Raffel et al., 2020]. Pix2struct [Lee et al., 2022] uses 10K fine-tuning steps for vision tasks. We use 20K steps as CoDA takes longer to train.
Model training We use the same data and procedure described in T5 [Raffel et al., 2020], BEST-RQ [Chiu et al., 2022] and Pix2struct [Lee et al., 2022] for pre-training the respective text, speech and vision models. We use the same training hyper-parameters, such as batch size, input sequence length, the number of pre-training steps and the choice of optimizer and learning rate scheduling. All models have been pre-trained using 128 or 256 TPUv3/TPUv4 chips.
We run CODA pre-training for text and vision models, using an additional 100K steps and 200K steps respectively. For text models, the input sequence length is  $n = 512$  and we set the number of selected tokens  $k = 192$ . For vision models, the input sequence contains  $n = 4096$  image patches and we set  $k = 1024$ . CODA pre-training is not used for our speech models because there are
Figure 7: Analyzing the speed and quality of CoDA. We select 3 representative tasks including BoolQ (classification), SQuAD (question answering) and XSum (summarization). (a) Relative decoding speed given different reduction factor  $r$ . (b) Speed-quality trade-off of CoDA applied on different tasks and model sizes. CoDA achieves better quality than the dense Parallel Adapter baseline when running at a similar inference speed. The black line shows the performance of Parallel Adapter given Small to XL model size. Each blue line represents the performance of CoDA for reduction  $r = 1, 3, 5$  for a given model size. When  $r = 1$ , CoDA is equivalent to the dense baseline.
sufficient fine-tuning data. Following standard practice in speech, we use the 1K hour data from the LibriSpeech dataset [Panayotov et al., 2015] and another 30K hour data generated using the noisy student self-training method [Xie et al., 2020, Zhang et al., 2022].
Table 8 lists the hyper-parameters used for fine-tuning, including the sequence length, learning rate, batch size and the number of fine-tuning steps used. For NLP datasets, we set the maximum input length and decoding length to the 98th percentile of lengths in the training set. For vision datasets, we set the input length following the suggested values in Pix2struct. We also find that annealing the number of routed tokens  $k$  can achieve better finetuning results. Specifically, we decrease  $k$  linearly from the sequence length  $n$  down to the target value  $n / r$  using the first  ${10}\%$  to  ${20}\%$  of the finetuning steps.
B.1 NLP
Table 9 contains the complete fine-tuning results on the 6 language datasets. As discussed in §6, the gap between CoDA and its counterpart without conditional computation is large at Base size. As the model size increases, CoDA retains almost the same level of quality given 3x computation reduction  $(r = 3)$ . The reduction leads to decoding speed-ups, as shown in Figure 7. More importantly, we see that larger model benefits more from CoDA, achieving a speed-up factor close to the reduction factor
Model	Trainable Params	Reduction r	MNLI Acc.	RTE Acc.	BoolQ Acc.	SQuAD F1	ReCord F1	XSum R2
Base
Parallel Adapter	-	88.2	75.8	80.1	91.2	76.7	14.1
CoDA	2M	3	85.8	68.6	78.7	89.1	69.1	13.5
CoDA	5	82.8	60.3	78.0	87.3	61.8	13.1
Large
Parallel Adapter	-	90.5	90.6	86.7	93.9	87.2	18.3
CoDA	5M	3	90.0	84.8	84.3	93.1	84.6	17.0
CoDA	5	89.4	88.5	83.9	92.2	81.2	16.7
XL
Parallel Adapter	-	91.5	91.0	88.5	94.8	91.4	22.3
CoDA	10M	3	91.2	90.3	87.5	94.1	89.3	21.2
CoDA	5	90.7	89.5	87.3	93.5	87.6	20.7
Table 9: Fine-tuning results on 6 language tasks  $\times$  3 model sizes. We report the best results on the development set.
Model	r	Base	Large	XL	Δ Avg
dev	test	dev	test	dev	test
clean	other	clean	other	clean	other	clean	other	clean	other	clean	other
w2v-BERT	-	1.7	3.6	1.8	3.6	1.5	2.9	1.5	2.9	1.5	2.6	1.5	2.9	-
BEST-RQ	-	1.6	3.5	1.7	3.5	1.5	2.8	1.6	2.9	1.4	2.7	1.4	2.7	-
Parallel Adapter	-	1.5	3.5	1.6	3.5	1.4	3.0	1.4	3.0	1.4	2.7	1.4	2.7	±0.0
CoDA	2	1.5	3.5	1.6	3.5	1.4	3.0	1.4	3.0	1.4	2.7	1.4	2.8	+0.01
CoDA	4	1.6	3.6	1.6	3.6	1.4	3.1	1.5	3.1	1.4	2.8	1.4	2.8	+0.07
Table 10: Comparison of CoDA and the parallel adapter baselines on all 4 splits (dev-clean, dev-other, test-clean, test-other) of Librispeech.
$r$ . These results highlight the potential of CoDA for large-scale models, which we plan to investigate in future work.
B.2 Speech