SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our main results are shown in Table 1. We follow [7, 13] in dividing the 21 languages of the test set into three groups: High-resource (High), Medium-resource (Med), and Low-resource (Low). We report the average BLEU score for each group as well as the average for all 21 languages (Avg). ComSL Medium and Large outperform Whisper Medium and Large, achieving BLEU scores of 29.7 and 30.1, respectively. The main factors contributing to this improvement are the replacement of the Whisper decoder with mBART and the application of our training strategies and losses in training the combined Whisper and mBART models. We observe that the performance of ComSL Medium is on par with that of Whisper Large, which has significantly more parameters. It is noteworthy that the BLEU scores for Whisper Medium and Large surpass those reported in the original paper [26], which shows zero-shot performance on the CoVoST testing set. The scores for Whisper models, listed in Table 1, result from fine-tuning Whisper with CoVoST 2 training data. Our BLEU score for the fine-tuned Whisper is higher than that shown in [44], which may be due to differences in the versions used for the experiments. When pseudo ST data is employed, both ComSL Medium and ComSL Large outperform the current SOTA, i.e., USM [44]. Our ComSL Large reaches a BLEU score of 31.5, which is 0.8 higher than USM with 0.7B fewer parameters. According to the data description in [44], the Common Voice corpus, where we extra data and generate pseudo translations, has also been included in USM training through self-supervised learning. Furthermore, our ComSL Medium and Large single models outperform Cascaded models that first conduct ASR with Whisper and then run MT with mBART-50 using ASR hypotheses as inputs. However, there is still a performance degradation due to ASR errors when compared to using ground-truth transcription as inputs for MT.
5.1 Ablation study on training tasks/losses and strategies
We conduct an ablation study on training tasks/losses and strategies using ComSL Medium. The results, shown in Table 2, demonstrate how our strategies and tasks steadily improve overall perfor
<sup>4</sup>https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt
5The breakdown of BLEU scores in the different configurations on each of the 21 language pairs is listed in the Appendix.
Model	PARAMS	High	Med	Low	Avg
mSLAM [7]	2 B	37.8	29.6	18.5	24.8
Maestro [11]	0.6 B	38.2	31.3	18.4	25.2
Mu2SLAM [13]	0.6 B	37.0	30.6	23.5	27.1
Whisper Medium (Baseline Medium)	0.8 B	36.4	32.8	24.3	28.6
Whisper Large (Baseline Large)	1.6 B	37.0	33.8	25.6	29.7
USM [44]	2 B	-	-	-	30.7
Our model trained without pseudo ST data
ComSL Medium	0.9 B	37.4†	34.1†	25.3†	29.7
ComSL Large	1.3 B	37.7‡	34.5‡	25.8	30.1
Our model trained with pseudo ST data
ComSL Medium	0.9 B	37.3†	35.6†	26.6†	30.8
ComSL Large	1.3 B	37.6‡	35.9‡	27.5‡	31.5
Non-E2EST
Ground-truth Transcription + mBART-50	0.6 B	42.1	39.9	35.8	38.0
Whisper Medium + mBART-50	1.3 B	37.3	34.5	26.4	30.4
Whisper Large + mBART-50	2.2 B	37.4	34.6	26.4	30.5
Table 1: Speech translation performance in terms of the average BLEU scores on non-English to English (xx-en) directions of the CoVoST 2 testing set. All models including Whisper and mBART-50 are fine-tuned with CoVoST 2 training set. The symbols "†" and "‡" are used to indicate that the model performance is significantly better  $(p < 0.01)$  than the Baseline Medium and Large, respectively.
Description	ST BLEU	ST Delta	MT BLEU
w/ ST task only	26.48	-	29.13
+ MT task	26.31	-0.17	36.43
+ DDM	27.87	1.56	36.21
+ ASR task	28.29	0.42	35.92
+ Freeze encoder	28.64	0.35	35.28
+ MT regularization	29.40	0.76	37.40
+ CML Loss	29.69	0.29	37.30
+ Pseudo data	30.77	1.08	37.46
Table 2: Ablation study on the training strategies and tasks/losses. We start training the model with ST loss only and then adding a strategy or a task/loss over the previous experiment. We present the BLEU score of ST task, as well as the performance gain of ST task brought by each change.
mance. The model trained with only ST loss achieves a BLEU score of 26.48. Adding the MT task results in a slight drop to 26.31, indicating that directly applying the MT task has little impact on performance. However, when DDM is leveraged, the BLEU score increases significantly to 27.87. The ASR task and corresponding training strategy of freezing speech encoder parameters in the early stage of training improve the BLEU score by 0.42 and 0.35, respectively. MT regularization, which prevents overfitting in the MT task, enhances performance by an additional 0.76 BLEU score. CML, which matches speech and text representations, boosts performance to BLEU score of 29.69. Finally, using pseudo ST data further increases the BLEU score to 30.77. To further investigate the performance of the integrated language model in ComSL, we feed ground-truth speech transcription to it, even though ComSL only requires speech as input for the ST task. The corresponding BLEU score of the MT task shown in Table 2 indicates that our training strategies and tasks continue to reduce the degradation of MT performance from mBART-50, finally achieving a BLEU score of 37.46 compared to 38.0 for Ground-truth Transcription + mBART-50 shown in Table 1.
5.2 Comparison among different cross-modality learning (CML) methods
We compare our cross-modality learning methods with previous works. The model trained with all strategies and tasks except the methods of minimizing modality gap refers to the baseline. The performance of the baseline and those with our CML method and previous MML, ConST, and WACO methods are listed in Table 3. Our CML reaches 29.69 BLEU score, the best one among all listed methods in the table. MML also performs well, achieving a BLEU score of 29.6, slightly lower than ours, but it requires internal alignment at the token level. Calculation of alignment by
Method	Forced-Alignment	Avg BLEU
Baseline	N	29.40
MML [11]	Token level	29.60
ConST [43]	N	29.18
WACO [24]	Word level	29.37
Our CML	N	29.69
Table 3: Comparison between different methods for minimizing modality gap on ComSL Medium. The second column shows what kind of forced-alignment is needed for the method. N denotes 'Not Needed'.
whatever internal or external token-level forced aligner, adds a burden to the training process. In our implementation, contrastive-based methods (ConST and WACO) fail to surpass the baseline model. We conjecture that it may be due to the nature of contrastive loss, which pulls negative and positive samples apart, which may cause an undesirable gradient direction for a supervised pretrained model like Whipser.
We also take an in-depth look at the hidden level to see what happens after applying cross-modality learning tasks. Figure 3 shows the similarity matrices of speech and text representations. These representations are taken from the output of the 4th layer in the mBART encoder. A higher number, indicated by a darker color in the figure, means that the representations are more similar to each other. Here are our findings, 1) Our model can match text and speech representations fairly well even without explicitly applying any modality gap minimizing methods, as shown in Figure 3-a). We believe this is most likely due to the use of knowledge from ASR task, which leaves less room for the CML method to further improve performance. 2) The model trained with our CML can produce a unified representation not only for speech and text but also for non-speech tokens like silence. As illustrated by the green box in the figure, non-speech tokens like punctuation (period) and silence are aligned much better than those without CML. This is a unique phenomenon observed in our approach. (We tried plotting matrices using other methods listed in Table 3, but all looked like the left figure.) 3) Our CML can help encode speech information more precisely. For example, in the blue box area, there is a compound two-syllable word with a very short pause in between. Our model better identifies the short pause and sharpens the distributions of surrounding tokens.
(a) w/o Cross-modality learning
(b) w/ Cross-modality learning
Figure 3: Similarity matrices of speech hidden states and text hidden states(softmax alone the text axis). These hidden states are taken from the output of the 4-th text encoder layer.
6 Conclusions, Limitations and Future Work
In this paper, we presented a Composite Speech and Language (ComSL) model to perform E2E speech-to-text translation tasks. We bridged the gap between speech and text representations through cross-modality learning tasks, in addition to other auxiliary tasks like speech recognition and machine translation. Our composite model outperformed the constituent speech model (Whisper) or cascaded speech and language models (Whisper+mBART) in a pipeline manner. By adding pseudo ST data, our model achieved new SOTA performance on the public CoVoST2 evaluation set. However, our approaches have not been exploited on decoder-only based Large Language Models (LLMs) in this study due to their recent public availability and the requirement of large GPU resources. Additionally,
our model has only been evaluated on speech-to-text translation tasks due to the limitations of currently available E2E datasets. In the future, we plan to combine our approaches with Low-Rank Adaptation (LoRA) [17] or other light-weight adapters and use efficient data augmentation techniques to fine-tune a large composite model for various downstream spoken language tasks.
[38] Chengyi Wang, Yu Wu, Shujie Liu, Ming Zhou, and Zhenglu Yang. Curriculum pre-training for end-to-end speech translation. arXiv preprint arXiv:2004.10093, 2020.
[39] Ron J Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. Sequence-to-sequence models can directly translate foreign speech. arXiv preprint arXiv:1703.08581, 2017.
[40] Chen Xu, Bojie Hu, Yanyang Li, Yuhao Zhang, Qi Ju, Tong Xiao, Jingbo Zhu, et al. Stacked acoustic- and-textual encoding: Integrating the pre-trained models into speech translation encoders. arXiv preprint arXiv:2105.05752, 2021.
[41] Jian Xue, Peidong Wang, Jinyu Li, and Eric Sun. A weakly-supervised streaming multilingual speech model with truly zero-shot capability. arXiv preprint arXiv:2211.02499, 2022.
[42] Rong Ye, Mingxuan Wang, and Lei Li. End-to-end speech translation via cross-modal progressive training. arXiv preprint arXiv:2104.10380, 2021.
[43] Rong Ye, Mingxuan Wang, and Lei Li. Cross-modal contrastive learning for speech translation. arXiv preprint arXiv:2205.02444, 2022.
[44] Yu Zhang, Wei Han, James Qin, Yongqiang Wang, Ankur Bapna, Zhehuai Chen, Nanxin Chen, Bo Li, Vera Axelrod, Gary Wang, et al. Google usm: Scaling automatic speech recognition beyond 100 languages. arXiv preprint arXiv:2303.01037, 2023.
[45] Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, and Furu Wei. Speechut: Bridging speech and text with hidden-unit for encoder-decoder based speech-text pre-training. arXiv preprint arXiv:2210.03730, 2022.
[46] Renjie Zheng, Junkun Chen, Mingbo Ma, and Liang Huang. Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation. In International Conference on Machine Learning, pages 12736-12746. PMLR, 2021.