SlowGuess/NAACL_2022_OCR huggingface dataset:
We train several SLM models with varying sizes in our experiments and choose the best model based on the top-1 accuracy. As we increase the number of layers in the LM model from 2 to 4 to 6 layers, the SLM and the proposed MMA with future information models have shown performance improvements. However, increasing the number of layers greater than 6 does not yield any performance improvements. We also notice this degradation of performance with the XLM model while varying the number of hidden layers in the LM head.
We follow the training process similar to Ma et al. (2020) training process. We train an English ASR model using the source speech data. Next, we train a simultaneous model without the latency loss (setting  $\lambda_{\text{latency}} = 0$ ) after initializing the encoder from the English ASR model. After this step, we finetune the simultaneous model for different  $\lambda$ s. This training process is repeated for all the reported
models and for each task. The details regarding the hyperparameters for the model have been provided in Table 2.
F BLEU-AL Numbers
As mentioned in the results section of the main paper, we vary the latency weight hyperparameter  $(\lambda)$  to train different models to obtain different latency regimes. We also vary the step-size/speech segment size during inference. In total, we obtain 18 different data points corresponding to each model. In Table 3, we compare the results obtained using MMA, MMA-XLM and MMA-SLM under similar hyperparameter settings. It will help the reader to quantify the benefits obtained from our proposed approach.
Task	# Hours	# Sentences	# Talks	# Words
Train	Dev	Test	Source	Target
English-German	408	225k	1,423	2,641	2,093	4.3M	4M
English-French	492	269k	1,412	2,632	2,510	5.2M	5.4M
Table 1: Dataset Statistics(# - Number of)
Hyperparameter	MMA	MMA-XLM/CLM
encoder layers	12	12
encoder embed dim	292	256
encoder ffn embed dim	2048	2048
encoder attention heads	4	4
decoder layers	6	6
decoder embed dim	292	256
decoder ffn embed dim	2048	2048
monotonic ffn embed dim	-	2048
decoder attention heads	4	4
dropout	0.1	0.1
optimizer	adam	adam
adam-β	(0.9, 0.999)	(0.9, 0.999)
clip-norm	10.0	10.0
lr scheduler	inverse sqrt	inverse sqrt
learning rate	0.0001	0.0001
warmup-updates	4000	4000
label-smoothing	0.0	0.0
max tokens	40000	40000
conv layers	2	2
conv stride	(2,2)	(2,2)
#params	≈ 39M	≈ 39M
Table 2: Model Hyperparameters
λ	Model	step size (AL(msec) / BLEU)
120	200	280	360	440	520
English-German Task
0.1	MMA	378 / 3.54	887 / 7.17	1317 / 9.72	1671 / 11.54	1935 / 12.95	2376 / 13.98
MMA-XLM	348 / 5.03	848 / 8.77	1269 / 10.4	1631 / 12.78	1961 / 14.22	2272 / 15.34
MMA-SLM	748 / 8.83	1192 / 10.43	1566 / 12.43	1857 / 13.82	2156 / 14.29	2421 / 15.44
0.05	MMA	775 / 6.5	1220 / 10.08	1683 / 11.72	1891 / 12.92	2484 / 13.85	2441 / 14.51
MMA-XLM	766 / 7.76	1200 / 10.84	1654 / 13.2	1873 / 13.72	2655 / 16.36	2456 / 15.90
MMA-SLM	1250 / 12.12	1588 / 13.53	1899 / 14.68	2171 / 15.18	2424 / 15.72	2665 / 15.99
0.01	MMA	1841 / 13.33	2183 / 14.24	2455 / 14.58	2683 / 15.11	2839 / 16.05	3079 / 16.18
MMA-XLM	1846 / 14.83	2167 / 15.57	2439 / 16.13	2662 / 16.21	2855 / 16.87	3085 / 17.24
MMA-SLM	2047 / 14.43	2039 / 15.33	2420 / 16.87	2503 / 16.87	2786 / 17.01	2871 / 17.35
English-French Task
0.1	MMA	471 / 6.77	924 / 13.2	1299 / 17.37	1667 / 19.78	1975 / 20.77	2269 / 22.28
MMA-XLM	478 / 7.54	902 / 14.48	1284 / 17.74	1607 / 21.02	1972 / 22.17	2251 / 22.79
MMA-SLM	460 / 7.89	701 / 14.86	1313 / 18.40	1604 / 21.23	1971 / 22.50	2211 / 22.91
0.05	MMA	806 / 12.9	1209 / 18.13	1533 / 20.57	1825 / 22.30	2137 / 22.95	2390 / 23.98
MMA-XLM	796 / 14.06	1184 / 19.19	1512 / 21.25	1807 / 23.17	2117 / 23.75	2363 / 24.78
MMA-SLM	794 / 14.20	1197 / 19.79	1504 / 21.44	1878 / 23.5	2122 / 24.27	2341 / 24.96
0.01	MMA	1728 / 22.84	1725 / 23.6	2204 / 25.07	2416 / 25.44	2632 / 25.56	2824 / 25.81
MMA-XLM	1713 / 23.11	1701 / 24.89	2116 / 26.24	2420 / 26.19	2631 / 26.07	2796 / 26.17
MMA-SLM	1725 / 23.33	1704 / 25.16	2217 / 26.54	2412 / 26.63	2631 / 26.57	2812 / 26.55
Table 3: BLEU vs Average Lagging results for MMA, MMA-XLM and MMA-SLM models on English-German and English-French tasks. The models are trained using different latency loss weights  $(\lambda = 0.1,0.05,0.01)$