SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We train each model for a total of 20 epochs, where one epoch consumes the whole training data equal to 1,013,630 samples. We use a batch size of 20 with the 12,500 warmup steps and train on 1 A100 GPU. We use a learning rate of 0.00003, AdamW optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , a weight decay of 0.0001, and gradient clipping norm of 10.
A.2.2 Effects of using pre-trained model weights, locking location, and batch size
Each reported number in this section is Top-1 accuracy  $(\%)$  on SLURP data for speech intent classification.
Pre-trained speech encoder vs randomly initialized: In Table 7, we compared starting multi-modal training from scratch and from pre-trained weights. The performance is significantly boosted by initializing the speech encoder using weights from the encoder part of the Whisper ASR model [48]. However, regardless of using random weights and pre-trained model weights, training with CWCL results in a much better downstream performance.
Table 7: Comparison between using randomly initialized weights and pre-trained weights for speech encoders during training: Top-1 accuracy (%) on SLURP data
Method	Random initialization	Pre-trained weights
CL	13.80	22.73
CWCL	26.17	53.12
Locking location: We have 4 ways to lock our model during multi-modal training since we have pre-trained speech and text models. We compared all the locking options and the result is shown in Table 8. In both baseline and CWCL losses, locking the text model works best. This can be seen as transferring the knowledge of semantic relationships in text models to speech models.
Table 8: Locking location vs. performance: Top-1 accuracy (%) on SLURP data
Locking location	none	speech	text	both
CL	18.77	7.89	24.03	9.82
CWCL	17.50	27.39	53.12	16.70
Batch size vs. performance: Since the large batch size was shown to improve performance with contrastive loss in computer vision, we also did a similar experiment to see how the batch size affects the performance as it gets larger. As the batch size increases, we also increased the learning rate proportionally, e.g., if  $\mathrm{bs} = 20$  has  $\mathrm{lr} = 1$ ,  $\mathrm{bs} = 40$  has  $\mathrm{lr} = 2$ . The results are in Table 9.
Table 9: Effective batch size vs. performance: Top-1 accuracy (%) on SLURP data
batch size	20	40	80
CL	24.03	25.20	24.51
CWCL	53.12	53.94	51.80
A.2.3 Further evidence of modality alignment due to CWCL
In Figure 2, we showed the alignment (measured as inner product) between speech features and text features obtained from models trained using just CL and those trained using CWCL. We use the speech and text data from the SLURP test dataset. We illustrated that speech and text embeddings that belong to the same intent class were much more aligned compared to speech and text from mismatched classes. In this section, we provide more examples that support this observation.
In Figures 6, 7, we show the alignment between the speech and text embeddings where the speech and text samples belong to classes other than those used in Figure 2. We again see that the alignment between samples in the same class is much higher than that between samples in different classes. In general, we observe the same pattern to hold across all the classes in the dataset, thus confirming that our results are not due to sampling bias.
Figure 6: Cosine-similarity between speech and text embeddings obtained by sampling 6 classes randomly from the SLURP test dataset. In this case, the sampled classes are different from those used in Figures 2 and 7.
A.2.4 Additional tables for reference
Table 10 additionally shows the Top-5 accuracy over speech-text experiments. Since most of the previous works did not report this metric, we only include our own experimental results. Table 11 shows the existing supervised model performances where the models are either trained or fine-tuned on the labeled Google Speech Command Dataset V2 for performing the keyword spotting (KWS) task, while our methods did not require any labeled KWS data for performing the task.
A.2.5 General template as a python list
To test the speech-text alignment models, we use a "general" set of templates in addition to the one obtained by using the text from the training data itself. This general set of templates aims to mimic a scenario where the no example texts maybe available. We list the set of template sentences used in the general set here.
Figure 7: Another example of alignment between speech and text embeddings. The sampled classes are different from those used in Figures 2 and 6.
Table 10: Top-5 accuracy for zero-shot speech-to-intent classification (SLURP and STOP) and KWS on Google Speech Command Dataset V2. Superscript  $\#$  is used to indicate use of general templates.
Method	Text model	SLURP	SLURP#	STOP	STOP#	KWS	KWS#
CL	RoBERTa+S	69.57	49.86	98.19	94.03	82.22	82.53
CL	BART+Y	52.97	24.87	95.27	81.63	84.02	78.14
CWCL (Ours)	RoBERTa+S	84.53	68.58	99.38	96.52	91.20	92.42
CWCL (Ours)	BART+Y	79.48	57.34	99.48	97.71	93.79	94.30
Text-intent (upper bound)	RoBERTa+S	95.66	83.36	98.93	95.20	100	98.20
BART+Y	99.58	73.82	99.45	98.40	100	100
General template sentences: [ it is about {}, it was about {}, it will be about {}, this is about {}, this was about {}, this will be about {}, it is related to {}, it was related to {}, it will be related to {}, this is related to {}, this was related to {}, this will be related to {}, it is talking about {}, it was talking about {}, it will be talking about {}, this is talking about {} this was talking about {}, this will be talking about {}, I am talking about {}, I was talking about {}, I will be talking about {}, You are talking about {}, You were talking about {}, You will be talking about {}, They are talking about {}, They were talking about {}, They will be talking about {}, We are talking about {}, We were talking about {}, We will be talking about {}, it talks about {}, it talked about {}, it will talk about {}, this talks about {}, this talked about {}, this will talk about {}, I talk about {}, I talked about {}, I will talk about {}, You talk about {}, You talked about {}, You will talk about {}, They talk about {}, They will talk about {}, We talk about {}, We will talk about {},
Table 11: Keyword spotting Top-1 accuracies on GSCV2 from existing supervised models.
Method	KWS
Attention RNN [53]	93.9
KWT-2 [46]	97.74
Wav2Vec2 [54]	96.6
M2D [55]	95.4
M2D - Fine tuned [55]	98.5