SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given the training datasets  $\mathcal{D} = \{\mathcal{X}_i, y_{\mathcal{X}_i}\}_{i=1}^{N_{\mathcal{D}}}$ , our method aims to learn  $M$  visual and  $N$  textual prompts simultaneously. All parameters in ALIGN are optimized by minimizing the cross-entropy loss end-to-end. We summarize the training algorithm at Algorithm 1.
Table B. 2: Hyperparameter setting used in the previous experiments.
Hyperparameters	Values
Batch Size	4
Input Size	224 × 224
Input Interpolation	"Bicubic"
Input Pixel Mean	[0.48145466, 0.4578275, 0.40821073]
Input Pixel STD	[0.26862954, 0.26130258, 0.27577711]
Transforms	["random resized crop", "random filp", "normalize"]
Optimizer	SGD
Learning Rate	0.0035
LR Scheduler	"cosine"
Warmup Epoch	1
Warmup Type	"constant"
Warmup LR	1e-5
Backbone	ViT-B/16
Number of Textual Prompts	4
Number of Visual Prompts	4
Learnable Prompt Length	2
Fixed Prompt Length	2
weight of token-level cost	1
weight of regularization in OT	0.1
Prompt Initialization	"a photo of a"
Precision	"fp16"
Algorithm 1 Training algorithm of ALIGN.
Input: Training dataset  $\mathcal{D}$ , a pre-trained vision-language model, class name set, number of visual prompts  $M$ , number of textual prompts  $N$ , and the training epoch.
Output: The learned ALIGN, which discovers multi-modal multi-mode prompts for downstream tasks.
Initialize: The  $M$  and  $N$  multi-modal prompt embeddings.
Preprocess: Built  $N \times K$  textual token inputs according to Sec 2.1 in the manuscript.
for iter  $= 1,2,3,\dots$  do
1. Feed the textual input into the text encoder  $g$  and collect the outputs with the corresponding prompt-level representation  $\{h_k^n\}_{k=1,n=1}^{K,N}$  and token embeddings  $\{s_k^n\}_{k=1,n=1}^{K,N}$ , where each  $s_k^n$  is the output token embeddings of  $n$ -th prompt of  $k$ -th label with length  $b + k_l$ .
2. Sample a batch of  $J$  images. Built  $N\times B$  visual patch inputs according to Sec2.1 in the manuscript. Feed the visual input into the visual encoder  $f$  and collect the outputs with the corresponding prompt-level representation  $\{z_j^m\}_{j = 1,m = 1}^{J,M}$  and patch embeddings  $\{\pmb {r}_j^m\}_{j = 1,m = 1}^{J,M}$ , where each  $\pmb {r}_j^m$  denotes the output patch embeddings of  $m$ -th prompt of  $j$ -th image with length  $b + O$ .
Two-level OT
3. Calculate the token-level OT distance between each image and each label in Eq.5 with the collected patch set and token set.
4. Calculate the cost matrix in prompt-level OT according to Eq.6, and then get the prompt-level OT distance in Eq.4.
Compute the cross-entropy loss  $L$  with the obtained prompt-level OT distance according to Eq.8 and update all learnable parameters by minimizing  $L$  with the stochastic gradient descent algorithm.
We in this section report additional results of other datasets on the few-shot task and conduct the ablation studies on the prompt and token-level OT.
Harmonic mean (H) with various  $\beta$
Figure D1: Harmonic mean (H) results of ALIGN on Base-to-New task under different  $\beta$ .
D.1 Few-shot Results
We report the numerical results of various methods on 11 datasets at Table. D. 1.. From the results, we find that our method ALIGN outperforms baselines in most cases, which demonstrates the efficiency of the token-level prompt alignment module.
D.2 Ablation studies
Recall that the proposed model consists of the prompt-level and token-level OT, which align the textual and visual modalities from hierarchical semantics. In the previous experiment, we view the prompt-level and token-level OT equally and set the hyperparameter weight  $\beta = 1$  in Eq.6 in the manuscript. Here want to analyze how those two OTs affect the model performance. To this end, we rewrite the cost matrix in Eq.6 in the manuscript as:
$$
C _ {m n} = (1 - \beta) \left(1 - \operatorname {s i m} \left(\boldsymbol {z} ^ {m}, \boldsymbol {h} ^ {n}\right)\right) + \beta d _ {\mathrm {O T}} ^ {\lambda} \left(\boldsymbol {x} _ {m}, \boldsymbol {y} _ {n}; \hat {\mathbf {C}} ^ {m n}\right). \tag {9}
$$
Note that  $\beta = 0$  and  $\beta = 1$  denote two of our variants, where the former denotes only prompt-OT works and the latter means we only focus on token-level similarity. We report the ablation results of ALIGN on Base-to-New tasks under various settings, e.g.,  $\beta = [0, 0.2, 0.5, 0.7, 1.0]$  at Fig. D1. We have the following interesting findings: 1) The combined ALIGN works better than each of them; 2) After finetuning  $\beta$  for each dataset, one can obtain better results than the reported values in our paper.
Table D. 1.: The few-shot results of various methods on 11 datasets. We report mean value over 3 different seeds. The best results are highlighted.
Dataset	Methods	1 shot	2 shots	4 shots	8 shots	16 shots
Caltech101	CoOp	92.4	93.2	93.5	94.0	94.8
PLOT	88.40	89.95	91.50	93.00	93.24
UPT	93.66	94.17	94.09	95.04	95.95
MaPLe	91.73	93.33	94.23	94.43	95.26
ALIGN	93.97	94.13	95.00	95.43	96.00
DTD	CoOp	48.4	51.5	59.2	64.4	69.5
PLOT	51.90	55.95	58.24	65.50	70.52
UPT	45.01	52.97	60.74	65.44	70.62
MaPLe	51.16	54.70	61.63	65.63	70.60
ALIGN	54.07	56.53	63.3	67.6	71.40
EuroSAT	CoOp	51.8	60.9	69.0	76.0	84.1
PLOT	60.10	68.45	72.97	79.84	83.12
UPT	66.46	69.07	75.36	85.62	90.77
MaPLe	66.67	79.26	84.25	89.96	92.14
ALIGN	53.23	71.43	80.93	85.97	90.77
FGVCAircraft	CoOp	24.2	25.8	27.9	32.7	34.2
PLOT	21.50	21.71	23.96	27.02	30.24
UPT	28.43	29.91	33.34	39.50	46.61
MaPLe	26.64	27.86	33.56	40.66	49.93
ALIGN	29.57	31.63	34.03	40.95	49.99
Flowers102	CoOp	72.9	80.4	85.7	92.3	96.2
PLOT	70.00	81.34	88.29	92.84	95.10
UPT	74.97	81.81	91.90	95.17	97.41
MaPLe	80.24	88.14	90.07	95.10	96.34
ALIGN	81.33	88.77	92.53	95.43	96.57
FOOD101	CoOp	81.6	80.9	81.5	82.4	84.9
PLOT	69.10	72.89	74.89	76.70	77.87
UPT	84.21	85.01	85.34	86.16	86.83
MaPLe	78.73	77.30	79.03	80.10	82.43
ALIGN	85.29	86.05	86.66	86.74	86.90
ImageNet	CoOp	68.07	69.26	69.60	70.35	71.53
PLOT	67.51	68.80	70.00	70.21	71.40
UPT	69.55	69.88	70.28	71.58	72.64
MaPLe	69.56	69.94	70.65	71.80	72.74
ALIGN	69.80	70.02	70.84	71.77	72.45
OxfordPets	CoOp	90.0	89.8	92.3	92.0	92.1
PLOT	83.21	85.77	86.02	89.13	89.95
UPT	82.93	85.40	85.97	87.40	88.10
MaPLe	89.80	86.76	90.76	90.23	91.30
ALIGN	91.36	91.93	93.4	93.67	94.17
StanfordCars	CoOp	66.4	69.2	70.1	72.8	75.2
PLOT	46.20	51.67	54.35	60.52	65.32
UPT	67.60	69.57	75.88	80.19	84.17
MaPLe	65.96	69.10	75.73	79.76	85.36
ALIGN	68.27	72.84	76.58	81.65	86.75
SUN397	CoOp	65.2	66.6	68.1	70.5	73.2
PLOT	55.33	60.02	63.21	66.02	67.98
UPT	68.84	69.76	72.12	74.00	75.90
MaPLe	61.73	63.23	67.60	69.13	73.00
ALIGN	69.14	69.98	71.88	74.15	76.57
UCF101	CoOp	70.7	73.8	76.6	79.6	80.4
PLOT	51.42	54.89	61.23	67.45	70.85
UPT	71.98	74.93	77.49	80.91	83.86
MaPLe	73.23	73.00	77.45	81.2	84.67
ALIGN	74.42	75.87	80.18	81.99	85.69