SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Alg. 1 and Alg. 2, we present the training and meta-testing procedures of our Meta-CETM.
E Derivation of Formulas
In this section, we provide the detailed derivation process of variational evidence lower bound (ELBO) in Eq. 8 and the expectation maximization solver process for multivariate Gaussian distribution in Eq. 7 in Sec. 2.2.
Algorithm 1: Training process
Input: A set of training corpora  $\{\mathcal{D}_c\}_{c = 1}^C$ ; initialized model parameters  $\Psi$  Randomly sample tasks from each training corpus  $\mathcal{D}_c$  to obtain  $\{\mathcal{T}^{(i)}\}_{i = 1}^{M}$ ; for each task  $\mathcal{T}^{(i)}, i = 1,2,\dots ,M$  do
Build semantic graph  $\mathbf{A}^{(i)}$  with established dependency parsing tools;  
Infer adaptive word embeddings  $\mathbf{Z}^{(i)}$  according to Eq. 6;  
Initialize parameters of the Gaussian mixture prior:  $\pi_{k},\pmb{\mu}_{k}$  and  $\pmb{\Sigma}_{k}$ ;  
Update to the optimal value  $\pi_k^{(i)},\pmb {\mu}_k^{(i)}$  and  $\pmb{\Sigma}_{k}^{(i)}$  using EM based on Eq. 7;  
Compute the topic-word matrix  $\beta^{(i)}$  according to Eq. 3;  
Infer the latent context varibale  $c^{(i)}$  using Eq. 5;  
for each document  $\mathbf{x}_j^{(i)},j = 1,2,\dots ,J$  do  
    Infer topic proportion  $\theta_j^{(i)}$  with Eq. 4;  
    Calculate the log-likelihood  $p(\mathbf{x}_j^{(i)}|\pmb {\theta}_j^{(i)},\pmb {\beta}^{(i)})$   
Derive the ELBO as Eq. 8 and update  $\Psi$  using SGD;
Algorithm 2: Meta-test for a new task
Input: A new corpus  $\mathcal{D}_{test}$ , trained model parameters  $\Psi$   
Output: Adaptive topic-word matrix  $\beta$   
Randomly sample a task  $\mathcal{T}_{new}$  from the given corpus  $\mathcal{D}_{test}$ ;  
Get the corresponding BoWs  $\mathbf{X}_{new}$  and dependency graph  $\mathbf{A}_{new}$  for the current task;  
Infer the adaptive word embeddings  $\mathbf{Z}_{new}$  with part of the trained model parameters  $\Psi$ ;  
Initialize parameters of the Gaussian mixture prior:  $\pi_k, \mu_k$  and  $\Sigma_k$ ;  
Compute optimal  $\pi_k^*, \mu_k^*$  and  $\Sigma_k^*$  using EM based on Eq. 7;  
Derive the adaptive topic-word matrix  $\beta_{new}$  by Eq. 3;
E.1 Variational ELBO
$$
\begin{array}{l} \log p (X ^ {(i)}, A ^ {(i)}) = \log \iiint p (X ^ {(i)}, A ^ {(i)}, \Theta^ {(i)}, c ^ {(i)}, Z ^ {(i)}) d \Theta^ {(i)} d c ^ {(i)} d Z ^ {(i)} \\ = \log \iiint p (X ^ {(i)} \mid \Theta^ {(i)}, Z ^ {(i)}) p (\Theta^ {(i)} \mid c ^ {(i)}) p (c ^ {(i)}) p (A ^ {(i)} \mid Z ^ {(i)}) p (Z ^ {(i)}) d \Theta^ {(i)} d c ^ {(i)} d Z ^ {(i)} \\ = \log \mathbb {E} _ {Q} \left[ \frac {p \left(X ^ {(i)} \mid \Theta^ {(i)} , Z ^ {(i)}\right) p \left(\Theta^ {(i)} \mid c ^ {(i)}\right) p \left(c ^ {(i)}\right) p \left(A ^ {(i)} \mid Z ^ {(i)}\right) p \left(Z ^ {(i)}\right)}{q \left(\Theta^ {(i)} \mid X ^ {(i)} , c ^ {(i)}\right) q \left(c ^ {(i)} \mid X ^ {(i)}\right) q \left(Z ^ {(i)} \mid A ^ {(i)} , E ^ {(i)}\right)} \right] \\ \geq \mathbb {E} _ {Q} \left[ \log \frac {p (X ^ {(i)} \mid \Theta^ {(i)} , Z ^ {(i)}) p (\Theta^ {(i)} \mid c ^ {(i)}) p (c ^ {(i)}) p (A ^ {(i)} \mid Z ^ {(i)}) p (Z ^ {(i)})}{q (\Theta^ {(i)} \mid X ^ {(i)} , c ^ {(i)}) q (c ^ {(i)} \mid X ^ {(i)}) q (Z ^ {(i)} \mid A ^ {(i)} , E ^ {(i)})} \right] \\ = \mathbb {E} _ {Q} \left[ \log \prod_ {j = 1} ^ {J} p (x _ {j} ^ {(i)} \mid \theta_ {j} ^ {(i)}, Z ^ {(i)}) \right] + \mathbb {E} _ {Q} \left[ \log \prod_ {j = 1} ^ {J} \frac {p (\theta_ {j} ^ {(i)} \mid c ^ {(i)})}{q (\theta_ {j} ^ {(i)} \mid x _ {j} ^ {(i)} , c ^ {(i)})} \right] \\ + \mathbb {E} _ {Q} \left[ \log \frac {p (c ^ {(i)})}{q \left(c ^ {(i)} \mid X ^ {(i)}\right)} \right] + \mathbb {E} _ {Q} \left[ \log p \left(A ^ {(i)} \mid Z ^ {(i)}\right) \right] + \mathbb {E} _ {Q} \left[ \log \frac {p \left(Z ^ {(i)}\right)}{q \left(Z ^ {(i)} \mid A ^ {(i)} , E ^ {(i)}\right)} \right] \\ = \sum_ {j = 1} ^ {J} \mathbb {E} _ {Q} \left[ \log p \left(x _ {j} ^ {(i)} \mid \theta_ {j} ^ {(i)}, Z ^ {(i)}\right) \right] + \sum_ {j = 1} ^ {J} \mathbb {E} _ {Q} \left[ \log \frac {p \left(\theta_ {j} ^ {(i)} \mid c ^ {(i)}\right)}{q \left(\theta_ {j} ^ {(i)} \mid x _ {j} ^ {(i)} , c ^ {(i)}\right)} \right] \\ + \mathbb {E} _ {Q} \left[ \log \frac {p (c ^ {(i)})}{q (c ^ {(i)} \mid X ^ {(i)})} \right] + \mathbb {E} _ {Q} \left[ \log p (A ^ {(i)} \mid Z ^ {(i)}) \right] + \mathbb {E} _ {Q} \left[ \log \frac {p (Z ^ {(i)})}{q (Z ^ {(i)} \mid A ^ {(i)} , E ^ {(i)})} \right] \\ = \mathcal {L} _ {E L B O} \tag {11} \\ \end{array}
$$
E.2 Solving topic parameters  $\{\pi_k^{(i)},\mu_k^{(i)},\Sigma_k^{(i)}\}_{k = 1}^K$  with Expectation Maximization
The log likelihood function is given by
$$
\ln p \left(Z ^ {(i)} \mid \pi^ {(i)}, \mu^ {(i)}, \Sigma^ {(i)}\right) = \sum_ {v = 1} ^ {V} \ln \left[ \sum_ {k = 1} ^ {K} \pi_ {k} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {k} ^ {(i)}, \Sigma_ {k} ^ {(i)}\right) \right]. \tag {12}
$$
1. Deriving  $\mu_{k}^{(i)}$
Setting the derivatives of  $\ln p(Z^{(i)}\mid \pi^{(i)},\mu^{(i)},\Sigma^{(i)})$  w.r.t the means  $\mu_k^{(i)}$  to zero, we have
$$
- \sum_ {v = 1} ^ {V} \frac {\pi_ {k} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {k} ^ {(i)} , \Sigma_ {k} ^ {(i)}\right)}{\sum_ {s = 1} ^ {K} \pi_ {s} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {s} ^ {(i)} , \Sigma_ {s} ^ {(i)}\right)} \Sigma_ {k} ^ {(i)} \left(z _ {v} ^ {(i)} - \mu_ {k} ^ {(i)}\right) = 0. \tag {13}
$$
Define the posterior probabilities as
$$
\gamma_ {v k} = p \left(y _ {v} ^ {(i)} = k \mid z _ {v} ^ {(i)}\right) = \frac {\pi_ {k} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {k} ^ {(i)} , \Sigma_ {k} ^ {(i)}\right)}{\sum_ {s = 1} ^ {K} \pi_ {s} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {s} ^ {(i)} , \Sigma_ {s} ^ {(i)}\right)}. \tag {14}
$$
Multiplying by  $\Sigma_k^{(i)^{-1}}$  and rearranging, we can obtain the updating formula for  $\mu_k^{(i)}$  as
$$
\mu_ {k} ^ {(i)} = \frac {\sum_ {v} \gamma_ {v k} \cdot z _ {v} ^ {(i)}}{\sum_ {v} \gamma_ {v k}}. \tag {15}
$$
2. Deriving  $\Sigma_k^{(i)}$
Similarly, we set the derivatives of  $\ln p(Z^{(i)}\mid \pi^{(i)},\mu^{(i)},\Sigma^{(i)})$  w.r.t  $\Sigma_k^{(i)}$  to zero, then we have
$$
- \frac {1}{2} \sum_ {v = 1} ^ {V} \frac {\pi_ {k} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {k} ^ {(i)} , \Sigma_ {k} ^ {(i)}\right)}{\sum_ {s = 1} ^ {K} \pi_ {s} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {s} ^ {(i)} , \Sigma_ {s} ^ {(i)}\right)} \Sigma_ {k} ^ {(i) - 1} \left[ 1 + \left(z _ {v} ^ {(i)} - \mu_ {k} ^ {(i)}\right) ^ {T} \Sigma_ {k} ^ {(i) - 1} \left(z _ {v} ^ {(i)} - \mu_ {k} ^ {(i)}\right) \right] = 0. \tag {16}
$$
Using  $\gamma_{vk}$  in Eq. 4 and rearranging, we get the updating formula for  $\Sigma_k^{(i)}$  as
$$
\Sigma_ {k} ^ {(i)} = \frac {\sum_ {v} \gamma_ {v k} \cdot \left(z _ {v} ^ {(i)} - \mu_ {k} ^ {(i)}\right) \left(z _ {v} ^ {(i)} - \mu_ {k} ^ {(i)}\right) ^ {T}}{\sum_ {v} \gamma_ {v k}}. \tag {17}
$$
3. Deriving  $\pi_k^{(i)}$
Finally, using Lagrange multiplier algorithm, our goal is to maximize the following formula:
$$
\sum_ {v = 1} ^ {V} \ln \left[ \sum_ {k = 1} ^ {K} \pi_ {k} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {k} ^ {(i)}, \Sigma_ {k} ^ {(i)}\right) \right] + \lambda \left(\sum_ {k = 1} ^ {K} \pi_ {k} ^ {(i)} - 1\right), \tag {18}
$$
where  $\sum_{k=1}^{K} \pi_k^{(i)} = 1$ .
Then setting the derivatives of the above equation w.r.t  $\pi_k^{(i)}$  to zero, we have
$$
\sum_ {v = 1} ^ {V} \frac {\pi_ {k} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {k} ^ {(i)} , \Sigma_ {k} ^ {(i)}\right)}{\sum_ {s = 1} ^ {K} \pi_ {s} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {s} ^ {(i)} , \Sigma_ {s} ^ {(i)}\right)} + \lambda = 0. \tag {19}
$$
Multiplying  $\pi_k^{(i)}$  and rearranging, we obtain
$$
\pi_ {k} ^ {(i)} = - \frac {\sum_ {v = 1} ^ {V} \frac {\pi_ {k} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {k} ^ {(i)} , \Sigma_ {k} ^ {(i)}\right)}{\sum_ {s = 1} ^ {K} \pi_ {s} ^ {(i)} \mathcal {N} \left(z _ {v} ^ {(i)} \mid \mu_ {s} ^ {(i)} , \Sigma_ {s} ^ {(i)}\right)}}{\lambda} = - \frac {\sum_ {v} \gamma_ {v k}}{\lambda}. \tag {20}
$$
Considering  $\sum_{k=1}^{K} \pi_k^{(i)} = 1$ , then  $\sum_{k} - \frac{\sum_{v} \gamma_{vk}}{\lambda} = 1$ , and  $\lambda = \sum_{v} \sum_{k} \gamma_{vk}$ .
Hence the updating formula for  $\pi_k^{(i)}$  as
$$
\pi_ {k} ^ {(i)} = \frac {\sum_ {v} \gamma_ {v k}}{\sum_ {v} \sum_ {k} \gamma_ {v k}}. \tag {21}
$$
Table 5: PPL results on four datasets with 20 documents in each task.
Methods	20NG	Yahoo	DB14	WOS
LDA[42]	2979	3916	3095	2370
PFA[12]	2439	2545	1903	1675
ProdLDA[43]	4807	6093	5819	4617
ETM[18]	3276	2781	2870	3189
MAML-ProdLDA*	4378	4033	4612	3908
MAML-ETM*	3287	3439	2819	4189
Meta-SawETM[30]	2657	2859	2355	3620
CombinedTM[21]	2331	2543	1863	2587
ZeroShotTM[22]	2673	2664	1722	2660
Meta-CETM	1216	1369	1109	1482
F.1 Perplexity results
In Table 1, we list the PPL results of different compared methods on four datasets, where the number of documents is 5 or 10 in each task. In this part, we provide more PPL quantitative results with varied numbers of texts per task  $\{20, 50, 100\}$  in Table 5, Table 6 and Table 7, respectively.
Table 6: PPL results on four datasets with 50 documents in each task.
Methods	20NG	Yahoo	DB14	WOS
LDA[42]	2443	3279	2353	2091
PFA[12]	2271	2326	1887	1663
ProdLDA[43]	4489	5784	5794	4386
ETM[18]	3215	3916	3095	2370
MAML-ProdLDA*	4372	3951	4463	3863
MAML-ETM*	3186	3315	2778	4062
Meta-SawETM[30]	3761	3037	2577	3365
CombinedTM[21]	2267	2481	1765	2473
ZeroShotTM[22]	2397	2496	1638	2497
Meta-CETM	1517	1440	1306	1576
F.2 Topic quality results
In Sec. 3.2.2, we present the topic interpretability results including topic diversity (TD) and topic coherence (TC) of six compared methods. Except for CombinedTM [21] and ZeroShotTM [22], we carry on experiments applying another contextual topic model (CTM) CETopicTM [24] with SimCSE pretrained word embeddings $^{8}$  on four datasets. The results are exhibited in Fig. 7. It can be notably noticed CETopicTM [24] achieves much competitive results on both TD and TC scores, even compared with CombinedTM [21] and ZeroShotTM [22]. Such superiority is owed to the fact that CETopicTM utilizes word embeddings learned from large-scale BERT data and it performs clustering on sentence embeddings to generate topics. In our settings, the aim is to provide a framework for training a sufficiently generalized topic model in low-resource regimes, while equipped with BERT embeddings, CETopicTM is highly likely to obtain context-related meanings in advance under most situations. But in some cases where the words or the word meanings have not been encountered or
<sup>8</sup>https://huggingface.co/princeton-nlp/unsup-simcse-bert-base-uncased
Table 7: PPL results on four datasets with 100 documents in each task.
Methods	20NG	Yahoo	DB14	WOS
LDA[42]	2118	2833	1858	1896
PFA[12]	2060	2169	1637	1643
ProdLDA[43]	4466	5736	6016	4369
ETM[18]	3199	2811	2837	3296
MAML-ProdLDA*	4359	4202	4381	3845
MAML-ETM*	3172	3256	2715	3947
Meta-SawETM[30]	3661	3251	2984	3101
CombinedTM[21]	2205	2286	1695	2330
ZeroShotTM[22]	2330	2319	1604	2372
Meta-CETM	2138	1743	1468	1786
learned by BERT, such as some specialized occasions, CETopicTM may fail to extract interpretable topics.
Figure 7: Topic diversity results (top row) and topic coherence results (bottom row) of seven compared methods on four datasets. Compared with Fig. 2, we add the results of CETopicTM [24] in this figure.
F.3 Topic visualization results
In Fig. 3, we visualize the adapted embedding space of different methods to demonstrate our Meta-CETM's successful fast adaption. Further, to better characterize meaningful and coherent topics learned by our model given a few number of documents, we display the text and topics extracted by Meta-SawETM [30], CombinedTM [21] and our Meta-CETM in Fig. 8.
Figure 8: A paragraph of text and top five words of three topics from Meta-SawETM, CombinedTM and our Meta-CETM. It can be clearly found that Meta-CETM learns the most relevant topics among the three models.