SlowGuess/NAACL_2021_OCR huggingface dataset:
data	labels / classes
TMN*	world, us, sport, business, sci_tech, entertainment, health
AGnews	business, sci_tech, sports, world
20NS	misc.forsale, comp.graphics, rec.autos, comp.windows.x,
20NSshort,	rec.sport.baseball, sci.space, rec.sport.hockey,
20NSsmall,	soc.religion.christian, rec.motorcycles, comp.sys.mac.hardware,
talk.religion.misc, sci.electronics, comp.os.ms-windows.misc,
sci.med, comp.sys.ibm.pc.hardware, talk.politics.mideast,
talk.politics.guns, talk.politics.misc, alt atheism, sci.crypt
R21578title	trade, grain, crude, corn, rice, rubber, sugar, palm-oil,
R21578	veg-oil, ship, coffee, wheat, gold, acq, interest, money-fx,
carcass, livestock, oilseed, soybean, earn, bop, gas, lead, zinc,
gnp, soy-oil, dlr, yen, nickel, groundnut, heat, sorghum, sunseed,
cocoa, rapeseed, cotton, money-supply, iron-steel, palladium,
platinum, strategic-metal, reserves, groundnut-oil, lin-oil, meal-feed,
sun-meat, sun-oil, hog, barley, potato, orange, soy-m meal, cotton-oil,
fuel, silver, income, wpi, tea, lei, coconut, coconut-oil, copra-cake,
propane, instal-debt, nzdr, housing, nkr, rye, castor-oil, palmkernel,
tin, copper, cpi, pet-chem, rape-oil, oat, naphtha, cpu, rand, alum
Table 11: Label space of the corpora. TMN*:TMN or TMNtitle
DocNADE, where each word vector is of  $H = 200$  dimension.
Since the row vector of  $\mathbf{W}_{j;\cdot}$  in DocNADE encodes  $j$ th topic feature, therefore each latent topic (i.e., row) in feature matrix  $\mathbf{W}$  is a vector of  $K$  dimension, corresponding the definition of topics that it is a distribution over vocabulary.  $H$  is the number of latent topics and  $K$  is the vocabulary size, where  $K$  varies across corpora. Thus, we train DocNADE to learn a feature matrix specific to each of the source corpora, e.g.  $\mathbf{W}^k\in \mathbb{R}^{H\times K}$  of  $k$ th source.
For a target corpus of vocabulary size  $K'$ , the DocNADE learns a feature matrix  $\mathbf{W}^T \in \mathbb{R}^{H \times K'}$ . Similarly,  $\mathbf{W}^k \in \mathbb{R}^{H \times K}$  for  $k$ th source of vocabulary size  $K$ . Since in the sparse-data setting for the target,  $K' < K$  due to additional word in the source. To perform GVT, we need the same topic feature dimensions in the target and source, i.e.,  $K'$  of the target. Therefore, we remove those column vectors from  $\mathbf{W}^k \in \mathbb{R}^{H \times K}$  of the  $k$ th source for which there is no corresponding word in the vocabulary of the target domain. As a result, we obtain  $\mathbf{Z}^k$  as a latent topic feature matrix to be used in knowledge transfer to the target domain. Following the similar steps, we prepare a KB of Zs such that each latent topic feature matrix from a source domain gets the same topic feature dimension as the target.
For DocNADE and DocNADEe in different knowledge transfer configurations, we follow the same experimental setup as in DocNADE and DocNADEe. We rerun DocNADE and DocNADEe using the code released for DocNADEe. For all the hyperpa
Hyperparameter	Search Space
retrieval fraction	[0.02]
learning rate	[0.001]
hidden units, H	[200]
activation function (g)	sigmoid
iterations	[100]
λk	[1.0, 0.5, 0.1]
γk	[0.1, 0.01, 0.001]
Table 12: Hyperparameters in Generalization experiments of DocNADE, DocNADEe, LVT, GVT and MVT
Hyperparameter	Search Space
retrieval fraction	[0.02]
learning rate	[0.001]
hidden units, H	[200]
activation function (g)	tanh
iterations	[100]
λk	[1.0, 0.5, 0.1]
γk	[0.1, 0.01, 0.001]
rameters, optimal values are selected based on the performance on development set.
C.1 Experimental Setup for Generalization
We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with sigmoid hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in PPL for  $H = 200$  topics than using 50, therefore we use  $H = 200$  in our experiments. See Table 12 for hyperparameters used in generalization task, i.e., computing PPL.
C.2 Experimental Setup for IR Task
We treat all test documents as queries to retrieve a fraction of the closest documents in the original training set using cosine similarity between their document vectors. To compute retrieval precision for each fraction (e.g., 0.02), we average the number of retrieved training documents with the same label as the query.
We set the maximum number of training passes to 100, topics to 200 and the learning rate to 0.001 with tanh hidden activation. Since the baseline DocNADE and DocNADEe reported better scores in precision for the retrieval task for  $H = 200$  topics than using 50, therefore we use  $H = 200$  in our experiments. We follow the similar experimental setup as in DocNADEe. For model selection,
Table 13: Hyperparameters search in the IR task, where  $\lambda^k$  and  $\gamma^k$  are weights for  $k$ th source.
Scores on Target Corpus (in sparse-data setting)
Type	20NSshort	TMNtitle	20NSsmall
PPL	COH	IR	PPL	COH	IR	PPL	COH	IR
+	MST	LVT	667	.661	.308	670	.730	.535	610	.440	.286
GVT	651	.658	.285	701	.712	.523	602	.460	.273
MVT	667	.660	.309	667	.730	.535	608	.441	.293
+ Glove	662	.677	.296	672	.731	.540	634	.412	.207
×	MST	LVT	640	.678	.308	663	.732	.547	596	.442	.277
GVT	658	.705	.305	704	.746	.550	599	.585	.326
MVT	656	.721	.314	680	.752	.556	600	.600	.285
+ Glove	644	.719	.293	687	.752	.538	609	.586	.282
Table 14:  $\{\lambda, \gamma\}$  as Parameter (+) vs Hyperparameters  $(\times)$ : Perplexity (PPL), topic coherence (COH) and precision@recall (IR) at retrieval fraction 0.02, when  $\lambda$  and  $\gamma$  are (1) learned with backpropagation, and (2) treated as hyperparameters. Results suggest the superiority of the second configuration.
we used the validation set as the query set and used the average precision at 0.02 retrieved documents as the performance measure. Note that the labels are not used during training. The class labels are only used to check if the retrieved documents have the same class label as the query document. To perform document retrieval, we use the same train/development/test split of documents as for PPL setup.
Given DocNADE, the representation of a document of size  $D$  can be computed by taking the last hidden vector  $\mathbf{h}_D$  at the autoregressive step  $D$ . Since, the RSM and DocNADE strictly outperformed LDA, therefore we only compare DocNADE and its recent extension DocNADEe. We use the same number of topic dimensions ( $H = 200$ ) across all the source and target in training DocNADE.
See Table 13 for the hyperparameters in the document retrieval task, where  $\lambda^k$  and  $\gamma^k$  are weights for  $k$ th source. We use the same grid-search for all the source domains. We set  $\gamma^k$  smaller than  $\lambda^k$  to control the degree of imitation of the source domain(s) by the target domain. We use the development set of the target corpus to find the optimal setting in different configurations of knowledge transfers from several sources.
C.3  $\{\lambda, \gamma\}$  as Parameter vs Hyperparameters
Here, we treat  $\lambda$  and  $\gamma$  as parameters of the model, instead of hyperparameters and learn them with backpropagation. We initialize each  $\lambda^k = 0.5$  and  $\gamma^k = 0.01$  for each of the sources. We perform experiments on short-text datasets in MST+LVT, MST+GVT and MST+MVT configurations. We evaluate the topic modeling using PPL, topic coherence and retrieval accuracy. Table 14 reports
the scores, when  $\lambda$  and  $\gamma$  are (1) learned with backpropagation, and (2) treated as hyperparameters. The experimental results suggest that the second configuration performs better the former. Thus, we have reported scores considering  $\{\lambda, \gamma\}$  as hyperparameters.
C.4 Reproducibility: Optimal Configurations of  $\lambda$  and  $\gamma$
As mentioned in Tables 12 and 13, the hyperparameter  $\lambda^k$  takes on values in [1.0, 0.5, 0.1] for each of the word embeddings matrix  $\mathbf{E}^k$  and  $\gamma^k$  in [0.1, 0.01, 0.001] for each of the latent topic features  $\mathbf{Z}^k$ , respectively for the  $k^{th}$  source domain. To determine an optimal configuration, we perform grid-search over the values and use the scores on the development set to determine the best setting. We have a common model for PPL and COH scores due to generalization.
To reproduce scores (best/bold in Table 5, we mentioned the best settings of  $(\lambda^k,\gamma^k)$  in MST+MVT configuration for each of the target and source combinations:
1. Generalization (PPL and COH) in MST+MVT when target is 20NSshort:  $\begin{array}{rlr}{(\lambda^{20NS} =} & {1.0,} & {\gamma^{20NS} = 0.001,}\\ {\lambda^{TMN} = 0.1,} & {\gamma^{TMN} = 0.001,}\\ {\lambda^{R21578} = 0.5,} & {\gamma^{R21578} = 0.001,} \end{array}$ $\lambda^{AGnews} = 0.1,\gamma^{AGnews} = 0.001$
2. Generalization (PPL and COH) in MST+MVT when target is TMN title:  $\begin{array}{rlrlrl}{(\lambda^{20NS}} & = & 0.1, & \gamma^{20NS} & = & 0.001,\\ {\lambda^{TMN}} & = & 1.0, & \gamma^{TMN} & = & 0.001,\\ {\lambda^{R21578}} & = & 0.5, & \gamma^{R21578} & = & 0.001,\\ {\lambda^{AGnews}} & = 1.0, & \gamma^{AGnews} & = 0.001 \end{array}$
3. Generalization (PPL and COH) in MST+MVT when target is R21578 title:  
 $\lambda^{20NS} = 0.1, \gamma^{20NS} = 0.001,$ $\lambda^{TMN} = 0.5, \gamma^{TMN} = 0.001,$ $\lambda^{R21578} = 1.0, \gamma^{R21578} = 0.001,$ $\lambda^{AGnews} = 1.0, \gamma^{AGnews} = 0.001$
4. Generalization (PPL and COH) in MST+MVT when target is 20NSsmall:  $\begin{array}{rlrlrl}{(\lambda^{20NS}} & = & {0.5,} & {\gamma^{20NS}} & = & {0.001,}\\ {\lambda^{TMN}} & = & {0.1,} & {\gamma^{TMN}} & = & {0.001,}\\ {\lambda^{R21578}} & = & {0.1,} & {\gamma^{R21578}} & = & {0.001,} \end{array}$ $\lambda^{AGnews} = 0.1,\gamma^{AGnews} = 0.001$
5. Generalization (PPL and COH) in MST+MVT when target is
$$
\begin{array}{l l l} \text {O h s u m e d t i t l e :} & (\lambda^ {A G n e w s} = & 0. 1, \\ \gamma^ {A G n e w s} = & 0. 0 0 1, & \lambda^ {P u b M e d} = 1. 0, \\ \gamma^ {P u b M e d} = 0. 0 0 1 \end{array}
$$
6. Generalization (PPL and COH) in MST+MVT when target is Ohsumed:  $\lambda^{AGnews} = 0.1$ ,  $\gamma^{AGnews} = 0.001$ ,  $\lambda^{PubMed} = 1.0$ ,  $\gamma^{PubMed} = 0.001$
7. IR in MST+MVT when target is 20NSshort:  $(\lambda^{20NS} = 1.0, \gamma^{20NS} = 0.1, \lambda^{TMN} = 0.5, \gamma^{TMN} = 0.01, \lambda^{R21578} = 0.1, \gamma^{R21578} = 0.001, \lambda^{AGnews} = 1.0, \gamma^{AGnews} = 0.01$
8. IR in MST+MVT when target is TMN title:  $\begin{array}{l}(\lambda^{20NS} = 0.1,\gamma^{20NS} = 0.01,\lambda^{TMN} = 1.0,\\ \gamma^{TMN} = 0.01,\lambda^{R21578} = 0.1,\gamma^{R21578} = \\ 0.01,\lambda^{AGnews} = 0.5,\gamma^{AGnews} = 0.001 \end{array}$
9. IR in MST+MVT when target is R21578 title:  $(\lambda^{20NS}) = 0.1,$ $\gamma^{20NS} = 0.01, \lambda^{TMN} = 1.0, \gamma^{TMN} = 0.01, \lambda^{R21578} = 1.0, \gamma^{R21578} = 0.01, \lambda^{AGnews} = 0.5, \gamma^{AGnews} = 0.001$
10. IR in MST+GVT when target is 20NSsmall:  $(\gamma^{20NS}) = 0.01,$ $\gamma^{TMN} = 0.01, \gamma^{R21578} = 0.1,$ $\gamma^{AGnews} = 0.01$
11. IR in MST+MVT when target is Ohsumedtitle:  $(\lambda^{AGnews} = 0.1, \gamma^{AGnews} = 0.001, \lambda^{PubMed} = 1.0, \gamma^{PubMed} = 0.1$
12. IR in MST+MVT when target is Ohsumed:  $\begin{array}{rlr}{(\lambda^{AGnews} =} & 0.1, & {\gamma^{AGnews} =} & 0.001,\\ {\lambda^{PubMed} = 0.5,} & {\gamma^{PubMed} = 0.1} \end{array}$
The hyper-parameters mentioned above also applies to a single source transfer configuration.