SlowGuess/NAACL_2022_OCR huggingface dataset:
Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In WSDM'15, pages 399-408.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL'16, pages 1715-1725.
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. IEEE TKDE, 30(10):1825-1837.
Suzanna Sia, Ayush Dalmia, and Sabrina J Mielke. 2020. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In EMNLP'20, pages 1728-1736.
Jian Tang, Meng Qu, and Qiaozhu Mei. 2015. Pte: Predictive text embedding through large-scale heterogeneous text networks. In KDD'15, pages 1165-1174.
Laure Thompson and David Mimno. 2020. Topic modeling with contextualized word representation clusters. arXiv preprint arXiv:2010.12626.
Dingding Wang, Shenghuo Zhu, Tao Li, and Yihong Gong. 2009. Multi-document summarization using sentence-based topic models. In ACL'09, pages 297-300.
Mu-Chun Wang, Zixuan Liu, and Sheng Wang. 2022. Textomics: A dataset for genomics data summary generation. In ACL'22.
Guangxu Xun, Yaliang Li, Jing Gao, and Aidong Zhang. 2017a. Collaboratively improving topic discovery and word embeddings by coordinating global and local contexts. In KDD'17, pages 535-543.
Guangxu Xun, Yaliang Li, Wayne Xin Zhao, Jing Gao, and Aidong Zhang. 2017b. A correlated topic model using word embeddings. In *IJCAI'17*, pages 4207-4213.
Chao Zhang, Keyang Zhang, Quan Yuan, Fangbo Tao, Luming Zhang, Tim Hanratty, and Jiawei Han. 2017. React: Online multimodal embedding for recency-aware spatiotemporal activity modeling. In SI-GIR'17, pages 245-254.
Yu Zhang, Shweta Garg, Yu Meng, Xiusi Chen, and Jiawei Han. 2022. Motifclass: Weakly supervised text classification with higher-order metadata information. In WSDM'22, pages 1357-1367.
Yu Zhang, Yu Meng, Jiaxin Huang, Frank F Xu, Xuan Wang, and Jiawei Han. 2020. Minimally supervised categorization of text with metadata. In SIGIR'20, pages 1231-1240.
A The Embedding Learning Objective
In Section 3.2, we propose the following embedding learning objective:
$$
\begin{array}{l} \mathcal {J} = \underbrace {\sum_ {d \in \mathcal {D}} \sum_ {w _ {i} \in d} \sum_ {w _ {j} \in \mathcal {C} (w _ {i} , h)} \frac {\exp (\boldsymbol {u} _ {w _ {i}} ^ {T} \boldsymbol {v} _ {w _ {j}})}{\sum_ {w ^ {\prime} \in \mathcal {V} _ {\mathcal {D}}} \exp (\boldsymbol {u} _ {w _ {i}} ^ {T} \boldsymbol {v} _ {w ^ {\prime}})}} _ {\mathcal {I} _ {\text {c o n t e x t}}} + \\ \underbrace{\sum_{d\in\mathcal{D}}\sum_{w\in d}\frac{\exp(\boldsymbol{u}_{w}^{T}\boldsymbol{v}_{d})}{\sum_{d^{\prime}\in\mathcal{D}}\exp(\boldsymbol{u}_{w}^{T}\boldsymbol{v}_{d^{\prime}})}}_{\mathcal{I}_{\text{document}}} + \\ \underbrace {\sum_ {c _ {i} \in \mathcal {C}} \sum_ {w \in \mathcal {S} _ {i}} \frac {\exp \left(\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {c _ {i}}\right)}{\sum_ {c ^ {\prime} \in \mathcal {C}} \exp \left(\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {c ^ {\prime}}\right)}} _ {\mathcal {I} _ {\text {c a t e g o r y}}} \cdot \tag {12} \\ \end{array}
$$
Now we prove that maximizing  $\mathcal{J}$  is implicitly performing the factorization in Eq. (3).
Levy and Goldberg (2014) have proved that maximizing  $\mathcal{I}_{\mathrm{context}}$  is implicitly doing the following factorization.
$$
\boldsymbol {u} _ {w _ {i}} ^ {T} \boldsymbol {v} _ {w _ {j}} = \log \left(\frac {\# _ {\mathcal {D}} \left(w _ {i} , w _ {j}\right) \cdot \lambda_ {\mathcal {D}}}{\# _ {\mathcal {D}} \left(w _ {i}\right) \cdot \# _ {\mathcal {D}} \left(w _ {j}\right) \cdot b}\right), \tag {13}
$$
$$
\text {i . e ,} \quad \mathbf {U} _ {w} ^ {T} \mathbf {V} _ {w} = \mathbf {X} _ {w w}.
$$
We follow their approach to consider the other two terms  $\mathcal{I}_{\mathrm{document}}$  and  $\mathcal{I}_{\mathrm{category}}$  in Eq. (12). Using the negative sampling strategy to rewrite  $\mathcal{I}_{\mathrm{document}}$ , we get
$$
\sum_ {w \in \mathcal {V} _ {\mathcal {D}}} \sum_ {d \in \mathcal {D}} \# _ {d} (w) \left(\log \sigma \left(\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {d}\right) + b \mathbb {E} _ {d ^ {\prime}} \left[ \log \sigma \left(- \boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {d ^ {\prime}}\right) \right]\right), \tag {14}
$$
where  $\sigma (\cdot)$  is the sigmoid function. Following (Levy and Goldberg, 2014; Qiu et al., 2018), we assume the negative sampling distribution  $\propto \lambda_{d}$ . Then, the objective becomes
$$
\sum_ {w \in \mathcal {V} _ {\mathcal {D}}} \sum_ {d \in \mathcal {D}} \# _ {d} (w) \log \sigma (\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {d}) +
$$
$$
\sum_ {w \in \mathcal {V} _ {\mathcal {D}}} \# _ {\mathcal {D}} (w) \sum_ {d ^ {\prime} \in \mathcal {D}} \frac {b \cdot \lambda_ {d ^ {\prime}}}{\lambda_ {\mathcal {D}}} \log \sigma \left(- \boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {d ^ {\prime}}\right). \tag {15}
$$
For a specific term-document pair  $(w, d)$ , we consider its effect in the objective:
$$
\mathcal {J} _ {w, d} = \# _ {d} (w) \log \sigma \left(\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {d}\right) + \# _ {\mathcal {D}} (w) \frac {b \cdot \lambda_ {d}}{\lambda_ {\mathcal {D}}} \log \sigma \left(- \boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {d}\right). \tag {16}
$$
Let  $x_{w,d} = \pmb{u}_w^T\pmb{v}_d$ . To maximize  $\mathcal{J}_{w,d}$ , we should have
$$
0 = \frac {\partial \mathcal {J} _ {w , d}}{\partial x _ {w , d}} = \# _ {d} (w) \sigma \left(- x _ {w, d}\right) - \frac {\# _ {\mathcal {D}} (w) \cdot b \cdot \lambda_ {d}}{\lambda_ {\mathcal {D}}} \sigma \left(x _ {w, d}\right). \tag {17}
$$
In practice, the negative sampling distribution  $\propto \lambda_d^{3/4}$ , but related studies (Levy and Goldberg, 2014; Qiu et al., 2018) usually assume a linear relationship in their derivation.
$$
e ^ {2 x _ {w, d}} - \left(\frac {\# _ {d} (w) \cdot \lambda_ {\mathcal {D}}}{\# _ {\mathcal {D}} (w) \cdot b \cdot \lambda_ {d}} - 1\right) e ^ {x _ {w, d}} - \frac {\# _ {d} (w) \cdot \lambda_ {\mathcal {D}}}{\# _ {\mathcal {D}} (w) \cdot b \cdot \lambda_ {d}} = 0. \tag {18}
$$
Therefore,  $e^{x_{w,d}} = -1$  (which is invalid) or  $e^{x_{w,d}} = \frac{\#_d(w)\cdot\lambda_{\mathcal{D}}}{\#\mathcal{D}(w)\cdot b\cdot\lambda_d}$ . In other words,
$$
\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {d} = x _ {w, d} = \log \left(\frac {\# _ {d} (w) \cdot \lambda_ {\mathcal {D}}}{\# _ {\mathcal {D}} (w) \cdot b \cdot \lambda_ {d}}\right), \tag {19}
$$
$$
i. e., \mathbf {U} _ {w} ^ {T} \mathbf {V} _ {d} = \mathbf {X} _ {w d}.
$$
Similarly, for  $\mathcal{I}_{\mathrm{category}}$ , the objective can be rewritten as
$$
\sum_ {w \in \mathcal {V} _ {\mathcal {D}}} \sum_ {c _ {i} \in \mathcal {C}} \mathbf {1} _ {w \in \mathcal {S} _ {i}} \log \sigma (\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {c _ {i}}) +
$$
$$
\sum_ {w \in \mathcal {V} _ {\mathcal {D}}} \mathbf {1} _ {w \in \mathcal {S} _ {1} \cup \dots \cup \mathcal {S} _ {| \mathcal {C} |}} \sum_ {c ^ {\prime} \in \mathcal {C}} \frac {b}{| \mathcal {C} |} \log \sigma \left(- \boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {c ^ {\prime}}\right). \tag {20}
$$
Following the derivation of  $\mathcal{I}_{\mathrm{document}}$ , we get
$$
\boldsymbol {u} _ {w} ^ {T} \boldsymbol {v} _ {c _ {i}} = x _ {w, c _ {i}} = \log \left(\frac {\mathbf {1} _ {w \in \mathcal {S} _ {i}} | \mathcal {C} |}{\mathbf {1} _ {w \in \mathcal {S} _ {1} \cup \dots \cup \mathcal {S} _ {| \mathcal {C} |}} \cdot b}\right), \tag {21}
$$
$$
i. e., \mathbf {U} _ {w} ^ {T} \mathbf {V} _ {c _ {i}} = \mathbf {X} _ {w c}.
$$
Putting Eqs. (13), (19), and (21) together gives us Eq. (3).
B The Ensemble Ranking Function
In Section 3.3, we propose the following ensemble ranking function:
$$
\operatorname {s c o r e} (w \mid \mathcal {S} _ {i}) = \left(\frac {1}{2} \left(\frac {1}{\operatorname {r a n k} _ {G} (w)}\right) ^ {\rho} + \frac {1}{2} \left(\frac {1}{\operatorname {r a n k} _ {L} (w)}\right) ^ {\rho}\right) ^ {1 / \rho}. \tag {22}
$$
Now we prove this ranking function is a generalization of the arithmetic mean reciprocal rank (i.e., MRR) and the geometric mean reciprocal rank:
$$
\lim  _ {\rho \rightarrow 1} \operatorname {s c o r e} (w | \mathcal {S} _ {i}) = \frac {1}{2} \left(\frac {1}{\operatorname {r a n k} _ {G} (w)} + \frac {1}{\operatorname {r a n k} _ {L} (w)}\right); \tag {23}
$$
$$
\lim  _ {\rho \rightarrow 0} \operatorname {s c o r e} (w | \mathcal {S} _ {i}) = \left(\frac {1}{\operatorname {r a n k} _ {G} (w)} \cdot \frac {1}{\operatorname {r a n k} _ {L} (w)}\right) ^ {1 / 2}.
$$
The case of  $\rho \to 1$  is trivial. When  $\rho \to 0$ , we aim to show that
$$
\lim  _ {\rho \rightarrow 0} \log \operatorname {s c o r e} (w | \mathcal {S} _ {i}) = \log \left(\frac {1}{\operatorname {r a n k} _ {G} (w)} \cdot \frac {1}{\operatorname {r a n k} _ {L} (w)}\right) ^ {1 / 2}. \tag {24}
$$
In fact, let  $r_G = \frac{1}{\operatorname{rank}_G(w)}$  and  $r_L = \frac{1}{\operatorname{rank}_L(w)}$ .
$$
\begin{array}{l} \lim  _ {\rho \rightarrow 0} \log \operatorname {s c o r e} (w | \mathcal {S} _ {i}) = \lim  _ {\rho \rightarrow 0} \log \left(\frac {1}{2} r _ {G} ^ {\rho} + \frac {1}{2} r _ {L} ^ {\rho}\right) ^ {1 / \rho} \\ = \lim  _ {\rho \rightarrow 0} \frac {\log \left(\frac {1}{2} r _ {G} ^ {\rho} + \frac {1}{2} r _ {L} ^ {\rho}\right)}{\rho} \\ = \lim  _ {\rho \rightarrow 0} \frac {\frac {\frac {1}{2} r _ {G} ^ {\rho} \log r _ {G} + \frac {1}{2} r _ {L} ^ {\rho} \log r _ {L}}{\frac {1}{2} r _ {G} ^ {\rho} + \frac {1}{2} r _ {L} ^ {\rho}}}{1} \\ = \frac {\lim  _ {\rho \rightarrow 0} \left(r _ {G} ^ {\rho} \log r _ {G} + r _ {L} ^ {\rho} \log r _ {L}\right)}{\lim  _ {\rho \rightarrow 0} \left(r _ {G} ^ {\rho} + r _ {L} ^ {\rho}\right)} \\ = \frac {\log r _ {G} + \log r _ {L}}{2} \\ = \log \left(r _ {G} \cdot r _ {L}\right) ^ {1 / 2}. \tag {25} \\ \end{array}
$$
The third line is obtained by applying L'Hopital's rule.