# Interpretable Topic Extraction and Word Embedding Learning Using Row-Stochastic DEDICOM

Lars Hillebrand, David Biesner, Christian Bauckhage, Rafet Sifa

► To cite this version:

Lars Hillebrand, David Biesner, Christian Bauckhage, Rafet Sifa. Interpretable Topic Extraction and Word Embedding Learning Using Row-Stochastic DEDICOM. 4th International Cross-Domain Conference for Machine Learning and Knowledge Extraction (CD-MAKE), Aug 2020, Dublin, Ireland. pp.401-422, 10.1007/978-3-030-57321-8_22. hal-03414746

HAL Id: hal-03414746

https://inria.hal.science/hal-03414746

Submitted on 4 Nov 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Interpretable Topic Extraction and Word
Embedding Learning using row-stochastic
DEDICOM

Lars Hillebrand*¹,², David Biesner*¹,², Christian Bauckhage¹,², and Rafet Sifa¹

¹ Fraunhofer IAIS

² University of Bonn

**Abstract.** The DEDICOM algorithm provides a uniquely interpretable matrix factorization method for symmetric and asymmetric square matrices. We employ a new row-stochastic variation of DEDICOM on the pointwise mutual information matrices of text corpora to identify latent topic clusters within the vocabulary and simultaneously learn interpretable word embeddings. We introduce a method to efficiently train a constrained DEDICOM algorithm and a qualitative evaluation of its topic modeling and word embedding performance.

**Keywords:** Word Embeddings · Topic Analysis · Matrix Factorization · Natural Language Processing.

# 1 Introduction

Matrix factorization methods have always been a staple in many natural language processing (NLP) tasks. Factorizing a matrix of word co-occurrences can create both low-dimensional representations of the vocabulary, so-called word embeddings [11, 15], that carry semantic and topical meaning within them, as well as representations of meaning that go beyond single words to latent topics.

DEcomposition into DIrectional COMponents (DEDICOM) is a matrix factorization technique that factorizes a square, possibly asymmetric, matrix of relationships between items into a loading matrix of low-dimensional representations of each item and an affinity matrix describing the relationships between the dimensions of the latent representation (see Figure 1 for an illustration).

We introduce a modified row-stochastic variation of DEDICOM, which allows for interpretable loading vectors and apply it to different matrices of word co-occurrence statistics created from Wikipedia based semi-artificial text documents. Our algorithm produces low-dimensional word embeddings, where one can interpret each latent factor as a topic that clusters words into meaningful categories. Hence, we show that row-stochastic DEDICOM successfully combines the task of learning interpretable word embeddings and extracting representative topics.

<footnote>* First authors, equal contribution.
Correspondence to lars.patrick.hillebrand@iais.fraunhofer.de</footnote>
Another interesting aspect of this type of factorization is the interpretability of the affinity matrix. An entry in the matrix directly describes the relationship between the topics of the respective row and column and one can therefore use this tool to extract topics that a certain text corpus deals with and analyse how these topics are connected in the given text.

In this work we first describe the aforementioned DEDICOM algorithm and provide details on the modified row-stochasticity constraint and on optimization. We then present results of various experiments on semi-artificial text documents (combinations of Wikipedia articles) that show how our approach is able to capture hidden latent topics within text corpora, cluster words in a meaningful way and find relationships between these topics within the documents.

## 2 Related Work

The DEDICOM algorithm has a long history of providing interpretable matrix factorization, mostly for rather low-dimensional tasks. First described in [6], it since has been applied to analysis of social networks [1], email correspondence [2] and video game player behaviour [16, 17]. DEDICOM also has successfully been employed to NLP tasks such as part of speech tagging [4], however to the best of our knowledge we provide the first implementation of DEDICOM for simultaneous word embedding learning and topic modeling.

Many works deal with the task of putting constraints on the factor matrices of the DEDICOM algorithm. In [2,17], the authors constrain the affinity matrix **R** to be non-negative, which aids interpretability and improves convergence behaviour if the matrix to be factorized is non-negative. However, their approach relies on the Kronecker product between matrices in the update step, solving a linear system of $n^2 \times k^2$, where *n* denotes the number of items in the input matrix and *k* the number of latent factors. These dimensions make the application on text data, where *n* describes the number of words in the vocabulary, a computationally futile task. Constraints on the loading matrix, **A**, include non-negativity as well (see [2]) or column-orthogonality as in [17].

In contrast, we propose a new modified row-stochasticity constraint on **A**, which is tailored to generate interpretable word embeddings that carry semantic meaning and represent a probability distribution over latent topics.

Previous matrix factorization based methods in the NLP context mostly dealt with either word embedding learning or topic modeling, but not with both tasks combined.

For word embeddings, the GloVe [15] model factorizes an adjusted co-occurrence matrix into two matrices of the same dimension. The work is based on a large text corpus with a vocabulary of *n* ≈ 400,000 and produces word embeddings of dimension *k* = 300. In order to maximize performance on the word analysis task, the authors adjusted the co-occurrence matrix to the logarithmized co-occurrence matrix and added bias terms to the optimization objective.

A model conceived around the same time, word2vec [13], calculates word embeddings not from a co-occurrence matrix but directly from the text corpus
using the skip-gram or continuous-bag-of-words approach. More recent work [11] has shown that this construction is equivalent to matrix factorization on the pointwise mutual information (PMI) matrix of the text corpus, which makes it very similar to the glove model described above.

Both models achieve impressive results on word embedding related tasks like word analogy, however the large dimensionality of the word embeddings makes interpreting the latent factors of the embeddings impossible.

On the topic modeling side, matrix factorization methods are routinely applied as well. Popular algorithms like non-negative matrix factorization (NMF) [10], singular value decomposition (SVD) [5, 18] and principal component analysis (PCA) [7] compete against the probabilistic latent dirichlet allocation (LDA) [3] to cluster the vocabulary of a word co-occurrence or document-term matrix into latent topics.<fnref n="3" /> Yet, we empirically show that the implicitly learned word embeddings of these methods lack semantic meaning in terms of the cosine similarity measure.

We benchmark our approach qualitatively against these methods in Section 4.3 and the appendix.

## 3 The row-stochastic DEDICOM Model

In this section we provide a detailed theoretical view at the proposed row-stochastic DEDICOM algorithm for factorizing word co-occurrence based positive pointwise mutual information matrices.

<figure><img src="image_2.png" /><figcaption>Figure 1: The DEDICOM algorithm factorizes a square matrix $S$ into a loading matrix $A$ and an affinity matrix $R$.</figcaption></figure>

For a given language corpus consisting of $n$ unique words $X = x_1, \dots, x_n$ we calculate a co-occurrence matrix $W \in \mathbb{R}^{n \times n}$ by iterating over the corpus on a word token level with a sliding context window of specified size. Then

$$W_{ij} = \#\text{word } i \text{ appears in context of word } j. \qquad (1)$$

Note that the word context window can be applied symmetrically or asymmetrically around each word. We choose a symmetric context window, which implies a symmetric co-occurrence matrix, $W_{ij} = W_{ji}$.

We then transform the co-occurrence matrix into the pointwise mutual information matrix (PMI), which normalizes the counts in order to extract meaningful

<footnote n="3">³ More recent expansions of these methods can be found in [9, 14].</footnote>
co-occurrences from the matrix. Co-occurrences of words that occur regularly in the corpus are decreased since their appearance together might be nothing more than a statistical phenomenon, the co-occurrence of words that appear less often in the corpus give us meaningful information about the relations between words and topics. We define the PMI matrix as

$$ \mathbf{PMI}_{ij} := \log \mathbf{W}_{ij} + \log N - \log N_i - \log N_j \quad (2) $$

where $N := \sum_{ij=1}^{n} \mathbf{W}_{ij}$ is the sum of all co-occurrence counts of $\mathbf{W}$, $N_i := \sum_{j=1}^{n} \mathbf{W}_{ij}$ the row sum and $N_j := \sum_{i=1}^{n} \mathbf{W}_{ij}$ the column sum.

Since the co-occurrence matrix $\mathbf{W}$ is symmetrical, the transformed PMI matrix is symmetrical as well. Nevertheless, DEDICOM is able to factorize both symmetrical and non-symmetrical matrices. We expand details on symmetrical and non-symmetrical relationships in Section 3.2.

Additionally, we want all entries of the matrix to be non-negative, our final matrix to be factorized is therefore the positive PMI (PPMI)

$$ S_{ij} = \mathbf{PPMI}_{ij} = \max\{0, \mathbf{PMI}_{ij}\}. \quad (3) $$

Our aim is to decompose this matrix using row-stochastic DEDICOM as

$$ S \approx \mathbf{ARA}^T, \quad \text{with} \quad S_{ij} \approx \sum_{b=1}^{k} \sum_{c=1}^{k} \mathbf{A}_{ib} \mathbf{R}_{bc} \mathbf{A}_{jc}, \quad (4) $$

where $\mathbf{A} \in \mathbb{R}^{n \times k}$, $\mathbf{R} \in \mathbb{R}^{k \times k}$, $\mathbf{A}^T$ denotes the transpose of $\mathbf{A}$ and $k \ll n$. Literature often refers to $\mathbf{A}$ as the loading matrix and $\mathbf{R}$ as the affinity matrix. $\mathbf{A}$ gives us for each word $i$ in the vocabulary a vector of size $k$, the number of latent topics we wish to extract. The square matrix $\mathbf{R}$ then provides possibility for interpretation of the relationships between these topics.

Empirical evidence has shown that the algorithm tends to favor columns unevenly, such that a single column receives a lot more weight in its entries than the other columns. We try to balance this behaviour by applying a column-wise z-normalization on $\mathbf{A}$, such that all columns have zero mean and unit variance.

In order to aid interpretability we wish each word embedding to be a distribution over all latent topics, i.e. entry $\mathbf{A}_{ib}$ in the word-embedding matrix provides information on how much topic $b$ describes word $i$.

To implement these constraints we therefore apply a row-wise softmax operation over the column-wise z-normalized $\mathbf{A}$ matrix by defining $\mathbf{A}' \in \mathbb{R}^{n \times k}$ as

$$ \mathbf{A}'_{ib} := \frac{\exp(\bar{\mathbf{A}}_{ib})}{\sum_{b'=1}^{k} \exp(\bar{\mathbf{A}}_{ib'})}, \quad \bar{\mathbf{A}}_{ib} := \frac{\mathbf{A}_{ib} - \mu_b}{\sigma_b}, \\ \mu_b := \frac{1}{n} \sum_{i=1}^{n} \mathbf{A}_{ib}, \quad \sigma_b := \sqrt{\frac{1}{n} \sum_{i=1}^{n} (\mathbf{A}_{ib} - \mu_b)^2} \quad (5) $$

and optimizing $\mathbf{A}$ for the objective

$$ S \approx \mathbf{A}'\mathbf{R}(\mathbf{A}')^T. \quad (6) $$
Note that after applying the row-wise softmax operation all entries of $\mathbf{A}'$ are non-negative.

To judge the quality of the approximation (6) we apply the Frobenius norm, which measures the difference between $\mathbf{S}$ and $\mathbf{A}'\mathbf{R}(\mathbf{A}')^T$. The final loss function we optimize our model for is therefore given by

$$ \mathcal{L}(\mathbf{S}, \mathbf{A}, \mathbf{R}) = \|\mathbf{S} - \mathbf{A}'\mathbf{R}(\mathbf{A}')^T\|_F^2 \quad (7) $$

$$ = \sum_{i=1}^{n} \sum_{j=1}^{n} \left( \mathbf{S}_{ij} - (\mathbf{A}'\mathbf{R}(\mathbf{A}')^T)_{ij} \right)^2 \quad (8) $$

with

$$ (\mathbf{A}'\mathbf{R}(\mathbf{A}')^T)_{ij} = \sum_{b=1}^{k} \sum_{c=1}^{k} \mathbf{A}'_{ib} \mathbf{R}_{bc} \mathbf{A}'_{jc} \quad (9) $$

and $\mathbf{A}'$ defined in (5).

To optimize the loss function we train both matrices using alternating gradient descent similar to [17]. Within each optimization step we apply

$$ \mathbf{A} \leftarrow \mathbf{A} - f_{\theta}(\nabla_{\mathbf{A}}, \eta_{\mathbf{A}}), \quad \text{where} \quad \nabla_{\mathbf{A}} = \frac{\partial \mathcal{L}(\mathbf{S}, \mathbf{A}, \mathbf{R})}{\partial \mathbf{A}} \quad (10) $$

$$ \mathbf{R} \leftarrow \mathbf{R} - f_{\theta}(\nabla_{\mathbf{R}}, \eta_{\mathbf{R}}), \quad \text{where} \quad \nabla_{\mathbf{R}} = \frac{\partial \mathcal{L}(\mathbf{S}, \mathbf{A}, \mathbf{R})}{\partial \mathbf{R}} \quad (11) $$

with $\eta_{\mathbf{A}}, \eta_{\mathbf{R}} > 0$ being individual learning rates for both matrices and $f_{\theta}(\cdot)$ representing an arbitrary gradient based update rule with additional hyperparameters $\theta$. For our experiments we employ automatic differentiation methods. For details on the implementation of the algorithm above refer to Section 4.2.

## 3.1 On Symmetry

The DEDICOM algorithm is able to factorize both symmetrical and asymmetrical matrices $\mathbf{S}$. For a given matrix $\mathbf{A}$, the symmetry of $\mathbf{R}$ dictates the symmetry of the product $\mathbf{A}\mathbf{R}\mathbf{A}^T$, since

$$ (\mathbf{A}\mathbf{R}\mathbf{A}^T)_{ij} = \sum_{b=1}^{k} \sum_{c=1}^{k} \mathbf{A}_{ib}\mathbf{R}_{bc}\mathbf{A}_{jc} = \sum_{b=1}^{k} \sum_{c=1}^{k} \mathbf{A}_{ib}\mathbf{R}_{cb}\mathbf{A}_{jc} \quad (12) $$

$$ = \sum_{c=1}^{k} \sum_{b=1}^{k} \mathbf{A}_{jc}\mathbf{R}_{cb}\mathbf{A}_{ib} = (\mathbf{A}\mathbf{R}\mathbf{A}^T)_{ji} \quad (13) $$

iff $\mathbf{R}_{cb} = \mathbf{R}_{bc}$ for all $b, c$. We therefore expect a symmetric matrix $\mathbf{S}$ to be decomposed into $\mathbf{A}\mathbf{R}\mathbf{A}^T$ with a symmetric $\mathbf{R}$, which is confirmed by our experiments. Factorizing a non-symmetric matrix leads to a non-symmetric $\mathbf{R}$, the asymmetric relation between items leads to asymmetric relations between the latent factors.
## 3.2 On Interpretability

We have

$$ S_{ij} \approx \sum_{b=1}^{k} \sum_{c=1}^{k} A_{ib} R_{bc} A_{jc}, \qquad (14) $$

i.e. we can estimate the probability of co-occurrence of two words $w_i$ and $w_j$ from the word embeddings $A_i$ and $A_j$ and the matrix $R$, where $A_i$ denotes the i-th row of $A$.

If we want to predict the co-occurrence between words $w_i$ and $w_j$ we consider the latent topics that make up the word embeddings $A_i$ and $A_j$, and sum up each component from $A_i$ with each component $A_j$ with respect to the relationship weights given in $R$.

Two words are likely to have a high co-occurrence if their word embeddings have larger weights in topics that are positively connected by the $R$ matrix. Likewise a negative entry $R_{b,c}$ makes it less likely for words with high weight in the topics $b$ and $c$ to occur in the same context. See Figure 2 for an illustrated example.

<figure><img src="image_3.png" /><figcaption>Figure 2: The affinity matrix $R$ describes the relationships between the latent factors. Illustrated here are two word embeddings, corresponding to the words $w_i$ and $w_j$. Darker shades represent larger values. In this example we predict a large co-occurrence at $S_{ii}$ and $S_{jj}$ because of the large weight on the diagonal of the $R$ matrix. We predict a low co-occurrence at $S_{ij}$ and $S_{ji}$ since the large weights on $A_{i1}$ and $A_{j3}$ interact with low weights on $R_{13}$ and $R_{31}$.</figcaption></figure>

Having an interpretable embedding model provides value beyond analysis of the affinity matrix of a single document. The worth of word embeddings is generally measured in their usefulness for downstream tasks. Given a prediction model based on word embeddings as one of the inputs, further analysis of the model behaviour is facilitated when latent input dimensions easily translate to semantic meaning.

In most word embedding models, the embedding vector of a single word is not particularly useful in itself. The information only lies in its relationship (i.e. closeness or cosine similarity) to other embedding vectors. For example, an analysis of the change of word embeddings and therefore the change of word meaning within a document corpus (for example a news article corpus) can only show how various words form different clusters or drift apart over time. Interpretability of latent dimensions would provide tools to also consider the development of single words within the given topics.
# 4 Experiments and Results

In the following section we describe our experimental setup in full detail<fnref n="4" /> and present our results on the simultaneous topic (relation) extraction and word embedding learning task. We compare these results against competing matrix factorization methods for topic modeling, namely NMF, LDA and SVD.

## 4.1 Data

To conduct our experiments we leverage a synthetically created text corpus, whose documents consist of triplets of individual English Wikipedia articles. The articles are retrieved as raw text via the official Wikipedia API using the **wikipedia-api** library. Always three articles a time get concatenated to form a new artificially generated text document. We differentiate between thematically similar (e.g. “Dolphin” and “Whale”) and thematically different articles (e.g. “Soccer” and “Donald Trump”). Each synthetic document is categorized into one of three classes: All underlying Wikipedia articles are thematically different, two articles are thematically similar and one is different, and all articles are thematically similar. Table 3 in the appendix shows this categorization and the overall setup of our generated documents.

On each document we apply the following textual preprocessing steps. First, the whole document gets lower-cased. Second, we tokenize the text making use of the word-tokenizer from the **nltk** library and remove common English stop words, including contractions such as “you’re” and “we’ll”. Lastly we clear the text from all remaining punctuation and delete digits and single characters.

As described in Section 3 we utilize our preprocessed document text to calculate a symmetric word co-occurrence matrix, which, after being transformed to a positive PMI matrix, functions as input and target matrix for the row-stochastic DEDICOM algorithm. To avoid any bias or prior information from the structure and order of the Wikipedia articles, we randomly shuffle the vocabulary before creating the co-occurrence matrix. When generating the matrix we only consider context words within a symmetrical window of size 7 around the base word. Like in [15], each context word only contributes $1/d$ to the total word pair count, given it is *d* words apart from the base word.

The next section sheds light upon the training process of row-stochastic DEDICOM and the above mentioned competing matrix factorization methods, which will be benchmarked against our results in Section 4.3 and in the appendix.

## 4.2 Training

As theoretically described in Section 3 we train row-stochastic DEDICOM with the alternating gradient descent paradigm utilizing automatic differentiation from the **PyTorch** library.

<footnote n="4">⁴ All results are completely reproducible based on the information in this section. Our Python implementation to reproduce the results is available on https://github.com/LarsHill/text-dedicom-paper.</footnote>
First, we initialize the factor matrices $A \in \mathbb{R}^{n \times k}$ and $R \in \mathbb{R}^{k \times k}$, by randomly sampling all elements from a uniform distribution centered around 1, $U(0, 2)$. Note that after applying the softmax operation on $A$ all rows of $A$ are stochastic. Therefore, scaling $R$ by

$$ \bar{s} := \frac{1}{n^2} \sum_{ij}^{n} S_{ij}, \qquad (15) $$

will result in the initial decomposition $A'R(A')^T$ yielding reconstructed elements in the range of $\bar{s}$, the element mean of the PPMI matrix $S$, and thus, speeding up convergence.

Second, $A$ and $R$ get iteratively updated employing the Adam optimizer [8] with constant individual learning rates of $\eta_A = 0.001$ and $\eta_R = 0.01$ and hyperparameters $\beta_1 = 0.9$, $\beta_2 = 0.999$ and $\epsilon = 1 \times 10^{-8}$. Both learning rates were identified through an exhaustive grid search. We train for `num_epochs` = 15,000 until convergence, where each epoch consists of an alternating gradient update with respect to $A$ and $R$. Algorithm 1 illustrates the just described training procedure.

<caption>**Algorithm 1** The row-stochastic DEDICOM algorithm</caption>

1: initialize $A, R \leftarrow U(0, 2) \cdot \bar{s}$ ▷ See Equation (15) for the definition of $\bar{s}$
2: initialize $\beta_1, \beta_2, \epsilon$ ▷ Adam algorithm hyperparameters
3: initialize $\eta_A, \eta_R$ ▷ Individual learning rates
4: **for** i in 1, ..., **num_epochs** **do**
5: &nbsp;&nbsp;&nbsp;&nbsp;Calculate loss $\mathcal{L} = \mathcal{L}(S, A, R)$ ▷ See Equation (8)
6: &nbsp;&nbsp;&nbsp;&nbsp;$A \leftarrow A$ - Adam$_{\beta_1, \beta_2, \epsilon}(\nabla_A, \eta_A)$, where $\nabla_A = \frac{\partial \mathcal{L}}{\partial A}$
7: &nbsp;&nbsp;&nbsp;&nbsp;$R \leftarrow R$ - Adam$_{\beta_1, \beta_2, \epsilon}(\nabla_R, \eta_R)$, where $\nabla_R = \frac{\partial \mathcal{L}}{\partial R}$
8: **return** $A'$ and $R$, where $A' = \text{row\_softmax}(\text{col\_norm}(A))$ ▷ See Equation (5)

We implement NMF, LDA and SVD using the `sklearn` library. In all cases the learnable factor matrices are initialized randomly and default hyperparameters are applied during training. For NMF the multiplicative update rule from [10] is utilized. Figure 3 shows the convergence behaviour of the row-stochastic DEDICOM training process and the final loss of NMF and SVD. Note that LDA optimizes a different loss function, which is why the calculated loss is not comparable and therefore excluded. We see that the final loss of DEDICOM locates just above the other losses, which is reasonable when considering the row stochasticity constraint on $A$ and the reduced parameter amount of $nk + k^2$ compared to NMF ($2nk$) and SVD ($2nk + k^2$).
<figure><img src="image_4.png" /><figcaption>Figure 3: Reconstruction loss development during training. The x-axis plots the number of epochs, the y-axis plots the corresponding reconstruction error for each matrix factorization method.</figcaption></figure>

## 4.3 Results

In the following, we present our results of training row-stochastic DEDICOM to simultaneously learn interpretable word embeddings and meaningful topic clusters and their relations. For compactness reasons we focus our main analysis on document id 3 in Table 3, “Soccer, Bee and Johnny Depp”, and set the number of latent topics to $k = 6$. We refer the interested reader to the appendix for results on other article combinations and comparison to other matrix factorization methods.<fnref n="5" />

In a first step, we evaluate the quality of the learned latent topics by assigning each word embedding $A'_i \in \mathbb{R}^{1 \times k}$ to the latent topic dimension that represents the maximum value in $A'_i$, i.e.

$$A'_i = [0.05 \ 0.03 \ 0.02 \ 0.14 \ 0.70 \ 0.06] \\ \argmax(A'_i) = 5,$$

and thus, $A'_i$ gets matched to Topic 5. Next, we decreasingly sort the words within each topic based on their matched topic probability. Table 1 shows the overall number of allocated words and the resulting top 10 words per topic together with each matched probability.

Indicated by the high assignment probabilities, one can see that columns 1, 2, 4, 5 and 6 represent distinct topics, which easily can be interpreted. Topic 1 and 4 are related to soccer, where 1 focuses on the game mechanics and 4 on the organisational and professional aspect of the game. Topic 2 and 6 clearly

<footnote n="5">⁵ We provide a large scale evaluation of all article combinations listed in Table 3, including different choices for $k$, as supplementary material at https://bit.ly/3cBxsGI.</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 4: (a) 2-dimensional representation of word embeddings $\mathbf{A}'$ colored by topic assignment. (b) 2-dimensional representation of word embeddings $\mathbf{A}'$ colored by original Wikipedia article assignment (words that occur in more than one article are excluded). (c) Colored heatmap of affinity matrix $\mathbf{R}$.</figcaption></figure>

<table><caption>Table 1: Each column lists the top 10 representative words per dimension of the basis matrix $\mathbf{A}'$.</caption><thead><tr><th></th><th>Topic 1<br>#619</th><th>Topic 2<br>#1238</th><th>Topic 3<br>#628</th><th>Topic 4<br>#595</th><th>Topic 5<br>#612</th><th>Topic 6<br>#389</th></tr></thead><tbody><tr><td>1</td><td>ball<br>(0.77)</td><td>film<br>(0.857)</td><td>salazar<br>(0.201)</td><td>cup<br>(0.792)</td><td>bees<br>(0.851)</td><td>heard<br>(0.738)</td></tr><tr><td>2</td><td>penalty<br>(0.708)</td><td>starred<br>(0.613)</td><td>geoffrey<br>(0.2)</td><td>football<br>(0.745)</td><td>species<br>(0.771)</td><td>court<br>(0.512)</td></tr><tr><td>3</td><td>may<br>(0.703)</td><td>role<br>(0.577)</td><td>rush<br>(0.2)</td><td>fifa<br>(0.731)</td><td>bee<br>(0.753)</td><td>depp<br>(0.505)</td></tr><tr><td>4</td><td>referee<br>(0.667)</td><td>series<br>(0.504)</td><td>brenton<br>(0.199)</td><td>world<br>(0.713)</td><td>pollen<br>(0.658)</td><td>divorce<br>(0.454)</td></tr><tr><td>5</td><td>goal<br>(0.66)</td><td>burton<br>(0.492)</td><td>hardwicke<br>(0.198)</td><td>national<br>(0.639)</td><td>honey<br>(0.602)</td><td>alcohol<br>(0.435)</td></tr><tr><td>6</td><td>team<br>(0.651)</td><td>character<br>(0.465)</td><td>thwaites<br>(0.198)</td><td>uefa<br>(0.623)</td><td>insects<br>(0.576)</td><td>paradis<br>(0.42)</td></tr><tr><td>7</td><td>players<br>(0.643)</td><td>played<br>(0.451)</td><td>catherine<br>(0.198)</td><td>continental<br>(0.582)</td><td>food<br>(0.536)</td><td>relationship<br>(0.419)</td></tr><tr><td>8</td><td>player<br>(0.639)</td><td>director<br>(0.45)</td><td>kaya<br>(0.198)</td><td>teams<br>(0.576)</td><td>nests<br>(0.529)</td><td>abuse<br>(0.41)</td></tr><tr><td>9</td><td>play<br>(0.606)</td><td>success<br>(0.438)</td><td>melfi<br>(0.198)</td><td>european<br>(0.57)</td><td>solitary<br>(0.513)</td><td>stating<br>(0.408)</td></tr><tr><td>10</td><td>game<br>(0.591)</td><td>jack<br>(0.434)</td><td>raimi<br>(0.198)</td><td>association<br>(0.563)</td><td>eusocial<br>(0.505)</td><td>stated<br>(0.402)</td></tr></tbody></table>
<table><caption>Table 2: For the most significant two words per topic, the four nearest neighbors based on cosine similarity are listed.</caption><thead><tr><th></th><th>Topic 1</th><th>Topic 2</th><th>Topic 3</th><th>Topic 4</th><th>Topic 5</th><th>Topic 6</th></tr></thead><tbody><tr><td>0</td><td>ball<br>(1.0)</td><td>film<br>(1.0)</td><td>salazar<br>(1.0)</td><td>cup<br>(1.0)</td><td>bees<br>(1.0)</td><td>heard<br>(1.0)</td></tr><tr><td>1</td><td>penalty starred<br>(0.994)</td><td>geoffrey<br>(0.978)</td><td>fifa<br>(1.0)</td><td>bee<br>(0.995)</td><td>court<br>(0.996)</td><td></td></tr><tr><td>2</td><td>referee role<br>(0.992)</td><td>rush<br>(0.964)</td><td>national species<br>(0.991)</td><td>divorce<br>(0.995)</td><td></td><td></td></tr><tr><td>3</td><td>may<br>(0.989)</td><td>burton<br>(0.937)</td><td>bardem<br>(1.0)</td><td>world<br>(0.988)</td><td>alcohol<br>(0.986)</td><td></td></tr><tr><td>4</td><td>goal series<br>(0.986)</td><td>brenton<br>(0.935)</td><td>uefa<br>(1.0)</td><td>honey<br>(0.987)</td><td>abuse<br>(0.971)</td><td></td></tr><tr><td>0</td><td>penalty starred<br>(1.0)</td><td>geoffrey<br>(1.0)</td><td>football<br>(1.0)</td><td>species<br>(1.0)</td><td>court<br>(1.0)</td><td></td></tr><tr><td>1</td><td>referee role<br>(0.999)</td><td>rush<br>(0.994)</td><td>fifa<br>(0.994)</td><td>bees<br>(0.995)</td><td>divorce<br>(0.995)</td><td></td></tr><tr><td>2</td><td>goal series<br>(0.998)</td><td>salazar<br>(0.985)</td><td>national bee<br>(0.983)</td><td>alcohol<br>(0.99)</td><td></td><td></td></tr><tr><td>3</td><td>player burton<br>(0.997)</td><td>brenton<br>(0.981)</td><td>cup<br>(0.983)</td><td>pollen<br>(0.99)</td><td>abuse<br>(0.982)</td><td></td></tr><tr><td>4</td><td>ball film<br>(0.994)</td><td>thwaites world<br>(0.978)</td><td>insects<br>(0.982)</td><td>settlement<br>(0.977)</td><td></td><td></td></tr></tbody></table>

refer to Johnny Depp, where 2 focuses on his acting career and 6 on his difficult relationship to Amber Heard. The fifth topic obviously relates to the insect “bee”. In contrast, Topic 3 does not allow for any interpretation and all assignment probabilities are significantly lower than for the other topics.

Further, we analyze the relations between the topics by visualizing the trained $R$ matrix as a heatmap (see Figure 4c).

One thing to note is the symmetry of $R$ which is a first indicator of a successful reconstruction, $A'R(A')^T$, (see Section 3.1). Also, the main diagonal elements are consistently blue (positive), which suggests a high distinction between the topics. Although not very strong one can still see a connection between Topic 2 and 6 indicated by the light blue entry $R_{26} = R_{62}$. While the suggested relation between Topic 1 and 4 is not clearly visible, element $R_{14} = R_{41}$ is the least negative one for Topic 1. In order to visualize the topic cluster quality we utilize UMAP (Uniform Manifold Approximation and Projection) [12] to map the $k$-dimensional word embeddings to a 2-dimensional space. Figure 4a illustrates this low-dimensional representation of $A'$, where each word is colored based on the above described word to topic assignment. In conjunction with Table 1 one can nicely see that Topic 2 and 6 (Johnny Depp) and Topic 1 and 4 (Soccer) are
close to each other. Hence, Figure 4a implicitly shows the learned topic relations
as well and arguably better than **R**.

As an additional benchmark, Figure 4b plots the same 2-dimensional representation, but now each word is colored based on the original Wikipedia article it belonged to. Words that occur in more than one article are not considered in this plot.

Directly comparing Figure 4b and 4a shows that row-stochastic DEDICOM does not only recover the original articles but also finds entirely new topics, which in this case represent subtopics of the articles. Let us emphasize that for all thematically similar article combinations, the found topics are usually not subtopics of a single article, but rather novel topics that might span across multiple Wikipedia articles (see for example Table 5 in the appendix). As mentioned at the top of this section, we are not only interested in learning meaningful topic clusters, but also in training interpretable word embeddings that capture semantic meaning.

Hence, we select within each topic the two most representative words and calculate the cosine similarity between their word embeddings and all other word embeddings stored in $\mathbf{A}'$. Table 2 shows the 4 nearest neighbors based on cosine similarity for the top 2 words in each topic. We observe a high thematical similarity between words with large cosine similarity, indicating the usefulness of the rows of $\mathbf{A}'$ as word embeddings.

In comparison to DEDICOM, other matrix factorization methods also provide a useful clustering of words into topics, with varying degree of granularity and clarity. However, the application of these methods as word embedding algorithms mostly fails on the word similarity task, with words close in cosine similarity seldom sharing the same thematical similarity we have seen in DEDICOM. This can be seen in Table 4, which shows for each method, NMF, LDA and SVD, the resulting word to topic clustering and the cosine nearest neighbors of the top two word embeddings per topic. While the individual topics extracted by NMF look very reasonable, its word embeddings do not seem to carry any semantic meaning based on cosine similarity; e.g. the four nearest neighbors of “ball” are “invoke”, “replaced”, “scores” and “subdivided”. A similar nonsensical picture can be observed for the other main topic words. LDA and SVD perform slightly better on the similar word task, although not all similar words appear to be sensible, e.g. “children”, “detective”, “crime”, “magazine” and “barber”. Also, some topics cannot be clearly defined due to mixed word assignments, e.g. Topic 4 for LDA and Topic 1 for SVD.

For a comprehensive overview of our results for other article combinations, we refer to Tables 5, 6, 7, 8 and Figures 5, 6 in the Appendix.

# 5 Conclusion and Outlook

We propose a row-stochasticity constrained version of the DEDICOM algorithm that is able to factorize the pointwise mutual information matrices of text documents into meaningful topic clusters all the while providing interpretable word
embeddings for each vocabulary item. Our study on semi-artificial data from
Wikipedia articles has shown that this method recovers the underlying struc-
ture of the text corpus and provides topics with thematic granularity, meaning
the extracted latent topics are more specific than a simple clustering of articles.
A comparison to related matrix factorization methods has shown that the com-
bination of top modeling and interpretable word embedding learning given by
our algorithm is unique in its class.

In future work we will expand on the idea of comparing topic relationships
between multiple documents, possibly over time, with individual co-occurrence
matrices resulting in stacked topic relationship matrices but shared word embed-
dings. Further extending this notion, we plan to utilize time series analysis to
discover temporal relations between extracted topics and to potentially identify
trends.

6 Acknowledgement

The authors of this work were supported by the Competence Center for Ma-
chine Learning Rhine Ruhr (ML2R) which is funded by the Federal Ministry
of Education and Research of Germany (grant no. 01|S18038C). We gratefully
acknowledge this support.

References

1. Andrzej, A.H., Cichocki, A., Dinh, T.V.: Nonnegative dedicom based on tensor decompositions for social networks exploration. Aust. J. Intell. Inf. Process. Syst. **12** (2010)

2. Bader, B.W., Harshman, R.A., Kolda, T.G.: Pattern analysis of directed graphs using dedicom: an application to enron email. (2006)

3. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. **3**, 993–1022 (2003)

4. Chew, P., Bader, B., Rozovskaya, A.: Using DEDICOM for completely unsupervised part-of-speech tagging. In: Proceedings of the Workshop on Unsupervised and Minimally Supervised Learning of Lexical Semantics. pp. 54–62. Association for Computational Linguistics, Boulder, Colorado, USA (2009)

5. Furnas, G.W., Deerwester, S., Dumais, S.T., Landauer, T.K., Harshman, R.A., Streeter, L.A., Lochbaum, K.E.: Information Retrieval Using A Singular Value Decomposition Model of Latent Semantic Structure. In: Proc. of ACM SIGIR (1988)

6. Harshman, R., Green, P., Wind, Y., Lundy, M.: A model for the analysis of asymmetric data in marketing research. Marketing Science **1**, 205–242 (1982)

7. Jolliffe, I.: Principal component analysis. John Wiley and Sons Ltd (2005)

8. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

9. Lebret, R., Collobert, R.: Word embeddings through hellinger PCA. In: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. pp. 482–490. Association for Computational Linguistics, Gothenburg, Sweden (2014)
10. Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Proceedings of the 13th International Conference on Neural Information Processing Systems. pp. 535–541. NIPS’00, MIT Press, Cambridge, MA, USA (2000)

11. Levy, O., Goldberg, Y.: Neural word embedding as implicit matrix factorization. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. pp. 2177–2185. NIPS’14, MIT Press, Cambridge, MA, USA (2014)

12. McInnes, L., Healy, J., Melville, J.: Umap: Uniform manifold approximation and projection for dimension reduction (2018)

13. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality (2013)

14. Nguyen, D.Q., Billingsley, R., Du, L., Johnson, M.: Improving topic models with latent feature word representations. Transactions of the Association for Computational Linguistics 3(0) (2015)

15. Pennington, J., Socher, R., Manning, C.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 1532–1543. Association for Computational Linguistics, Doha, Qatar (2014)

16. Sifa, R., Ojeda, C., Bauckhage, C.: User churn migration analysis with dedicom. In: Proceedings of the 9th ACM Conference on Recommender Systems. pp. 321–324. RecSys ’15, Association for Computing Machinery, New York, NY, USA (2015)

17. Sifa, R., Ojeda, C., Cvejoski, K., Bauckhage, C.: Interpretable matrix factorization with stochasticity constrained nonnegative dedicom (2018)

18. YongchangWang, Zhu, L.: Research and implementation of svd in machine learning. In: 2017 IEEE/ACIS 16th International Conference on Computer and Information Science (ICIS). pp. 471–475 (2017)
Appendix

<table><caption>Table 3: Overview of our semi-artificial dataset. Each synthetic sample consists of the corresponding Wikipedia articles 1 – 3. We differentiate between *different* articles, i.e. articles that have little thematical overlap (for example a person and a city, a fish and an insect or a ball game and a combat sport), and *similar* articles, i.e. articles with large thematical overlap (for example European countries, tech companies or aquatic animals). We group our dataset into different samples (3 articles that are pairwise different), similar samples (3 articles that are all similar) and mixed samples (2 similar articles, 1 different).</caption><thead><tr><th>ID</th><th>Selection Type</th><th>Article 1</th><th>Article 2</th><th>Article 3</th></tr></thead><tbody><tr><td>1</td><td></td><td>Donald Trump</td><td>New York City</td><td>Shark</td></tr><tr><td>2</td><td>different</td><td>Shark</td><td>Bee</td><td>Elephant</td></tr><tr><td>3</td><td></td><td>Soccer</td><td>Bee</td><td>Johnny Depp</td></tr><tr><td>4</td><td></td><td>Tennis</td><td>Dolphin</td><td>New York City</td></tr><tr><td>5</td><td></td><td>Donald Trump</td><td>New York City</td><td>Michael Bloomberg</td></tr><tr><td>6</td><td>mixed</td><td>Soccer</td><td>Tennis</td><td>Boxing</td></tr><tr><td>7</td><td></td><td>Brad Pitt</td><td>Leonardo Dicaprio</td><td>Rafael Nadal</td></tr><tr><td>8</td><td></td><td>Apple (company)</td><td>Google</td><td>Walmart</td></tr><tr><td>9</td><td></td><td>Shark</td><td>Dolphin</td><td>Whale</td></tr><tr><td>10</td><td>similar</td><td>Germany</td><td>Belgium</td><td>France</td></tr><tr><td>11</td><td></td><td>Soccer</td><td>Tennis</td><td>Rugby football</td></tr><tr><td>12</td><td></td><td>Apple (company)</td><td>Google</td><td>Amazon (company)</td></tr></tbody></table>
Articles: “Soccer”, “Bee”, “Johnny Depp”

<table><caption>Table 4: For each evaluated matrix factorization method we display the top 10 words for each topic and the 5 most similar words based on cosine similarity for the 2 top words from each topic.</caption><thead><tr><th></th><th>Topic 1</th><th>Topic 2</th><th>Topic 3</th><th>Topic 4</th><th>Topic 5</th><th>Topic 6</th></tr></thead><tbody><tr><td></td><td>#619</td><td>#1238</td><td>#628</td><td>#595</td><td>#612</td><td>#389</td></tr><tr><td></td><td>1 ball</td><td>bees</td><td>film</td><td>football</td><td>heard</td><td>album</td></tr><tr><td></td><td>2 may</td><td>species</td><td>starred</td><td>cup</td><td>depp</td><td>band</td></tr><tr><td></td><td>3 penalty</td><td>bee</td><td>role</td><td>world</td><td>court</td><td>guitar</td></tr><tr><td></td><td>4 referee</td><td>pollen</td><td>series</td><td>fifa</td><td>alcohol</td><td>vampires</td></tr><tr><td></td><td>5 players</td><td>honey</td><td>burton</td><td>national</td><td>relationship</td><td>rock</td></tr><tr><td></td><td>6 team</td><td>insects</td><td>character</td><td>association</td><td>stated</td><td>hollywood</td></tr><tr><td></td><td>7 goal</td><td>food</td><td>films</td><td>international</td><td>divorce</td><td>song</td></tr><tr><td></td><td>8 game</td><td>pests</td><td>box</td><td>women</td><td>abuse</td><td>released</td></tr><tr><td></td><td>9 player</td><td>solitary</td><td>office</td><td>teams</td><td>paradis</td><td>perry</td></tr><tr><td></td><td>10 play</td><td>eusocial</td><td>jack</td><td>uefa</td><td>stating</td><td>debut</td></tr><tr><td rowspan="10">NMF</td><td>0 ball</td><td>bees</td><td>film</td><td>football</td><td>heard</td><td>album</td></tr><tr><td>1 invoke</td><td>odors</td><td>burton directed</td><td>athenaeus</td><td>crew</td><td>jones</td></tr><tr><td>2 replaced</td><td>tufts</td><td>tone</td><td>paralympic</td><td>alleging</td><td>marilyn</td></tr><tr><td>3 scores</td><td>colour</td><td>landau</td><td>governing</td><td>oped</td><td>roots</td></tr><tr><td>4 subdivided</td><td>affected</td><td>brother</td><td>varieties</td><td>asserted</td><td>drums</td></tr><tr><td>0 may</td><td>species</td><td>starred</td><td>cup</td><td>depp</td><td>band</td></tr><tr><td>1 yd</td><td>niko</td><td>shared</td><td>inaugurated</td><td>refer</td><td>heroes</td></tr><tr><td>2 ineffectiveness</td><td>commercially</td><td>whitaker</td><td>confederation</td><td>york</td><td>bowie</td></tr><tr><td>3 tactical</td><td>microbiota</td><td>eccentric</td><td>gold</td><td>leaders</td><td>debut</td></tr><tr><td>4 slower</td><td>strategies</td><td>befriends</td><td>headquarters</td><td>nonindian</td><td>solo</td></tr><tr><td></td><td>#577</td><td>#728</td><td>#692</td><td>#607</td><td>#663</td><td>#814</td></tr><tr><td></td><td>1 film</td><td>football</td><td>depp</td><td>penalty</td><td>bees</td><td>species</td></tr><tr><td></td><td>2 series</td><td>women</td><td>children</td><td>heard</td><td>flowers</td><td>workers</td></tr><tr><td></td><td>3 man</td><td>association</td><td>life</td><td>ball</td><td>bee</td><td>solitary</td></tr><tr><td></td><td>4 played</td><td>fifa</td><td>role</td><td>direct</td><td>honey</td><td>players</td></tr><tr><td></td><td>5 pirates</td><td>teams</td><td>starred</td><td>referee</td><td>pollen</td><td>colonies</td></tr><tr><td></td><td>6 character</td><td>games</td><td>alongside</td><td>red</td><td>food</td><td>eusocial</td></tr><tr><td></td><td>7 along</td><td>world</td><td>actor</td><td>time</td><td>increased</td><td>nest</td></tr><tr><td></td><td>8 cast</td><td>cup</td><td>stated</td><td>goal</td><td>pollination</td><td>may</td></tr><tr><td></td><td>9 also</td><td>game</td><td>burton</td><td>scored</td><td>times</td><td>size</td></tr><tr><td></td><td>10 hollow</td><td>international</td><td>playing</td><td>player</td><td>larvae</td><td>egg</td></tr><tr><td rowspan="10">LDA</td><td>0 film</td><td>football</td><td>depp</td><td>penalty</td><td>bees</td><td>species</td></tr><tr><td>1 charlie</td><td>cup</td><td>critical</td><td>extra</td><td>bee</td><td>social</td></tr><tr><td>2 near</td><td>canada</td><td>february</td><td>kicks</td><td>insects</td><td>chosen</td></tr><tr><td>3 thinking</td><td>zealand</td><td>script</td><td>inner</td><td>authors</td><td>females</td></tr><tr><td>4 shadows</td><td>activities</td><td>song</td><td>moving</td><td>hives</td><td>subspecies</td></tr><tr><td>0 series</td><td>women</td><td>children</td><td>heard</td><td>flowers</td><td>workers</td></tr><tr><td>1 crybaby</td><td>fifa</td><td>detective</td><td>allison</td><td>always</td><td>carcases</td></tr><tr><td>2 waters</td><td>opera</td><td>crime</td><td>serious</td><td>eusociality</td><td>lived</td></tr><tr><td>3 sang</td><td>exceeding</td><td>magazine</td><td>allergic</td><td>varroa</td><td>provisioned</td></tr><tr><td>4 cast</td><td>cuju</td><td>barber</td><td>cost</td><td>wing</td><td>cuckoo</td></tr><tr><td></td><td>#1228</td><td>#797</td><td>#628</td><td>#369</td><td>#622</td><td>#437</td></tr><tr><td></td><td>1 bees</td><td>depp</td><td>game</td><td>cup</td><td>heard</td><td>beekeeping</td></tr><tr><td></td><td>2 also</td><td>film</td><td>ball</td><td>football</td><td>court</td><td>increased</td></tr><tr><td></td><td>3 bee</td><td>starred</td><td>team</td><td>fifa</td><td>divorce</td><td>honey</td></tr><tr><td></td><td>4 species</td><td>role</td><td>players</td><td>world</td><td>stating</td><td>described</td></tr><tr><td></td><td>5 played</td><td>series</td><td>penalty</td><td>european</td><td>alcohol</td><td>use</td></tr><tr><td></td><td>6 time</td><td>burton</td><td>play</td><td>uefa</td><td>paradis</td><td>wild</td></tr><tr><td></td><td>7 one</td><td>character</td><td>may</td><td>national</td><td>documents</td><td>varroa</td></tr><tr><td></td><td>8 first</td><td>actor</td><td>referee</td><td>Europe</td><td>abuse</td><td>mites</td></tr><tr><td></td><td>9 two</td><td>released</td><td>competitions</td><td>continental</td><td>settlement</td><td>colony</td></tr><tr><td></td><td>10 pollen</td><td>release</td><td>laws</td><td>confederation</td><td>sued</td><td>flowers</td></tr><tr><td rowspan="10">SVD</td><td>0 bees</td><td>depp</td><td>game</td><td>cup</td><td>heard</td><td>beekeeping</td></tr><tr><td>1 bee</td><td>iii</td><td>correct</td><td>continental</td><td>alleging</td><td>varroa</td></tr><tr><td>2 develops</td><td>racism</td><td>abandoned</td><td>contested</td><td>attempting</td><td>animals</td></tr><tr><td>3 studied</td><td>appropriation</td><td>maximum</td><td>confederations</td><td>finalized</td><td>mites</td></tr><tr><td>4 crops</td><td>march</td><td>clear</td><td>conmebol</td><td>submitted</td><td>plato</td></tr><tr><td>0 also</td><td>film</td><td>ball</td><td>football</td><td>court</td><td>increased</td></tr><tr><td>1 although</td><td>waters</td><td>finely</td><td>er</td><td>declaration</td><td>usage</td></tr><tr><td>2 told</td><td>robinson</td><td>poised</td><td>suffix</td><td>issued</td><td>farmers</td></tr><tr><td>3 chosen</td><td>scott</td><td>worn</td><td>word</td><td>restraining</td><td>mentioned</td></tr><tr><td>4 stars</td><td>costars</td><td>manner</td><td>appended</td><td>verbally</td><td>aeneid</td></tr></tbody></table>
<table><caption>Articles: “Dolphin”, “Shark”, “Whale”</caption><thead><tr><th></th><th>Topic 1<br>#460</th><th>Topic 2<br>#665</th><th>Topic 3<br>#801</th><th>Topic 4<br>#753</th><th>Topic 5<br>#854</th><th>Topic 6<br>#721</th></tr></thead><tbody><tr><td>1</td><td>shark<br>(0.665)</td><td>calf<br>(0.428)</td><td>ship<br>(0.459)</td><td>conservation<br>(0.334)</td><td>water<br>(0.416)</td><td>dolphin<br>(0.691)</td></tr><tr><td>2</td><td>sharks<br>(0.645)</td><td>months<br>(0.407)</td><td>became<br>(0.448)</td><td>countries<br>(0.312)</td><td>similar<br>(0.374)</td><td>dolphins<br>(0.655)</td></tr><tr><td>3</td><td>fins<br>(0.487)</td><td>calves<br>(0.407)</td><td>poseidon<br>(0.44)</td><td>government<br>(0.309)</td><td>tissue<br>(0.373)</td><td>captivity<br>(0.549)</td></tr><tr><td>4</td><td>killed<br>(0.454)</td><td>females<br>(0.399)</td><td>riding<br>(0.426)</td><td>wales<br>(0.304)</td><td>body<br>(0.365)</td><td>wild<br>(0.467)</td></tr><tr><td>5</td><td>million<br>(0.451)</td><td>blubber<br>(0.374)</td><td>dionysus<br>(0.422)</td><td>bycatch<br>(0.29)</td><td>swimming<br>(0.357)</td><td>behavior<br>(0.461)</td></tr><tr><td>6</td><td>fish<br>(0.448)</td><td>young<br>(0.37)</td><td>ancient<br>(0.42)</td><td>cancelled<br>(0.288)</td><td>blood<br>(0.346)</td><td>bottlenose<br>(0.453)</td></tr><tr><td>7</td><td>international<br>(0.442)</td><td>sperm<br>(0.356)</td><td>deity<br>(0.412)</td><td>eastern<br>(0.287)</td><td>surface<br>(0.344)</td><td>sometimes<br>(0.449)</td></tr><tr><td>8</td><td>fin<br>(0.421)</td><td>born<br>(0.355)</td><td>ago<br>(0.398)</td><td>policy<br>(0.286)</td><td>oxygen<br>(0.34)</td><td>human<br>(0.421)</td></tr><tr><td>9</td><td>fishing<br>(0.405)</td><td>feed<br>(0.349)</td><td>melicertes<br>(0.395)</td><td>control<br>(0.285)</td><td>system<br>(0.336)</td><td>less<br>(0.42)</td></tr><tr><td>10</td><td>teeth<br>(0.398)</td><td>mysticetes<br>(0.341)</td><td>greeks<br>(0.394)</td><td>imminent<br>(0.282)</td><td>swim<br>(0.336)</td><td>various<br>(0.418)</td></tr><tr><td>0</td><td>shark<br>(1.0)</td><td>calf<br>(1.0)</td><td>ship<br>(1.0)</td><td>conservation<br>(1.0)</td><td>water<br>(1.0)</td><td>dolphin<br>(1.0)</td></tr><tr><td>2</td><td>sharks<br>(0.981)</td><td>calves<br>(0.978)</td><td>dionysus<br>(0.995)</td><td>south<br>(0.981)</td><td>prey<br>(0.964)</td><td>dolphins<br>(0.925)</td></tr><tr><td>3</td><td>fins<br>(0.958)</td><td>females<br>(0.976)</td><td>riding<br>(0.992)</td><td>states<br>(0.981)</td><td>swimming<br>(0.959)</td><td>sometimes<br>(0.909)</td></tr><tr><td>4</td><td>killed<br>(0.929)</td><td>months<br>(0.955)</td><td>deity<br>(0.992)</td><td>united<br>(0.978)</td><td>allows<br>(0.957)</td><td>another<br>(0.904)</td></tr><tr><td>5</td><td>fishing<br>(0.916)</td><td>young<br>(0.948)</td><td>poseidon<br>(0.987)</td><td>endangered<br>(0.976)</td><td>swim<br>(0.947)</td><td>bottlenose<br>(0.903)</td></tr><tr><td>0</td><td>sharks<br>(1.0)</td><td>months<br>(1.0)</td><td>became<br>(1.0)</td><td>countries<br>(1.0)</td><td>similar<br>(1.0)</td><td>dolphins<br>(1.0)</td></tr><tr><td>2</td><td>shark<br>(0.981)</td><td>born<br>(0.992)</td><td>old<br>(0.953)</td><td>eastern<br>(0.991)</td><td>surface<br>(0.992)</td><td>behavior<br>(0.956)</td></tr><tr><td>3</td><td>fins<br>(0.936)</td><td>young<br>(0.992)</td><td>later<br>(0.946)</td><td>united<br>(0.989)</td><td>brain<br>(0.97)</td><td>sometimes<br>(0.945)</td></tr><tr><td>4</td><td>tiger<br>(0.894)</td><td>sperm<br>(0.985)</td><td>ago<br>(0.939)</td><td>caught<br>(0.987)</td><td>sound<br>(0.968)</td><td>various<br>(0.943)</td></tr><tr><td>5</td><td>killed<br>(0.887)</td><td>calves<br>(0.984)</td><td>modern<br>(0.937)</td><td>south<br>(0.979)</td><td>object<br>(0.965)</td><td>less<br>(0.937)</td></tr></tbody></table>

<figure><img src="image_6.png" /><figcaption>Figure 5: (a) 2-dimensional representation of word embeddings $A'$ colored by topic assignment. (b) 2-dimensional representation of word embeddings $A'$ colored by original Wikipedia article assignment (words that occur in more than one article are excluded). (c) Colored heatmap of affinity matrix $R$.</figcaption></figure>
Articles: “Dolphin”, “Shark”, “Whale”

<table><caption>Table 6: For each evaluated matrix factorization method we display the top 10 words for each topic and the 5 most similar words based on cosine similarity for the 2 top words from each topic.</caption><thead><tr><th></th><th>Topic 1</th><th>Topic 2</th><th>Topic 3</th><th>Topic 4</th><th>Topic 5</th><th>Topic 6</th></tr></thead><tbody><tr><td rowspan="10">NMF</td><td>#492</td><td>#907</td><td>#452</td><td>#854</td><td>#911</td><td>#638</td></tr><tr><td>1 blood</td><td>international</td><td>evidence</td><td>sonar</td><td>ago</td><td>calf</td></tr><tr><td>2 body</td><td>killed</td><td>selfawareness</td><td>may</td><td>teeth</td><td>young</td></tr><tr><td>3 heart</td><td>states</td><td>ship</td><td>surface</td><td>million</td><td>females</td></tr><tr><td>4 gills</td><td>conservation</td><td>dionysus</td><td>clicks</td><td>mysticetes</td><td>captivity</td></tr><tr><td>5 bony</td><td>new</td><td>came</td><td>prey</td><td>whales</td><td>calves</td></tr><tr><td>6 oxygen</td><td>united</td><td>another</td><td>use</td><td>years</td><td>months</td></tr><tr><td>7 organs</td><td>shark</td><td>important</td><td>underwater</td><td>baleen</td><td>born</td></tr><tr><td>8 tissue</td><td>world</td><td>poseidon</td><td>sounds</td><td>cetaceans</td><td>species</td></tr><tr><td>9 water</td><td>endangered</td><td>mark</td><td>known</td><td>modern</td><td>male</td></tr><tr><td>10 via</td><td>islands</td><td>riding</td><td>similar</td><td>extinct</td><td>female</td></tr><tr><td rowspan="10">LDA</td><td>0 blood</td><td>international</td><td>evidence</td><td>sonar</td><td>ago</td><td>calf</td></tr><tr><td>1 travels</td><td>proposal</td><td>flaws</td><td>poisoned</td><td>consist</td><td>uninformed</td></tr><tr><td>2 enters</td><td>lipotidae</td><td>methodological</td><td>signals</td><td>specialize</td><td>primary</td></tr><tr><td>3 vibration</td><td>banned</td><td>nictating</td><td>—</td><td>legs</td><td>born</td></tr><tr><td>4 tolerant</td><td>iniidae</td><td>wake</td><td>emitted</td><td>closest</td><td>leaner</td></tr><tr><td>0 body</td><td>killed</td><td>selfawareness</td><td>may</td><td>teeth</td><td>young</td></tr><tr><td>1 crystal</td><td>law</td><td>legendary</td><td>individuals</td><td>fuel</td><td>brood</td></tr><tr><td>2 blocks</td><td>consumers</td><td>humankind</td><td>helping</td><td>lamp</td><td>lacking</td></tr><tr><td>3 modified</td><td>pontoporiidae</td><td>助手</td><td>waste</td><td>filterfeeding</td><td>accurate</td></tr><tr><td>4 slits</td><td>org</td><td>performing</td><td>depression</td><td>krill</td><td>consistency</td></tr><tr><td rowspan="10">SVD</td><td>#650</td><td>#785</td><td>#695</td><td>#815</td><td>#635</td><td>#674</td></tr><tr><td>1 killed</td><td>teeth</td><td>head</td><td>species</td><td>meat</td><td>air</td></tr><tr><td>2 system</td><td>baleen</td><td>fish</td><td>male</td><td>whale</td><td>using</td></tr><tr><td>3 endangered</td><td>mysticetes</td><td>dolphin</td><td>females</td><td>ft</td><td>causing</td></tr><tr><td>4 often</td><td>ago</td><td>fin</td><td>whales</td><td>fisheries</td><td>currents</td></tr><tr><td>5 close</td><td>jaw</td><td>eyes</td><td>sometimes</td><td>also</td><td>sounds</td></tr><tr><td>6 sharks</td><td>family</td><td>fat</td><td>captivity</td><td>ocean</td><td>groups</td></tr><tr><td>7 countries</td><td>water</td><td>navy</td><td>young</td><td>threats</td><td>sound</td></tr><tr><td>8 since</td><td>includes</td><td>popular</td><td>shark</td><td>children</td><td>research</td></tr><tr><td>9 called</td><td>allow</td><td>tissue</td><td>female</td><td>population</td><td>clicks</td></tr><tr><td>10 vessels</td><td>greater</td><td>tail</td><td>wild</td><td>bottom</td><td>burst</td></tr><tr><td rowspan="10">LDA</td><td>0 killed</td><td>teeth</td><td>head</td><td>species</td><td>meat</td><td>air</td></tr><tr><td>1 postures</td><td>dense</td><td>underside</td><td>along</td><td>porbeagle</td><td>australis</td></tr><tr><td>2 dolphinariums</td><td>cetacea</td><td>grooves</td><td>another</td><td>source</td><td>submerged</td></tr><tr><td>3 town</td><td>tourism</td><td>eyesight</td><td>long</td><td>activities</td><td>melbourne</td></tr><tr><td>4 onethird</td><td>planktonfeeders</td><td>osmoregulation</td><td>sleep</td><td>comparable</td><td>spear</td></tr><tr><td>0 system</td><td>baleen</td><td>fish</td><td>male</td><td>whale</td><td>using</td></tr><tr><td>1 dominate</td><td>mysticetes</td><td>mostly</td><td>females</td><td>live</td><td>communication</td></tr><tr><td>2 close</td><td>distinguishing</td><td>swim</td><td>aorta</td><td>human</td><td>become</td></tr><tr><td>3 controversy</td><td>unique</td><td>due</td><td>female</td><td>cold</td><td>associated</td></tr><tr><td>4 agree</td><td>remove</td><td>whole</td><td>position</td><td>parts</td><td>mirror</td></tr><tr><td rowspan="10">SVD</td><td>#1486</td><td>#544</td><td>#605</td><td>#469</td><td>#539</td><td>#611</td></tr><tr><td>1 dolphins</td><td>water</td><td>shark</td><td>million</td><td>poseidon</td><td>dolphin</td></tr><tr><td>2 species</td><td>body</td><td>sharks</td><td>years</td><td>became</td><td>meat</td></tr><tr><td>3 whales</td><td>tail</td><td>fins</td><td>ago</td><td>ship</td><td>family</td></tr><tr><td>4 fish</td><td>teeth</td><td>international</td><td>whale</td><td>riding</td><td>river</td></tr><tr><td>5 also</td><td>flippers</td><td>killed</td><td>two</td><td>evidence</td><td>similar</td></tr><tr><td>6 large</td><td>tissue</td><td>fishing</td><td>calf</td><td>melicertes</td><td>extinct</td></tr><tr><td>7 may</td><td>allows</td><td>fin</td><td>mya</td><td>deity</td><td>called</td></tr><tr><td>8 one</td><td>air</td><td>law</td><td>later</td><td>ino</td><td>used</td></tr><tr><td>9 animals</td><td>feed</td><td>new</td><td>months</td><td>came</td><td>islands</td></tr><tr><td>10 use</td><td>bony</td><td>conservation</td><td>mysticetes</td><td>made</td><td>genus</td></tr><tr><td rowspan="10">SVD</td><td>0 dolphins</td><td>water</td><td>shark</td><td>million</td><td>poseidon</td><td>dolphin</td></tr><tr><td>1 various</td><td>vertical</td><td>corpse</td><td>approximately</td><td>games</td><td>depicted</td></tr><tr><td>2 finding</td><td>unlike</td><td>stocks</td><td>assigned</td><td>phalanthus</td><td>makara</td></tr><tr><td>3 military</td><td>chew</td><td>galea</td><td>hybodonts</td><td>statue</td><td>capensis</td></tr><tr><td>4 selfmade</td><td>lack</td><td>galeomorphii</td><td>appeared</td><td>isthmian</td><td>goddess</td></tr><tr><td>0 species</td><td>body</td><td>sharks</td><td>years</td><td>became</td><td>meat</td></tr><tr><td>1 herd</td><td>heart</td><td>mostly</td><td>acanthodians</td><td>pirates</td><td>contaminated</td></tr><tr><td>2 reproduction</td><td>resisting</td><td>fda</td><td>spent</td><td>elder</td><td>harpoon</td></tr><tr><td>3 afford</td><td>fit</td><td>lists</td><td>stretching</td><td>mistook</td><td>practitioner</td></tr><tr><td>4 maturity</td><td>posterior</td><td>carcharias</td><td>informal</td><td>wealthy</td><td>pcbs</td></tr></tbody></table>
<table><caption>Articles: “Soccer”, “Tennis”, “Rugby”</caption><thead><tr><th></th><th>Topic 1<br>#539</th><th>Topic 2<br>#302</th><th>Topic 3<br>#563</th><th>Topic 4<br>#635</th><th>Topic 5<br>#650</th><th>Topic 6<br>#530</th></tr></thead><tbody><tr><td>1</td><td>may<br>(0.599)</td><td>leads<br>(0.212)</td><td>tournaments<br>(0.588)</td><td>greatest<br>(0.572)</td><td>football<br>(0.553)</td><td>net<br>(0.644)</td></tr><tr><td>2</td><td>penalty<br>(0.576)</td><td>sole<br>(0.205)</td><td>tournament<br>(0.517)</td><td>tennis<br>(0.497)</td><td>rugby<br>(0.542)</td><td>shot<br>(0.629)</td></tr><tr><td>3</td><td>referee<br>(0.564)</td><td>competes<br>(0.205)</td><td>events<br>(0.509)</td><td>female<br>(0.44)</td><td>south<br>(0.484)</td><td>stance<br>(0.553)</td></tr><tr><td>4</td><td>team<br>(0.517)</td><td>extending<br>(0.204)</td><td>prize<br>(0.501)</td><td>ever<br>(0.433)</td><td>union<br>(0.47)</td><td>stroke<br>(0.543)</td></tr><tr><td>5</td><td>goal<br>(0.502)</td><td>fixing<br>(0.203)</td><td>tour<br>(0.497)</td><td>navratilova<br>(0.405)</td><td>wales<br>(0.459)</td><td>serve<br>(0.537)</td></tr><tr><td>6</td><td>kick<br>(0.459)</td><td>triggered<br>(0.203)</td><td>money<br>(0.488)</td><td>modern<br>(0.401)</td><td>national<br>(0.446)</td><td>rotation<br>(0.513)</td></tr><tr><td>7</td><td>play<br>(0.455)</td><td>bleeding<br>(0.202)</td><td>cup<br>(0.486)</td><td>best<br>(0.4)</td><td>england<br>(0.438)</td><td>backhand<br>(0.508)</td></tr><tr><td>8</td><td>ball<br>(0.452)</td><td>fraud<br>(0.202)</td><td>world<br>(0.467)</td><td>wingfield<br>(0.394)</td><td>new<br>(0.416)</td><td>hit<br>(0.507)</td></tr><tr><td>9</td><td>offence<br>(0.444)</td><td>inflammation<br>(0.202)</td><td>atp<br>(0.464)</td><td>sports<br>(0.39)</td><td>europe<br>(0.406)</td><td>forehand<br>(0.499)</td></tr><tr><td>10</td><td>foul<br>(0.443)</td><td>conditions<br>(0.201)</td><td>men<br>(0.463)</td><td>williams<br>(0.389)</td><td>states<br>(0.404)</td><td>torso<br>(0.487)</td></tr><tr><td>0</td><td>may<br>(1.0)</td><td>leads<br>(1.0)</td><td>tournaments<br>(1.0)</td><td>greatest<br>(1.0)</td><td>football<br>(1.0)</td><td>net<br>(1.0)</td></tr><tr><td>2</td><td>goal<br>(0.98)</td><td>tiredness<br>(1.0)</td><td>events<br>(0.992)</td><td>female<br>(0.98)</td><td>union<br>(0.98)</td><td>shot<br>(0.994)</td></tr><tr><td>3</td><td>play<br>(0.959)</td><td>ineffectiveness<br>(1.0)</td><td>tour<br>(0.989)</td><td>ever<br>(0.971)</td><td>rugby<br>(0.979)</td><td>serve<br>(0.987)</td></tr><tr><td>4</td><td>penalty<br>(0.954)</td><td>recommences<br>(1.0)</td><td>money<br>(0.986)</td><td>navratilova<br>(0.967)</td><td>association<br>(0.96)</td><td>hit<br>(0.984)</td></tr><tr><td>5</td><td>team<br>(0.953)</td><td>mandated<br>(1.0)</td><td>prize<br>(0.985)</td><td>tennis<br>(0.962)</td><td>england<br>(0.958)</td><td>stance<br>(0.955)</td></tr><tr><td>0</td><td>penalty<br>(1.0)</td><td>sole<br>(1.0)</td><td>tournament<br>(1.0)</td><td>tennis<br>(1.0)</td><td>rugby<br>(1.0)</td><td>shot<br>(1.0)</td></tr><tr><td>2</td><td>referee<br>(0.985)</td><td>discretion<br>(1.0)</td><td>events<br>(0.98)</td><td>greatest<br>(0.962)</td><td>football<br>(0.979)</td><td>net<br>(0.994)</td></tr><tr><td>3</td><td>kick<br>(0.985)</td><td>synonym<br>(1.0)</td><td>event<br>(0.978)</td><td>female<br>(0.953)</td><td>union<br>(0.975)</td><td>serve<br>(0.987)</td></tr><tr><td>4</td><td>offence<br>(0.982)</td><td>violated<br>(1.0)</td><td>atp<br>(0.974)</td><td>year<br>(0.951)</td><td>england<br>(0.961)</td><td>hit<br>(0.983)</td></tr><tr><td>5</td><td>foul<br>(0.982)</td><td>layout<br>(1.0)</td><td>money<br>(0.966)</td><td>navratilova<br>(0.949)</td><td>wales<br>(0.949)</td><td>stance<br>(0.98)</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Figure 6: (a) 2-dimensional representation of word embeddings $A'$ colored by topic assignment. (b) 2-dimensional representation of word embeddings $A'$ colored by original Wikipedia article assignment (words that occur in more than one article are excluded). (c) Colored heatmap of affinity matrix $R$.</figcaption></figure>
Articles: “Soccer”, “Tennis”, “Rugby”

<table><caption>Table 8: For each evaluated matrix factorization method we display the top 10 words for each topic and the 5 most similar words based on cosine similarity for the 2 top words from each topic.</caption><thead><tr><th></th><th>Topic 1</th><th>Topic 2</th><th>Topic 3</th><th>Topic 4</th><th>Topic 5</th><th>Topic 6</th></tr></thead><tbody><tr><td></td><td>#511<br>net</td><td>#453<br>referee</td><td>#575<br>national</td><td>#657<br>tournaments</td><td>#402<br>rackets</td><td>#621<br>rules</td></tr><tr><td></td><td>shot</td><td>penalty</td><td>south</td><td>doubles</td><td>balls</td><td>wingfield</td></tr><tr><td></td><td>serve</td><td>may</td><td>football</td><td>singles</td><td>made</td><td>december</td></tr><tr><td></td><td>hit</td><td>kick</td><td>cup</td><td>events</td><td>size</td><td>game</td></tr><tr><td></td><td>stance</td><td>card</td><td>Europe</td><td>tour</td><td>must</td><td>sports</td></tr><tr><td></td><td>stroke</td><td>listed</td><td>fifa</td><td>prize</td><td>strings</td><td>lawn</td></tr><tr><td></td><td>backhand</td><td>foul</td><td>union</td><td>money</td><td>standard</td><td>modern</td></tr><tr><td></td><td>ball</td><td>misconduct</td><td>wales</td><td>atp</td><td>synthetic</td><td>greek</td></tr><tr><td></td><td>server</td><td>red</td><td>africa</td><td>men</td><td>leather</td><td>fa</td></tr><tr><td></td><td>service</td><td>offence</td><td>new</td><td>grand</td><td>width</td><td>first</td></tr><tr><td rowspan="5">NMF</td><td>0 net</td><td>referee</td><td>national</td><td>tournaments</td><td>rackets</td><td>rules</td></tr><tr><td>1 defensive</td><td>retaken</td><td>serbia</td><td>bruno</td><td>pressurisation</td><td>collection</td></tr><tr><td>2 closer</td><td>interference</td><td>gold</td><td>woodies</td><td>become</td><td>hourglass</td></tr><tr><td>3 somewhere</td><td>dismissed</td><td>north</td><td>eliminated</td><td>equivalents</td><td>unhappy</td></tr><tr><td>4 center</td><td>fully</td><td>headquarters</td><td>soares</td><td>size</td><td>originated</td></tr><tr><td></td><td>0 shot</td><td>penalty</td><td>south</td><td>doubles</td><td>balls</td><td>wingfield</td></tr><tr><td></td><td>1 rotated</td><td>prior</td><td>asian</td><td>combining</td><td>express</td><td>experimenting</td></tr><tr><td></td><td>2 execute</td><td>yellow</td><td>Argentina</td><td>becker</td><td>oz</td><td>llanelidan</td></tr><tr><td></td><td>3 strive</td><td>duration</td><td>la</td><td>exclusively</td><td>bladder</td><td>attended</td></tr><tr><td></td><td>4 curve</td><td>primary</td><td>kong</td><td>woodbridge</td><td>length</td><td>antiphanes</td></tr><tr><td></td><td>#413<br>used</td><td>#518<br>net</td><td>#395<br>wimbledon</td><td>#776<br>world</td><td>#616<br>penalty</td><td>#501<br>clubs</td></tr><tr><td></td><td>1 forehand</td><td>ball</td><td>episkyros</td><td>cup</td><td>score</td><td>rugby</td></tr><tr><td></td><td>2 use</td><td>serve</td><td>occurs</td><td>tournaments</td><td>goal</td><td>schools</td></tr><tr><td></td><td>3 large</td><td>shot</td><td>grass</td><td>football</td><td>team</td><td>navratilova</td></tr><tr><td></td><td>4 notable</td><td>opponent</td><td>roman</td><td>fifa</td><td>end</td><td>forms</td></tr><tr><td></td><td>5 also</td><td>hit</td><td>bc</td><td>national</td><td>players</td><td>playing</td></tr><tr><td></td><td>6 western</td><td>lines</td><td>occur</td><td>international</td><td>match</td><td>sport</td></tr><tr><td></td><td>7 twohanded</td><td>server</td><td>ad</td><td>europe</td><td>goals</td><td>greatest</td></tr><tr><td></td><td>8 doubles</td><td>service</td><td>island</td><td>tournament</td><td>time</td><td>union</td></tr><tr><td></td><td>9 injury</td><td>may</td><td>believed</td><td>states</td><td>scored</td><td>war</td></tr><tr><td rowspan="6">LDA</td><td>0 used</td><td>net</td><td>wimbledon</td><td>world</td><td>penalty</td><td>clubs</td></tr><tr><td>1 seconds</td><td>mistaken</td><td>result</td><td>british</td><td>measure</td><td>sees</td></tr><tr><td>2 restrictions</td><td>diagonal</td><td>determined</td><td>cancelled</td><td>crossed</td><td>papua</td></tr><tr><td>3 although</td><td>hollow</td><td>exists</td><td>combined</td><td>requiring</td><td>admittance</td></tr><tr><td>4 use</td><td>perpendicular</td><td>win</td><td>wii</td><td>teammate</td><td>forces</td></tr><tr><td>0 forehand</td><td>ball</td><td>episkyros</td><td>cup</td><td>score</td><td>rugby</td></tr><tr><td></td><td>1 twohanded</td><td>long</td><td>roman</td><td>multiple</td><td>penalty</td><td>union</td></tr><tr><td></td><td>2 grips</td><td>deuce</td><td>bc</td><td>inline</td><td>bar</td><td>public</td></tr><tr><td></td><td>3 facetiously</td><td>position</td><td>island</td><td>fifa</td><td>fouled</td><td>took</td></tr><tr><td></td><td>4 woodbridge</td><td>allows</td><td>believed</td><td>manufactured</td><td>hour</td><td>published</td></tr><tr><td></td><td>#1310<br>players</td><td>#371<br>net</td><td>#423<br>tournaments</td><td>#293<br>stroke</td><td>#451<br>greatest</td><td>#371<br>balls</td></tr><tr><td></td><td>2 player</td><td>ball</td><td>singles</td><td>forehand</td><td>ever</td><td>rackets</td></tr><tr><td></td><td>3 tennis</td><td>shot</td><td>doubles</td><td>stance</td><td>female</td><td>size</td></tr><tr><td></td><td>4 also</td><td>serve</td><td>tour</td><td>power</td><td>wingfield</td><td>square</td></tr><tr><td></td><td>5 play</td><td>opponent</td><td>slam</td><td>backhand</td><td>williams</td><td>made</td></tr><tr><td></td><td>6 football</td><td>may</td><td>prize</td><td>torso</td><td>navratilova</td><td>leather</td></tr><tr><td></td><td>7 team</td><td>hit</td><td>money</td><td>grip</td><td>game</td><td>weight</td></tr><tr><td></td><td>8 first</td><td>service</td><td>grand</td><td>rotation</td><td>said</td><td>standard</td></tr><tr><td></td><td>9 one</td><td>hitting</td><td>events</td><td>twohanded</td><td>serena</td><td>width</td></tr><tr><td></td><td>10 rugby</td><td>line</td><td>ranking</td><td>used</td><td>sports</td><td>past</td></tr><tr><td rowspan="5">SVD</td><td>0 players</td><td>net</td><td>tournaments</td><td>stroke</td><td>greatest</td><td>balls</td></tr><tr><td>1 breaking</td><td>pace</td><td>masters</td><td>rotates</td><td>lived</td><td>panels</td></tr><tr><td>2 one</td><td>reach</td><td>lowest</td><td>achieve</td><td>female</td><td>sewn</td></tr><tr><td>3 running</td><td>underhand</td><td>events</td><td>face</td><td>biggest</td><td>entire</td></tr><tr><td>4 often</td><td>air</td><td>tour</td><td>adds</td><td>potential</td><td>leather</td></tr><tr><td></td><td>0 player</td><td>ball</td><td>singles</td><td>forehand</td><td>ever</td><td>rackets</td></tr><tr><td></td><td>1 utilize</td><td>keep</td><td>indian</td><td>twohanded</td><td>autobiography</td><td>meanwhile</td></tr><tr><td></td><td>2 give</td><td>hands</td><td>doubles</td><td>begins</td><td>jack</td><td>laminated</td></tr><tr><td></td><td>3 converted</td><td>pass</td><td>pro</td><td>backhand</td><td>consistent</td><td>wood</td></tr><tr><td></td><td>4 touch</td><td>either</td><td>rankings</td><td>achieve</td><td>gonzales</td><td>strings</td></tr></tbody></table>