Venue: ACL 2017

# Semantic Word Clusters Using Signed Spectral Clustering

Joao Sedoc, Jean Gallier, Lyle Ungar

Computer & Information Science

University of Pennsylvania

joao, jean, ungar@cis.upenn.edu Dean Foster

Amazon LLC

dean@foster.net

###### Abstract

Vector space representations of words capture many aspects of word similarity, but such methods tend to produce vector spaces in which antonyms (as well as synonyms) are close to each other. For spectral clustering using such word embeddings, words are points in a vector space where synonyms are linked with positive weights, while antonyms are linked with negative weights. We present a new signed spectral normalized graph cut algorithm, _signed clustering_, that overlays existing thesauri upon distributionally derived vector representations of words, so that antonym relationships between word pairs are represented by negative weights. Our signed clustering algorithm produces clusters of words that simultaneously capture distributional and synonym relations. By using randomized spectral decomposition [1] and sparse matrices, our method is both fast and scalable. We validate our clusters using datasets containing human judgments of word pair similarities and show the benefit of using our word clusters for sentiment prediction.

## 1 Introduction

In distributional vector representations, opposite relations are not fully captured. Take, for example, words such as "great" and "awful" that can appear with similar frequency in the same sentence structure: "John had a great meeting" and "John had an awful day." Word embeddings, which are successful in a wide array of NLP tasks [23, 40], fail to capture this antonymy because they follow the _distributional hypothesis_ that similar words are used in similar contexts [6], thus assigning small cosine or euclidean distances between the vector representations of "great" and "awful".

While vector space models [25] such as word2vec [29], Global vectors (GloVe) [7], or Eigenwords [8] capture relatedness, they do not adequately encode synonymy and semantic similarity [1, 19]. Our goal is to create clusters of synonyms or semantically equivalent words and linguistically motivated unified constructs. Signed graphs, which are graphs with negative edge weights, were first introduced by Cartwright and Harary (1956). However, signed graph clustering for multiclass normalized cuts (K-clusters) has been largely unexplored until recently. We present a novel theory and method that extends multiclass normalized cuts (K-cluster) of Yu and Shi (2003) to signed graphs [1]1 and the work of Kunegis et al. (2010) to K-clustering. This extension allows the incorporation of knowledge base information, positive and negatively weighted links (see figure 2.1). Negative edges serve as repellent or opposite relationships between nodes.

Footnote 1: Gallier (2016) is a full theoretical exposition of our methods with proofs on arXiv.

Our signed spectral normalized graph cut algorithm (henceforth, signed clustering) builds negative edge relations into graph embeddings using similarity structure in vector spaces. It takes as input an initial set of vectors and edge relations, and hence is easy to combine with any word embedding method. This paper formally improves on the discrete optimization problem of Yu and Shi (2003).

Signed clustering gives better clusters than spectral clustering [10] of word embeddings, and it has better coverage and is more robust than desaurus look-

souri erroneously give equal weight to rare senses of a word - for example, "rich" as a rarely used synonym of "absurd". Also, the overlap between thesauri is small, due to their manual creation. Lin (1998) found 17.8397% overlap between synonym sets from Roget's Thesaurus and WordNet 1.5. We find similarly small overlap between all threehesauri tested.

We evaluate our clusters using SimLex-999 Hill et al. (2014) and SimVerb-3500 Gerz et al. (2016) as a ground truth for our cluster evaluation. Finally, we test our method on the sentiment analysis task. Overall, signed spectral clustering can augment methods using signed information and has broad application for many fields.

Our main contributions are: the novel extension of signed clustering to the multiclass (K-cluster), and the application of this method to create semantic word clusters that are agnostic to vector space representations and thesauri.

### Related Work

Semantic word cluster and distributional thesauri have been well studied in the NLP literature Lin (1998); Curran (2004). Recently there has been a line of research on incorporating synonyms and antonyms into word embeddings. Our approach is very much in the line of Vlachos et al. (2009). However, they explicitly made verb clusters using Dirichlet Process Mixture Models and must-link / cannot-link clustering. Furthermore, they note that cannot-link clustering does not improve performance whereas our signed clustering antonyms are key.

Most recent models either attempt to make richer contexts, in order to find semantic similarity, or overlay thesaurus information in a supervised or semi-supervised manner. One line of active research is post processing the word vector embedding by transforming the space using a single or multi-relational objective Yih et al. (2012); Tang et al. (2014); Chang et al. (2013); Tang et al. (2014); Zhang et al. (2014); Faruqui et al. (2015); Mrksic et al. (2016).

Alternatively, there are methods to modify the objective function for generating the word embeddings Ono et al. (2015); Pham et al. (2015); Schwartz et al. (2015).

Our approach differs from the aforementioned methods in that we created word clusters using the antonym relationships as negative links. Unlike the previous approaches using semi-supervised methods, we incorporated the thesauri as a knowledge base. Similar to word vector retrofitting and counter-fitting methods described in Faruqui et al. (2015) and Mrksic et al. (2016), our signed clustering method uses existing vector representations to create word clusters.

To our knowledge, this work is the first theoretical foundation of multiclass signed normalized cuts.2Zass and Shashua (2005) solved multiclass cluster from another approach, by relaxing the orthogonality assumption and focusing instead on the non-negativity constraint. This led to a doubly stochastic optimization problem. Negative edges are handled by a constrained hyperparameter. Hou (2005) used positive degrees of nodes in the degree matrix of a signed graph with weights (-1, 0, 1), which was advanced by Kolluri et al. (2004) and Kunegis et al. (2010) using absolute values of weights in the degree matrix. Interestingly, Chiang et al. (2014) presented a theoretical foundation for edge sign prediction and a recursive clustering approach. Mercado et al. (2016) found that using the geometric mean of the graph Laplacian improves performance.

Footnote 2: The full exposition by Gallier (2016) is available on arXiv.

Wang et al. (2016) used semi-supervised polarity induction Rao and Ravichandran (2009) to create clusters of words with similar valence and arousal. Must-link and cannot-link soft spectral clustering Rangapuram and Hein (2012) share similarities with our method, particularly in the limit where there are no must-link edges present. Both must-link and cannot-link clustering as well as polarity induction differ in optimization method. Our method is significantly faster due to the use of randomized SVD Halko et al. (2011) and can thus be applied to large scale NLP problems.

We developed a novel theory and algorithm that extends the clustering of Shi and Malik (2000) and Yu and Shi (2003) to the multiclass signed graph case.

## 2 Signed Graph Cluster Estimation

### Signed Normalized Cut

Weighted graphs for which the weight matrix is a symmetric matrix in which negative and positive entries are allowed are called _signed graphs_.

 

Such graphs (with weights \((-1,0,+1)\)) were introduced as early as 1953 by [2] to model social relations involving disliking, indifference, and liking. The problem of clustering the nodes of a signed graph arises naturally as a generalization of the clustering problem for weighted graphs. Figure 1 shows a signed graph of word similarities with a thesaurus overlay.

Gallier (2016) extends normalized cuts to signed graphs in order to incorporate antonym information into word clusters. A _weighted graph_ is a pair \(G=(V,W)\), where \(V=\{v_{1},\ldots,v_{m}\}\) is a set of _nodes_ or _vertices_, and \(W\) is a symmetric matrix called the _weight matrix_, such that \(w_{i\,j}\geq 0\) for all \(i,j\in\{1,\dots,m\}\), and \(w_{i\,i}=0\) for \(i=1,\ldots,m\). We say that a set \(\{v_{i},v_{j}\}\) is an edge iff \(w_{i\,j}>0\). The corresponding (undirected) graph \((V,E)\) with \(E=\{\{v_{i},v_{j}\}~{}|~{}w_{i\,j}>0\}\), is called the _underlying graph_ of \(G\). Given a signed graph \(G=(V,W)\) (where \(W\) is a symmetric matrix with zero diagonal entries), the _underlying graph_ of \(G\) is the graph with node set \(V\) and set of (undirected) edges \(E=\{\{v_{i},v_{j}\}~{}|~{}w_{ij}\neq 0\}\). If \((V,W)\) is a signed graph, where \(W\) is an \(m\times m\) symmetric matrix with zero diagonal entries and with the other entries \(w_{ij}\in\mathbb{R}\) arbitrary, for any node \(v_{i}\in V\), the _signed degree_ of \(v_{i}\) is defined as

\[\overline{d}_{i}=\overline{d}(v_{i})=\sum_{j=1}^{m}|w_{ij}|,\]

and the _signed degree matrix_\(\overline{D}\) as

\[\overline{D}=\operatorname{diag}(\overline{d}(v_{1}),\ldots,\overline{d}(v_{ m})).\]

For any subset \(A\) of the set of nodes \(V\), let

\[\operatorname{vol}(A)=\sum_{v_{i}\in A}\overline{d}_{i}=\sum_{v_{i}\in A}\sum _{j=1}^{m}|w_{ij}|.\]

For any two subsets \(A\) and \(B\) of \(V\) and \(A^{C}\) which is the complement of \(A\), define \(\operatorname{links}^{+}(A,B)\), \(\operatorname{links}^{-}(A,B)\), and \(\operatorname{cut}(A,A^{C})\) by

\[\operatorname{links}^{+}(A,B) =\sum_{\begin{subarray}{c}v_{i}\in A,v_{j}\in B\\ w_{ij}>0\end{subarray}}w_{ij}\] \[\operatorname{links}^{-}(A,B) =\sum_{\begin{subarray}{c}v_{i}\in A,v_{j}\in B\\ w_{ij}<0\end{subarray}}-w_{ij}\] \[\operatorname{cut}(A,A^{C}) =\sum_{\begin{subarray}{c}v_{i}\in A,v_{j}\in A^{C}\\ w_{ij}\neq 0\end{subarray}}|w_{ij}|.\]

Then, the _signed Laplacian_\(\overline{L}\) is defined by

\[\overline{L}=\overline{D}-W,\]

and its normalized version \(\overline{L}_{\mathrm{sym}}\) by

\[\overline{L}_{\mathrm{sym}}=\overline{D}^{-1/2}\,\overline{L}\,\overline{D}^ {-1/2}=I-\overline{D}^{-1/2}W\overline{D}^{-1/2}.\]

Kuneggis (2010) showed that \(\overline{L}\) is positive semidefinite. For a graph without isolated vertices, we have \(\overline{d}(v_{i})>0\) for \(i=1,\ldots,m\), so \(\overline{D}^{-1/2}\) is well defined. Given a partition of \(V\) into \(K\) clusters \((A_{1},\ldots,A_{K})\), if we represent the \(j\)th block of this partition by a vector \(X^{j}\) such that

\[X^{j}_{i}=\begin{cases}a_{j}&\text{if $v_{i}\in A_{j}$}\\ 0&\text{if $v_{i}\notin A_{j}$},\end{cases}\]

for some \(a_{j}\neq 0\). For illustration, suppose \(m=5\) and \(A_{1}=\{v_{1},v_{3}\}\) then \((X^{1})^{\top}=[a_{1},0,a_{1},0,0]\). The _signed normalized cut_\(\operatorname{sNcut}(A_{1},\ldots,A_{K})\) of the partition \((A_{1},...,A_{K})\) is defined as

\[\operatorname{sNcut}(A_{1},\ldots, A_{K})=\] \[\quad\qquad\qquad\sum_{j=1}^{K}\frac{\operatorname{cut}(A_{j},A_{ j}^{C})+2\operatorname{links}^{-}(A_{j},A_{j})}{\operatorname{vol}(A_{j})}.\]

Figure 1: Signed graph of words using a distance metric from the word embedding. The red dashed edges represent the antonym relation while solid edges represent synonymy relations.

 

It should be noted that this formulation differs significantly from Kunegis et al. (2010) and even more so from must-link / cannot-link clustering.

Observe that minimizing \({\rm sNcut}(A_{1},\ldots,A_{K})\) minimizes the number of positive and negative edges between clusters and also the number of negative edges within clusters. Removing the term \({\rm links^{-}}(A_{j},A_{j})\) reduces \({\rm sNcut}\) to normalized cuts.

A linear algebraic formulation is

\[{\rm sNcut}(A_{1},\ldots,A_{K})=\sum_{j=1}^{K}\frac{(X^{j})^{\top}\overline{L} X^{j}}{(X^{j})^{\top}\overline{D}X^{j}}.\]

where \(X\) is the \(N\times K\) matrix whose \(j\)th column is \(X^{j}\).

### Optimization Problem

We now formulate \(K\)-way clustering of a graph using normalized cuts.

If we let

\[\mathcal{X}=\Big{\{}[X^{1}~{}\dots~{}X^{K}]~{}|~{}X^{j}=a_{j}(x_{1}^{j},\ldots ,x_{N}^{j}),\]

\[x_{i}^{j}\in\{1,0\},a_{j}\in\mathbb{R},~{}X^{j}\neq 0\Big{\}}\]

our solution set is

\[\mathcal{K}=\Big{\{}X\in\mathcal{X}~{}|~{}(X^{i})^{\top}\overline{ D}X^{j}=0,\] \[1\leq i,j\leq K,\quad i\neq j\Big{\}}.\]

The resulting optimization problem is

minimize \[\sum_{j=1}^{K}\frac{(X^{j})^{\top}\overline{L}X^{j}}{(X^{j})^{ \top}\overline{D}X^{j}}\] subject to \[(X^{i})^{\top}\overline{D}X^{j}=0,\] \[1\leq i,j\leq K,~{}i\neq j,~{}~{}X\in\mathcal{X}.\]

The problem can be reformulated to an equivalent optimization problem:

minimize \[{\rm tr}(X^{\top}\overline{L}X)\] subject to \[X^{\top}\overline{D}X=I,\hskip 8.535827ptX\in\mathcal{X}.\]

We then form a relaxation of the above problem, dropping the condition that \(X\in\mathcal{X}\), giving

**Relaxed Problem**

minimize \[{\rm tr}(Y^{\top}\overline{D}^{-1/2}\overline{LD}^{-1/2}Y)\] subject to \[Y^{\top}Y=I.\]

The minimum of the relaxed problem is achieved by the \(K\) unit eigenvectors associated with the smallest eigenvalues of \(L_{\mathrm{sym}}\).

### Finding an Approximate Discrete Solution

Given a solution \(Z\) of the relaxed problem, we look for pairs \((X,Q)\) with \(X\in\mathcal{X}\) and where \(Q\) is a \(K\times K\) matrix with nonzero and pairwise orthogonal columns, with \(\left\lVert X\right\rVert_{F}=\left\lVert Z\right\rVert_{F}\), that minimize

\[\varphi(X,Q)=\left\lVert X-ZO\right\rVert_{F}.\]

Here, \(\left\lVert A\right\rVert_{F}\) is the Frobenius norm of \(A\).

This nonlinear optimization problem involves two unknown matrices \(X\) and \(Q\). To solve the relaxed problem, we proceed by alternating between minimizing \(\varphi(X,Q)=\left\lVert X-ZQ\right\rVert_{F}\) with respect to \(X\) holding \(Q\) fixed (step 5 in algorithm 1), and minimizing \(\varphi(X,Q)\) with respect to \(Q\) holding \(X\) fixed (steps 6 and 7 in algorithm 1).

This second stage in which \(X\) is held fixed has been studied, but it is still a hard problem for which no closed-form solution is known. Hence we divide the problem into steps 6 and 7 for which the solution is known. Since \(Q\) is of the form \(Q=R\Lambda\) where \(R\in\mathbf{O}(K)\) and \(\Lambda\) is a diagonal invertible matrix, we minimize \(\left\lVert X-ZR\Lambda\right\rVert_{F}\). The matrix \(R\Lambda\) is not a minimizer of \(\left\lVert X-ZR\Lambda\right\rVert_{F}\) in general, but it is an improvement on \(R\) alone, and both stages can be solved quite easily. In step 6 the problem reduces to minimizing \(-2{\rm tr}(Q^{\top}Z^{\top}X)\); that is, maximizing \({\rm tr}(Q^{\top}Z^{\top}X)\).

```
1:Input:\(W\) the weight matrix (without isolated nodes), K the number of clusters, and termination threshold \(\epsilon\).
2:Using the \(\overline{D}\) the degree matrix, and the signed Laplacian \(\overline{L}\), compute \(\overline{L}_{sym}\) the signed normalized Laplacian. ------------------------------------------------
3:Initialize \(\Lambda=I\), \(X=\overline{D}^{-\frac{1}{2}}U\) where \(U\) is the matrix of the eigenvectors corresponding to the K smallest eigenvalues of \(\overline{L}_{sym}\)..
4:while\(\left\lVert X-ZRA\right\rVert_{p}>\epsilon\)do
5: Minimize \(\left\lVert X-ZRA\right\rVert_{F}\) with respect to \(X\) holding \(Q\) fixed.
6: Fix \(X\), \(Z\), and \(\Lambda\), find \(R\in\mathbf{O}(K)\) that minimizes \(\left\lVert X-ZRA\right\rVert_{F}\).
7: Fix \(X\), \(Z\), and \(R\), find a diagonal invertible matrix \(\Lambda\) that minimizes \(\left\lVert X-ZRA\right\rVert_{F}\).
8:end while
9:Find the discrete solution \(X^{*}\) by choosing the largest entry \(x_{ij}\) on row \(i\) set \(x_{ij}=1\) and all other \(x_{ij}=0\) for row \(i\).
10:Output:\(X^{*}\).
```

**Algorithm 1** Signed Clustering 

removing the solution requirement that \(X^{j}\neq 0\), the algorithm can find \(k\leq K\) clusters.

## 3 Similarity Calculation

The main input to the spectral signed clustering algorithm is the similarity matrix \(W\), which overlays both the distributional properties and thesaurus information. Following Belkin and Niyogi (2003), we chose the heat kernel based on the Euclidean distance between word vector representations as our similarity metric, such that

\[W_{ij}=\begin{cases}0&\text{if $e^{-\frac{\left\|w_{i}-w_{j}\right\|^{2}}{ \sigma}}<\epsilon$}\\ e^{-\frac{\left\|w_{i}-w_{j}\right\|^{2}}{\sigma}}&\text{otherwise}\end{cases}.\]

where \(\sigma\) and \(\epsilon\) are hyperparameters found using grid search (see Supplemental material for more detail).

We represented the desaurus as two matrices where

\[T^{syn}_{ij}=\begin{cases}1&\text{ if words $i$ and $j$ are synonyms}\\ 0&\text{ otherwise}\end{cases}.\]

and

\[T^{ant}_{ij}=\begin{cases}-1&\text{ if words $i$ and $j$ are antonyms}\\ 0&\text{ otherwise}\end{cases}.\]

\(T^{syn}\) is the synonym graph and \(T^{ant}\) is the antionym graph. The signed graph can then be written in matrix form as \(\hat{W}=\gamma W+\beta^{ant}T^{ant}\odot W+\beta^{syn}T^{syn}\odot W\), where \(\odot\) computes Hadamard product (element-wise multiplication).

The parameters \(\gamma\), \(\beta^{syn}\), and \(\beta^{ant}\) are tuned to the data target dataset using cross validation. The reader should note that \(\sigma\) and \(\epsilon\) are not found using a target dataset, but instead using cross validation and grid search to minimize the number of negative edges within clusters and the number of disconnected components in the cluster.

## 4 Evaluation Metrics

We evaluated the clusters using both intrinsic and extrinsic methods. For intrinsic evaluation, we used thesaurus information for two novel metrics: 1) the number of negative edges (NNE) within the clusters, which in our semantic clusters is the number of antinorms in the same cluster, and 2) the number of disconnected components (NDC) in the synonym graph, so the number of groups of words that are not connected by a synonym relation in the thesaurus. The NDC thus has the disadvantage that it is a function of the thesaurus coverage. Our third intrinsic measure uses a gold standard designed to measure how well we capture word similarity: Semantically similar words should be in the same cluster and semantically dissimilar words should not. For extrinsic evaluation, as descibed below, we measure how much our clusters help to identify text polarity. We also compare multiple word embeddings and thesauri to demonstrate the stability of our method.

## 5 Experiments with Synthetic Data

In order to evaluate our signed graph clustering method, we first focused on intrinsic measures of cluster quality in synthetic data. To do so, we created random signed graphs with the same proportion of positive and negative edges as in our real dataset. Figure 2 demonstrates that the number of negative edges within a cluster is minimized using our clustering algorithm on simulated data. As the number of clusters becomes large, the number of disconnected components, which includes clusters of size one, consistently increases. Determining the optimal cluster size and similarity parameters requires making a trade off between NDC and NNE. For example, in figure 2 the optimal cluster size is 20. One can see that as the number of clusters increases NNE goes to zero, but the number of disconnected components becomes the number of vertices. In the extreme case all clusters contain one vertex. K-means, also shown in figure 2, does not optimize NNE.

Figure 2: The relation between disconnected component (NDC) and negative edge (NNE) using simulated signed graphs with 100 vertices.

 

Experimental Setup

### Word Embeddings

We used four different word embedding methods for evaluation: Skip-gram vectors (word2vec) Mikolov et al. (2013), Global vectors (GloVe) Pennington et al. (2014), Eigenwords Dhillon et al. (2015), and Global Context (GloCon) Huang et al. (2012); however, we only report the results for word2vec, which is the most popular word embedding (see the supplemental material for other embeddings). We used word2vec 300 dimensional embeddings which were trained on several billion words of English: the Gigaword and the English discussion forum data gathered as part of BOLT. Tokenization was performed using CMU's Twokenze.4

Footnote 4: [https://github.com/brendano/ark-tweet-nlp](https://github.com/brendano/ark-tweet-nlp)

### Thesauri

Several thesauri were used in order to test the robustness including Roget's Theseaurus Roget (1852), the Microsoft Word English (MS Word) theseaurus from Samsonovic et al. (2010) and WordNet 3.0 Miller (1995).

We chose a subset of 5108 words for the training dataset, which had high overlap between various sources. Changes to the training dataset had minimal effects on the optimal parameters. Within the training dataset, each of the thesauri had roughly 3700 antonym pairs; combined they had 6680. However, the number of distinct connected components varied, with Roget's Thesaurus having the fewest (629), and MS Word Thesaurus (1162) and WordNet (2449) having the most. These ratios were consistent across the full dataset.

### Gold Standard SimLex-999 And SimVerb-3500

Following the analysis of Vlachos et al. (2009), we threshold the semantically similar datasets to find word pairs which should or should not belong to the same cluster. As ground truth, we extracted 120 semantically similar words from SimLex-999 with a similarity score greater than 8 out of 10. SimLex-999 is a gold standard resource for semantic similarity, not relatedness, based on ratings by human annotators.

Our 120 pair subset of SimLex-999 has multiple parts-of-speech including Noun-Noun pairs, Verb-Verb pairs and Adjective-Adjective pairs. Within SimVerb-3500, we used a subset of 318 semantically similar verb pairs.

The community is attempting to define better gold standards; however, currently these are the best datasets that we are aware of. We tried to use WordNet, Roget, and the Paraphrase Database (PPDB) Ganitkevitch et al. (2013) as a gold standard, but manual inspection as well as empirical results showed that none of the automatically generated datasets were a sufficient gold standard. Possibly the symmetric pattern of Schwartz et al. (2015) would have been sufficient; we did not have time to validate this.

### Stanford Sentiment Treebank

We also evaluated our clusters by using them as features for predicting sentiment, using sentiment treebank 5 Socher et al. (2013) with coarsegrained labels on phrases and sentences from movie review excerpts. This dataset is widely used for the evaluation of sentiment analysis. We used the standard partition of the treebank into training (6920), development (872), and test (1821) sets.

Footnote 5: [http://nlp.stanford.edu/sentiment/treebank.html](http://nlp.stanford.edu/sentiment/treebank.html)

## 7 Cluster Evaluation

Table 1 shows the four most-associated words with "accept" using different methods.

We now turn to quantitative measures of word similarity and synonym cluster quality.

### Comparison with K-means and Normalized Cuts

In order to assess the model we tested (1) K-means, (2) normalized cuts without thesaurus, and (3) signed normalized cuts. As a baseline, we created clusters using K-means on the original word2vec vector representations where the number of K clusters was set to 750.

Table 2 shows the relative ratios of the different clustering methods of with respect to antonym pair inclusion and the number of disconnected components within the clusters. For both methods, over twenty percent of the clusters contain antonym pairs even though the median cluster size is six. Signed clustering radically reduced the number of anonymous within clusters compared to the other methods.

 



## 8 Empirical Results

Tables 3 and 5 present our main result. When using our signed clustering method with similar words, as labeled by SimLex-999 and SimVerb-3500, our clustering accuracy increased by 5% on both SimLex-999 and SimVerb-3000. Furthermore, by combining the thesauri lookup with our clustering, we achieved almost perfect accuracy (96%). Table 5 shows the sentiment analysis task performance. Our method outperforms all methods with similar complexity; however, we did not reach state-of-the-art results when compared to much more complex models which also use a richer dataset.

### Evaluation Using Word Similarity Datasets

In a perfect setting, all word pairs rated highly similar by human annotators would be in the same cluster, and all words which were rated dissimilar would be in different clusters. Since our clustering algorithm produced sets of words, we used this evaluation instead of the more commonly reported correlations.

In table 3 we show the results of the evaluation with SimLex-999. Combining thesaurus lookup and word2vec+CombThese clusters, labeled as Lookup + SC(W2V), yielded an accuracy of 0.96 (5 errors). Note that clusters using word2vec with normalized cuts does not improve accuracy. The MSW desaurus has much lower coverage, but 100 % accuracy, which is why when combined with the signed clustering the performance is 0.95. In table 3 we state the proportion of clusters containing dissimilar words as a sanity check for cluster size. (See supplemental material for full cluster size optimization information.) Another important result is that the verb accuracy yielded the largest accuracy gains, consistent with the results of Schwartz et al. (2015).

Table 4 clearly shows that the overall performance of all methods is lower for verb similarity. However, the improvement using both signed clustering as well as thesaurus look is also larger.

### Sentiment Analysis

We trained an \(l_{2}\)-norm regularized logistic regression Friedman et al. (2001) and simultaneously \(\gamma\), \(\beta^{syn}\), and \(\beta^{ant}\) using our word clusters in order to predict the coarse-grained sentiment at the sentence level. The \(\gamma\) and \(\beta\) parameters were found using a portion of the data where we iteratively switch between the logistic regression and the parameters, holding each fixed. However, hyperparameters \(\sigma\) and \(\epsilon\), and the number of clusters

\begin{table}
\begin{tabular}{|p{71.1pt}||p{71.1pt}|p{71.1pt}|p{71.1pt}|p{71.1pt}|} \hline
**Ref word** & **Roget** & **WordNet** & **MS Word** & **W2V** & **SC W2V** \\ \hline accept & adopt accept your fate & agree get be fooled by acquiesce & take get gatry & accepts reject & grant permit \\  & & be fooled by acquiesce & fancy find & consent & agree & let \\ \cline{2-5}  & acquiesce & hold & assume & accepting & okay \\ \hline \end{tabular}
\end{table}
Table 1: Qualitative comparison of clusters.

\begin{table}
\begin{tabular}{|p{71.1pt}||p{71.1pt}|p{71.1pt}|} \hline
**Method** & **Antonym Ratio** & **DC Ratio** \\ \hline K-Means & 0.24 & 0.95 \\ \hline NC & 0.21 & 0.97 \\ \hline SC & 0.06 & 0.49 \\ \hline \end{tabular}
\end{table}
Table 2: Clustering evaluation of K-means, normalized cuts, and signed normalized cuts with 750 clusters. Ratio of clusters with containing one or more antonym pair and ratio of clusters with disconnected components.

\begin{table}
\begin{tabular}{|p{71.1pt}||p{71.1pt}|p{71.1pt}|} \hline
**Method** & **Acc SimLex** & **Err** \\ \hline MSW Lookup & 0.70 & 0 \\ \hline Roget Lookup & 0.63 & 0 \\ \hline WordNet Lookup & 0.43 & 0 \\ \hline Combined Lookup & 0.90 & 0 \\ \hline NC(W2V) & 0.36 & 0.05 \\ \hline
**SC** (W2V) & **0.67** & 0 \\ \hline Lookup + NC(W2V) & 0.91 & 0.05 \\ \hline Lookup + SC(W2V) & **0.96** & 0 \\ \hline MSW + SC(W2V) & 0.95 & 0 \\ \hline \end{tabular}
\end{table}
Table 3: Clustering evaluation using SimLex-999 with 120 word pairs having similarity score over 8. SC stands for our signed clustering and NC is standard normalized cuts. **SC**(W2V) are the word clusters from signed clustering using word2vec and the combined thesauri. **Err** is the proportion of dissimilar words (with score \(<\) 2) present in the same cluster.



\(K\) were optimized minimizing error using grid search. We compared our model against existing models: Naive Bayes with bag of words (NB) Socher et al. (2013), sentence word embedding averages (VecAvg), retrofitted sentence word embeddings (RVecAvg) Faruqui et al. (2015) that incorporate thesaurus information, simple recurrent neural networks (RNN), and two baselines of normalized cuts and signed normalized cuts using only thesaurus information.

While the state-of-the art Convolutional Neural Network (CNN) Kim (2014) is at 0.881, our model performs quite well with much less information and complexity. Table 5 shows that signed clustering outperforms the baselines of Naive Bayes, normalized cuts, and signed cuts using just thesaurus information. Furthermore, we outperform comparable models, including retrofitting, which has thesaurus information, and the recurrent neural network, which has access to domain specific context information.

Signed clustering using only thesaurus information (**SC(Thes)**) performed significantly worse than all other methods. This was largely due to low coverage; rare words such as "WOW" and "???" are not covered. As expected, because normalized cut clusters include antonyms, the method performs worse than others. Nonetheless the improvement from 0.79 to 0.836 is quite drastic.

## 9 Conclusion

We developed a novel theory for signed normalized cuts and an algorithm for finding their discrete solution. We showed that we can find superior semantically similar clusters which do not require new word embeddings but simply overlay thesaurus information on preexisting ones. The clusters are general and can be used with many out-of-the-box word embeddings. By accounting for anonym relationships, our algorithm greatly outperforms simple normalized cuts. Finally, we examined our clustering method on the sentiment analysis task from Socher et al. (2013) sentiment treebank dataset and showed that it improved performance versus comparable models.

Our automatically generated clusters give better coverage than manually constructed thesauri. Our signed spectral clustering method allows us to incorporate the knowledge contained in these thesauri without modifying the word embeddings themselves. We further showed that use of the thesauri can be tuned to the task at hand.

Our signed spectral clustering method could be applied to a broad range of NLP tasks, such as prediction of social group clustering, identification of personal versus non-personal verbs, and analyses of clusters which capture positive, negative, and objective emotional content. It could also be used to explore multi-view relationships, such as aligning synonym clusters across multiple languages. Another possibility is to use thesauri and word vector representations together with word sense disambiguation to generate semantically similar clusters for multiple senses of words. Furthermore, signed spectral clustering has broader applications such as cellular biology, social networking, and electricity networks. Finally, we plan to extend the hard signed clustering presented here to probabilistic soft clustering.

\begin{table}
\begin{tabular}{|l||c|} \hline
**Method** & **Acc SimVerb** \\ \hline MSW Lookup & 0.45 \\ \hline Roget Lookup & 0.59 \\ \hline WordNet Lookup & 0.43 \\ \hline Combined Lookup & 0.83 \\ \hline \hline NC(W2V) & 0.24 \\ \hline
**SC** (W2V) & **0.56** \\ \hline Lookup + NC(W2V) & 0.83 \\ \hline Lookup + **SC**(W2V) & **0.88** \\ \hline \end{tabular}
\end{table}
Table 4: Clustering evaluation using SimVerWeb 3500 with 317 word pairs having similarity score over 8. SC stands for our signed clustering and NC is standard normalized cuts. **SC**(W2V) are the word clusters from signed clustering using word2vec and the combined thesauri.

\begin{table}
\begin{tabular}{|p{113.8pt}||c|} \hline
**Model** & **Accuracy** \\ \hline NB Socher et al. (2013) & 0.818 \\ \hline VecAvg (W2V) & 0.812 \\ Faruqui et al. (2015) & \\ \hline RVecAvg (W2V) & 0.821 \\ Faruqui et al. (2015) & \\ \hline RNN Socher et al. (2013) & 0.824 \\ \hline NC(W2V) & 0.79 \\ \hline
**SC**(These) & 0.752 \\ \hline
**SC**(**W2V**) & **0.836** \\ \hline \end{tabular}
\end{table}
Table 5: Sentiment analysis accuracy for binary predictions of signed clustering algorithm (SC) versus other models. **SC**(W2V) are the signed clusters using word2vec word representations.

 



## References

* Belkin and Niyogi (2003) Mikhail Belkin and Partha Niyogi. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. _Neural computation_ 15(6):1373-1396.
* Cartwright and Harary (1956) Dorwin Cartwright and Frank Harary. 1956. Structural balance: a generalization of heider's theory. _Psychological review_ 63(5):277.
* Chang et al. (2013) Kai-Wei Chang, Wen-tau Yih, and Christopher Meek. 2013. Multi-relational latent semantic analysis. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 1602-1612. [http://aclweb.org/anthology/D13-1167](http://aclweb.org/anthology/D13-1167).
* Chiang et al. (2014) Kai-Yang Chiang, Cho-Jui Hsieh, Nagarajan Natarajan, Inderjit S. Dhillon, and Ambuj Tewari. 2014. Prediction and clustering in signed networks: A local to global perspective. _Journal of Machine Learning Research_ 15:1177-1213. [http://jmlr.org/papers/v15/chiang14a.html](http://jmlr.org/papers/v15/chiang14a.html).
* Curran (2004) James Richard Curran. 2004. From distributional to semantic similarity .
* Dhillon et al. (2015) Paramveer S. Dhillon, Dean P. Foster, and Lyle H. Ungar. 2015. Eigenwords: Spectral word embeddings. _Journal of Machine Learning Research_ 16:3035-3078. [http://jmlr.org/papers/v16/dhillon15a.html](http://jmlr.org/papers/v16/dhillon15a.html).
* Faruqui et al. (2015) Manaal Faruqui, Jesse Dodge, Kumar Sujay Jauhar, Chris Dyer, Eduard Hovy, and A. Noah Smith. 2015. Retrofitting word vectors to semantic lexicons. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, pages 1606-1615. [https://doi.org/10.3115/v1/N15-1184](https://doi.org/10.3115/v1/N15-1184).
* Friedman et al. (2001) Jerome Friedman, Trevor Hastie, and Robert Tibshirani. 2001. _The elements of statistical learning_, volume 1. Springer series in statistics Springer, Berlin.
* Gallier (2016) Jean Gallier. 2016. Spectral theory of unsigned and signed graphs applications to graph clustering: a survey. _arXiv preprint arXiv:1601.04692_ .
* Ganitkevitch et al. (2013) Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, pages 758-764. [http://aclweb.org/anthology/N13-1092](http://aclweb.org/anthology/N13-1092).
* Gerz et al. (2016) Daniela Gerz, Ivan Vulic, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. Simverb-3500: A large-scale evaluation set of verb similarity. _arXiv preprint arXiv:1608.00869_ .
* Halko et al. (2011) Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. 2011. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. _SIAM review_ 53(2):217-288.
* Harary (1953) Frank Harary. 1953. On the notion of balance of a signed graph. _The Michigan Mathematical Journal_ 2(2):143-146.
* Harris (1954) Zellig S Harris. 1954. Distributional structure. _Word_ .
* Hill et al. (2014) Felix Hill, Roi Reichart, and Anna Korhonen. 2014. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. _arXiv preprint arXiv:1408.3456_ .
* Hou (2005) Yao Ping Hou. 2005. Bounds for the least laplacian eigenvalue of a signed graph. _Acta Mathematica Sinica_ 21(4):955-960.
* Huang et al. (2012) Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 873-882. [http://aclweb.org/anthology/P12-1092](http://aclweb.org/anthology/P12-1092).
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 1746-1751. [https://doi.org/10.3115/v1/D14-1181](https://doi.org/10.3115/v1/D14-1181).
* Kolluri et al. (2004) Ravikrishna Kolluri, Jonathan Richard Shewchuk, and James F O'Brien. 2004. Spectral surface reconstruction from noisy point clouds. In _Proceedings of the 2004 Eurographics/ACM SIGGRAPH symposium on Geometry processing_. ACM, pages 11-21.
* Kunegis et al. (2010) Jerome Kunegis, Stephan Schmidt, Andreas Lommatzsch, Jurgen Lerner, Ernesto William De Luca, and Sahin Albayrak. 2010. Spectral analysis of signed graphs for clustering, prediction and visualization. In _SDM_. SIAM, volume 10, pages 559-559.
* Lin (1998) Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In _36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2_. [http://aclweb.org/anthology/P98-2127](http://aclweb.org/anthology/P98-2127).
* Mercado et al. (2016) Pedro Mercado, Francesco Tudisco, and Matthias Hein. 2016. Clustering signed networks with the geometric mean of laplacians. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems 29_. Curran Associates, Inc., pages 4421-4429. [http://papers.nips.cc/paper/6164-clustering-signed-networks-with-the-geometric-mean-of-laplacians.pdf](http://papers.nips.cc/paper/6164-clustering-signed-networks-with-the-geometric-mean-of-laplacians.pdf)

* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_ .
* Miller (1995) George A Miller. 1995. Wordnet: a lexical database for english. _Communications of the ACM_ 38(11):39-41.
* Mohammad et al. (2013) M. Saif Mohammad, J. Bonnie Dorr, Graeme Hirst, and D. Peter Turney. 2013. Computing lexical contrast. _Computational Linguistics_ 39(3). [https://doi.org/10.1162/COLI_a.00143](https://doi.org/10.1162/COLI_a.00143).
* Mrksic et al. (2016) Nikola Mrksic, Diarmuid O Seaghdha, Blaise Thomson, Milica Gasic, M. Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, pages 142-148. [https://doi.org/10.18653/v1/N16-1018](https://doi.org/10.18653/v1/N16-1018).
* Ono et al. (2015) Masataka Ono, Makoto Miwa, and Yutaka Sasaki. 2015. Word embedding-based antonym detection using thesauri and distributional information. In _Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, pages 984-989. [https://doi.org/10.3115/v1/N15-1100](https://doi.org/10.3115/v1/N15-1100).
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 1532-1543. [https://doi.org/10.3115/v1/D14-1162](https://doi.org/10.3115/v1/D14-1162).
* Pham et al. (2015) The Nghia Pham, Angelik Lazaridou, and Marco Baroni. 2015. A multitask objective to inject lexical contrast into distributional semantics. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. Association for Computational Linguistics, pages 21-26. [https://doi.org/10.3115/v1/P15-2004](https://doi.org/10.3115/v1/P15-2004).
* Rangapuram and Hein (2012) Syama Sundar Rangapuram and Matthias Hein. 2012. Constrained 1-spectral clustering. _International conference on Artificial Intelligence and Statistics (AISTATS)_ 22:1143--1151.
* Rao and Ravichandran (2009) Delip Rao and Deepak Ravichandran. 2009. Semi-supervised polarity lexicon induction. In _Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)_. Association for Computational Linguistics, pages 675-682. [http://aclweb.org/anthology/E09-1077](http://aclweb.org/anthology/E09-1077).
* Roget (1852) Peter Mark Roget. 1852. _Roget's Thesaurus of English Words and Phrases..._ Longman Group Ltd.
* Samsonovic et al. (2010) Alexei V Samsonovic, Giorgio A Ascoli, and Jeffrey Krichmar. 2010. Principal semantic components of language and the measurement of meaning. _PloS one_ 5(6):e10921.
* Scheible et al. (2013) Silke Scheible, Sabine Schulte im Walde, and Sylvia Springorum. 2013. Uncovering distributional differences between synonyms and antonyms in a word space model. In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_. Asian Federation of Natural Language Processing, pages 489-497. [http://aclweb.org/anthology/I13-1056](http://aclweb.org/anthology/I13-1056).
* Schwartz et al. (2015) Roy Schwartz, Roi Reichart, and Ari Rappoport. 2015. Symmetric pattern based word embeddings for improved word similarity prediction. In _Proceedings of the Nineteenth Conference on Computational Natural Language Learning_. Association for Computational Linguistics, pages 258-267. [https://doi.org/10.18653/v1/K15-1026](https://doi.org/10.18653/v1/K15-1026).
* Shi and Malik (2000) Jianbo Shi and Jitendra Malik. 2000. Normalized cuts and image segmentation. _Pattern Analysis and Machine Intelligence, IEEE Transactions on_ 22(8):888-905.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, D. Christopher Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 1631-1642. [http://aclweb.org/anthology/D13-1170](http://aclweb.org/anthology/D13-1170).
* Tang et al. (2014) Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, pages 1555-1565. [https://doi.org/10.3115/v1/P14-1146](https://doi.org/10.3115/v1/P14-1146).
* Turney et al. (2010) Peter D Turney, Patrick Pantel, et al. 2010. From frequency to meaning: Vector space models of semantics. _Journal of artificial intelligence research_ 37(1):141-188.
* Vlachos et al. (2009) Andreas Vlachos, Anna Karbonen, and Zoubin Ghahramani. 2009. _Proceedings of the Workshop on Geometrical Models of Natural Language Semantics_, Association for Computational Linguistics, chapter Unsupervised and Constrained Dirichlet Process Mixture Models for Verb Clustering, pages 74-82. [http://aclweb.org/anthology/W09-0210](http://aclweb.org/anthology/W09-0210).
* Wang et al. (2016) Jin Wang, Liang-Chih Yu, K Robert Lai, and Xuejie Zhang. 2016. Community-based weighted graph model for valence-arousal prediction of affective words. _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 24(11):1957-1968 

Wen-tau Yih, Geoffrey Zweig, and John Platt. 2012. Polarity inducing latent semantic analysis. In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_. Association for Computational Linguistics, pages 1212-1222. [http://aclweb.org/anthology/D12-1111](http://aclweb.org/anthology/D12-1111).
* Yu and Shi (2003) Stella X Yu and Jianbo Shi. 2003. Multiclass spectral clustering. In _Computer Vision, 2003. Proceedings. Ninth IEEE International Conference on_. IEEE, pages 313-319.
* Zass and Shashua (2005) Ron Zass and Amnon Shashua. 2005. A unifying approach to hard and probabilistic clustering. In _Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on_. IEEE, volume 1, pages 294-301.
* Zhang et al. (2014) Jingwei Zhang, Jeremy Salwen, Michael Glass, and Alfio Glozzo. 2014. Word semantic representations using bayesian probabilistic tensor factorization. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, pages 1522-1531. [https://doi.org/10.3115/v1/D14-1161](https://doi.org/10.3115/v1/D14-1161).