# A Polarity Capturing Sphere for Word to Vector Representation
## 3 The Proposed Method  
Let \(x_{i}=(x_{i1},\ldots,x_{iN})^{T}\) be an \(N\)-dimensional vector of unity length that represents word \(i\); i.e., \(\|x_{i}\|=1\). The fact that the length of the vector is 1 means that the word is embedded on a sphere. The dot product between two vectors \(x_{i}^{T}x_{j}\) represents the similarity between their corresponding words. For example \(x_{i}^{T}x_{j}\) for the words _easy_ and _simple_ would be very high (close to 1). For the words _easy_ and _manageable_ it would also be high, but a shade lower. For the opposites _easy_ and _difficult_ it would be close to \(-1\), and for unrelated words, such as _easy_ and _cat_, it would be close to zero. It is well known that when picking up any two random vectors on a sphere of high dimension, their dot product will be close to zero. This means that the bulk of the words will have similarity close to zero with the word _easy_. Note also that because of the unit length property, the dot product \(x_{i}^{T}x_{j}\) and the distance are related in a one to one way (because \(\|x_{i}-x_{j}\|^{2}=\|x_{i}\|^{2}+\|x_{j}\|^{2}-2x_{i}^{T}x_{j}=2\big{[}1-x_{ i}^{T}x_{j}\big{]}\)). We collect a number of words for which we estimate a similarity number, on a scale from \(-1\) to 1. For example, for a pair of words with corresponding vectors \(x_{i}\) and \(x_{j}\), let the estimated similarity be \(s_{ij}\). We collect a very large training set, obtained from some well-known synonym and antonym lists. For any word relations that are not covered by these lists, we add a moderately sized but typically not-large training set, mainly labeled by an expert human. The expert-labeled dataset serves as the nucleus that will guide the training using the other larger collected datasets. We develop an algorithm that estimates the vectors \(x_{i}\) that would yield the similarity numbers as close as possible to the given similarity numbers. We define the following error function:  
\[E =\sum_{i}\sum_{j}w_{ij}\Big{[}x_{i}^{T}x_{j}-s_{ij}\Big{]}^{2}\] subject to \[\|x_{i}\|=1,i=1,\ldots,K \tag{1}\]  
where \(w_{ij}\) is a weighting coefficient representing the confidence in the similarity estimate \(s_{ij}\). For example, the word pairs labeled by an expert may have higher weighting coefficient than other word pairs in the remaining larger training set. It is hard to solve this large optimization problem if one seeks to obtain all \(K\) vectors \(x_{i}\) at once. However, we propose a relaxation algorithm, that tackles one \(x_{i}\) at a time. In this algorithm we focus on some \(x_{k}\) and fix all other \(x_{l}\)'s for the time being. Then we optimize \(E\) with respect to \(x_{k}\). This is feasible and gives a close form solution. Then, we move on to the next vector, and fix the others, optimize with respect to that vector. We continue in this manner until we complete all vectors. Then we start another cycle, and re-optimize each \(x_{k}\), one at a time. We perform a few similar cycles until the algorithm converges. Consider that we are focusing now on vector \(x_{k}\), while keeping all other vectors constant. Then we can decompose the objective function in (1) into a component containing \(x_{k}\) and other components that do not have \(x_{k}\) in them, as follows. \[E =\sum_{i\
eq k}\sum_{j\
eq k}w_{ij}\Big{[}x_{i}^{T}x_{j}-s_{kj} \Big{]}^{2}+2\sum_{j\
eq k}w_{kj}\Big{[}x_{k}^{T}x_{j}-s_{kj}\Big{]}^{2}\] \[=R_{k}+2\sum_{j\
eq k}w_{kj}\Big{[}x_{k}^{T}x_{j}-s_{kj}\Big{]}^{2} \tag{2}\]where \(R_{k}\) is the term not containing \(x_{k}\). We skipped the term \(x_{k}^{T}x_{k}-s_{kk}\) because it equals zero always (the similarity between a vector and itself is 1, and the norm of any vector is enforced as 1 too). The factor 2 in the RHS is to account for the existence of \(x_{k}\) in first summation, and in the second summation. The optimization will now focus on optimizing the second term in the summation.