Title: FUZZY PARAPHRASES IN LEARNING WORD REPRESEN-TATIONS WITH A LEXICON

Abstract: A synonym of a polysemous word is usually only the paraphrase of one sense among many. When lexicons are used to improve vector-space word representations, such paraphrases are unreliable and bring noise to the vector-space. The prior works use a coefficient to adjust the overall learning of the lexicons. They regard the paraphrases equally. In this paper, we propose a novel approach that regards the paraphrases diversely to alleviate the adverse effects of polysemy. We annotate each paraphrase with a degree of reliability. The paraphrases are randomly eliminated according to the degrees when our model learns word representations. In this way, our approach drops the unreliable paraphrases, keeping more reliable paraphrases at the same time. The experimental results show that the proposed method improves the word vectors. Our approach is an attempt to address the polysemy problem keeping one vector per word. It makes the approach easier to use than the conventional methods that estimate multiple vectors for a word. Our approach also outperforms the prior works in the experiments. 1

:

• Although some prior works propose to solve the polysemy problem by estimating one vector per word sense or type, using such word vectors requires additional pre-process.

Our proposed method keeps one vector per word. It makes the word vectors easier to use in practical terms: it is neither necessary to disambiguate the word senses nor to tag the part-of-speeches before we use the word vectors.

We give an introduction of our proposed method in section 2. We show the effects of different paraphrase sets, parameters, corpus size, and evaluate the effectiveness of our approach by comparing to simpler algorithms in section 3. We compare our approach with the prior works via an evaluation experiment in section 4. We give the findings, conclusions and outlook in section 5.

2. THE PROPOSED METHOD:

2.1 FUZZY PARAPHRASES As described in section 1, whether a polysemous word's paraphrase is the same as the original depends on the context.

Henceforth, if we simply use all the paraphrases of a word in the lexicon to improve the word vector without discrimination, they may sometimes bring noise to the vector-space.

A conventional method for them is to give each word sense a vector. However, such vector-spaces require additional word sense disambiguation in practical use.

Here, we propose a method to alleviate the adverse effects of polysemous words' paraphrases without word sense disambiguation. Our idea is to annotate each paraphrase with a degree about its reliability, like a member of a fuzzy set. We call such paraphrases as "fuzzy paraphrases", and their degrees as the "memberships."

2.2. LEARNING WITH FUZZY PARAPHRASES:

We also propose a novel method to jointly learn corpus with a lexicon, in order to use fuzzy paraphrases to improve the word vectors.

If the meanings of two words are totally the same, they can replace each other in a text without changing the semantic features. Henceforth, we can learn the lexicon by replacing the words in the corpus with its lexical paraphrases.

We learn the word vectors by maximizing the probability of a word for a given context, and also for a generated context where words are replaced by their paraphrases randomly. The memberships of the fuzzy paraphrases are used here to control the probability that the replacements occur by a control function as shown in Figure 1 .

For a text corpus T , denote w i the ith word in T , c the context window, w j a word in the context window, L wj the paraphrase set of w j in the lexicon L, w k the kth fuzzy paraphrase in L wj , and x jk the membership of w k for w j , the objective is

EQUATION

The function f (x jk ) of the membership x jk is a specified drop-out function. It returns 0 more for the paraphrases that have lower memberships, and 1 more for the others.

2.3. MEMBERSHIP ESTIMATION & CONTROL FUNCTION f (x):

Looking for a control function that is easy to train, we notice that if two words are more often to be translated to the same word in another language, the replacement of them are less likely to change the meaning of the original sentence. Thus, we use a function of the bilingual similarity (denoted as S jk ) as the membership function:

x jk = g(S jk ).

(2)

There have been works about calculating the similarity of words using such bilingual information.

A lexicon called the paraphrase database (PPDB) provides scores of the similarity of paraphrases on the basis of bilingual features (Ganitkevitch et al., 2013; Pavlick et al., 2015b; a) .

We scale the similarity score of the paraphrase w k to [0, 1] in PPDB2.0 as the memberships, and draw the values of f (x jk ) from a Bernoulli distribution subjected to them. Denote S jk the similarity score of word w j and w k in PPDB2.0, the value of f (x jk ) is drawn from the Bernoulli distribution:

EQUATION

x jk = S jk max j∈T,k∈L S jk .

(4)

2.4. TRAINING:

We do not need to train f (x jk ) using the method described above. The model can be trained by negative sampling (Mikolov et al., 2013b) : For word w O and a word w I in its context, denote A I as the set of the paraphrases for w I accepted by f (x jk ), we maximize log p(w O |w I ) by distinguishing the noise words from a noise distribution P n (w) from w O and its accepted paraphrases in A I by logistic regression:

EQUATION

Here, v w O T and v wi T stand for the transposed matrices of v w O and v wi , respectively. n is the number of negative samples used. σ(x) is a sigmoid function, σ(x) = 1/(1 + e -x ).

3. MODEL EXPLORATION:



3.1. CORPUS FOR EXPERIMENTS:

We use enwiki91 mainly for tuning and model exploration. It has a balanced size(1 GB), containing 123,353,508 tokens. It provides enough data to alleviate randomness while it does not take too much time for our model to learn. We use ukWaC (Baroni et al., 2009) to compare with the prior works in section 4. But we do not use it for model exploration, because it takes more than 20 hours to learn it, as an enormous corpus containing 12 GB text.

3.2. BENCHMARKS:

We used several benchmarks. They are Wordsim-353 (WS353) (Finkelstein et al., 2001) (353 word pairs), SimLex-999 (SimLex) (Hill et al., 2016) (999 word pairs), the Stanford Rare Word Similarity Dataset (RW) (Luong et al., 2013) (2034 word pairs), the MEN Dataset (MEN) (Bruni et al., 2014) (3000 word pairs), and the Mikolov's (Google's) word analogical reasoning task (Mikolov et al., 2013a) .

WS353, SimLex, and RW are gold standards. They provide the similarity of words labeled by humans. We report the Spearman's rank correlation (ρ) for them.

Mikolov's word analogical reasoning task is another widely used benchmark for word vectors. It contains a semantic part (SEM), and a syntactic part (SYN). We use the basic way suggested in their paper to find the answer for it: to guess word b related to b in the way how a is related to a, the word closest in cosine similarity to a -a + b is returned as b .

We find that the benchmark scores change every time we learn the corpus, even under the same settings. It is because that the models involve random numbers. Therefore we should consider the margin of error of the changes when we use the benchmarks.

To test the margin of error, we firstly used our proposed method to repeat learning enwiki9 for 10 times under the same parameters. Then we tested the vectors under each benchmark, to find the margin of error. In each test, we used the same parameters: the vector dimension was set to 100 for speed, the window size was set to 8, and 25 negative samples were used. The results are shown in Table 1 . We use them to analyze the other experimental results later.

3.3. DIFFERENT TYPES OF PARAPHRASES:

In PPDB2.0, there are six relationships for paraphrases. For word X and Y , the different relationships between them defined in PPDB2.0 are shown in Table 2 . We do not consider the exclusion and independent relations because they are not semantic paraphrases. Those of equivalence are the most reliable because they are the closest ones. But we still want to know whether it is better to take Table 2 : Different types of relationships of paraphrases in PPDB2.0 (Pavlick et al., 2015b; a) .

Relationship Type Description:

Equivalence X is the same as Y the entailment and the other related paraphrases into consideration. We learn enwiki9 with different paraphrase sets and use SimLex to evaluate the trained vectors.

Forward Entailment X is more specific than/is a type of Y Reverse Entailment X is more general than/encompasses Y Exclusion X is the opposite of Y / X is

Figure 2 compares the performance using different paraphrase sets, tested by SimLex. We can see that it is best to use the equivalence and entailment (forward + reverse) paraphrases together or use only the equivalence paraphrases. Only using the entailment paraphrases is weak. Involving the other related paraphrases deteriorates the performance. We use the Equivalence and Entailment paraphrases in the experiments according to these results.

3.4. EFFECTS OF PARAMETERS:

We use our proposed method to learn enwiki9 under different parameter settings to evaluate the effects of parameters. We firstly learn enwiki9 under different parameter settings and then test the vectors using SimLex, WS353, RW, MEN, SEM and SYN. We report Spearman's rank correlation ρ for SimLex, WS353, RW and MEN, the percentage of correct answers for SEM and SYN. Figure 3 : The scores of the benchmarks using different vector-space dimensions. For WS353, SimLex, RW and MEN, we report 100 * ρ (Spearman's rank correlation). For word analogical reasoning, we report the percentage of the correct answers. The context window size is set to 8. The number of negative samples is set to 25.

3.4.1. EFFECTS OF VECTOR SPACE DIMENSION:

We compare the benchmarks using different vector-space dimensions. Figure 3 shows the change of each benchmark's scores under different dimensions.

We find that:

• The larger vectors do not bring the better performance for most of the benchmarks (except SimLex), although some previous works suggest that the higher dimensions brings better performance for their methods (Pennington et al., 2014; Levy & Goldberg, 2014b) .

• The curves of SimLex and SYN are gradual. However, there are several abrupt changes in the others. And those of WS353 and RW do not change gradually.

• The best dimension for different benchmarks is not consistent.

The differences in the content of the benchmarks may cause the inconsistence. For example, SimLex rates related but dissimilar words lower than the other word similarity benchmarks (Hill et al., 2016; Chiu et al., 2016) . The results suggest that the best dimensions for our method depends on the task.

3.4.2. EFFECTS OF CONTEXT WINDOW SIZE:

We compared the benchmarks using different context window sizes. They are shown in Figure 4 . Previous works argue that larger window sizes introduce more topic words, and smaller ones emphasize word functions (Turney, 2012; Levy & Goldberg, 2014a; Levy et al., 2015; Hill et al., 2016; Chiu et al., 2016) . Different context window sizes provide different balances between relatedness and similarity. The best window size depends on what we want the vectors to be. We also see that in our results.

The relationship between the window size and performance depends on how they rate the pairs. For example, WS353 rates word pairs according to association rather than similarity (Finkelstein et al., 2001; Hill et al., 2016) . As larger window capture relatedness rather than similarity, the results show Figure 4 : The scores of the benchmarks using different context window sizes. For WS353, SimLex, RW and MEN, we report 100 * ρ (Spearman's rank correlation). For word analogical reasoning, we report the percentage of the correct answers. We use 100-dimension vectors. The number of negative samples is set to 25.

that the larger the window is, the better for WS353. The MEN dataset also prefer relatedness than similarity (Bruni et al., 2014) , but they gave annotators examples involving similarity2 . It may be the reason that the windows larger than 8 deteriorate the benchmarks based on MEN (Figure 4d ). The standards of WS353 and MEN to rate the words are similar (Bruni et al., 2014) . It leads to their similar curves (Figure 4b and 4d ). The worst window sizes of them are also close. When the window size is set to about 2 or 3, respectively, the balance of similarity and relatedness is the worst for them.

Unlike the other word similarity dataset, SimLex rates synonyms high and related dissimilar word pairs low. Therefore, the smallest window is the most suitable for SimLex because it is best for capturing the functional similarity.

The results of RW differs from the others (Figure 4c ). There are many abrupt changes. The best window size is 10, but 1 is better than 2-9. The dataset contains rare words. Because of their low frequencies, usage of broad context window may be better to draw features for them. However, additional words introduced by larger windows may also deteriorate the vectors of unusual words.

For such tasks requiring rare word vectors of high quality, we should be careful in tuning the context window size.

For Google's word analogical tasks (SEM and SYN), the questions are quite related to the topic or domain. For examples, there are questions about the capitals of the countries. They are associated but not synonymous. Therefore a larger window is usually better. However for SYN, using window size 9 is a little better than 10 in Figure 4d and for MEN 8 is best in Figure 4f . It may be because that if the window is too large, it introduces too many words and reduces the sparsity (Chiu et al., 2016) .

We can consider that the best context window size depends on the task, but we should avoid using too large window.

3.4.3. EFFECTS OF NEGATIVE SAMPLES:

We also explored the effects of the number of negative samples. The results are shown in Figure 5 . Figure 5 : The scores of the benchmarks using different numbers of negative samples. For WS353, SimLex, RW and MEN, we report 100 * ρ (Spearman's rank correlation). For word analogical reasoning, we report the percentage of the correct answers. We use 100-dimension vectors. The context window size is set to 8.

In Figures 5a, 5c and 5f, we see that overfitting occurs when we use more than 15 negative samples.

In Figure 5b and Figure 5e , it occurs from 25 and 20, respectively. In Figure 5d , the performance does not change very much when we use more than 30 negative samples.

The results indicate that too many negative samples may cause overfitting. For 3 of the 6 benchmarks, it is best to use 15 negative samples. But we should be careful in practice use because the other different results suggest that the best number depends on the task.

The abrupt change at around 15 in Figure 5b is interesting. WS353 is the smallest dataset among those we used. Because of the small size, the effects of randomness may cause such singularities when the vector-space is not well trained.

3.5. EFFECTS OF THE CONTROL FUNCTION & THE CORPUS SIZE:

In this section, we evaluate the effectiveness of our fuzzy approach, by comparing to the situations that set f (x) in Equation (1) as:

• f (x) = 1: It makes the model regard all paraphrases equally. They are all used without drop-out. • f (x) = 0: It makes the model use no paraphrases, equivalent to CBOW.

It is also a good way to show the effects of corpus size by comparing the proposed method to the situations above using corpora in varying size. Therefore we discuss them together in this section.

We use text83 together with eEnwiki9 and ukWaC described in section 3.1. It is a small corpus containing 100 MB text. To show the difference, we report the benchmarks scores including not only SimLex, but also MEN, and the word analogical task (SEM and SYN). They are the other benchmarks that are shown relatively solid in section 3.2. The vector-space dimension is set to 300. The context window size is set to 8. 25 negative samples are used in learning. The results are shown in Figure 6 . We can see that:

• The proposed function outperforms the others for SimLex and MEN under text8, for all the benchmarks under enwiki9, for SimLex, SEM and SYN under ukWaC.

• The proposed function is always better than f (x) = 1 in the experiments, no matter what the benchmark is or how big the corpus is.

• For SEM, the proposed function is weaker than f (x) = 0 under text8, slightly better under enwiki9, and obviously outperforms f (x) = 0 under ukWaC. As the proposed function outperforms under larger corpora, the relatively low scores under text8 may be caused by the effects of randomness: the proposed function involves random numbers; they bring huge instability under such tiny corpora. Another possible reason is that the control function is less useful for text8 because there are few polysemous words in the tiny corpus.

• There is no advantages to use f (x) = 1 instead of f (x) = 0 for both text8 and enwiki9. It shows that learning the context words replaced by paraphrases may be not a good idea without fuzzy approaches. However, if we use the proposed control function, the results are better and go beyond those of f (x) = 0 in most tests. It shows that the control function utilizing fuzzy paraphrases improves the performance.

Therefore, we can see that the proposed control function using the fuzzy paraphrases annotated with the degrees of reliability improves the quality of the learned word vector-space.

4. COMPARISON WITH THE PRIOR WORKS:

We compared our work to the prior works using a lexicon to improve word vectors. However, we failed to use the public code to reproduce the works of Yu & Dredze (2014) and Bollegala et al. (2016) . We also failed to find an available implementation of Xu et al. (2014) . Hence, we use the 2016) and compare our results with the reported scores of the prior works in their paper. The benchmarks are:

• The MEN Dataset (MEN);

• Word Analogical Reasoning Task (SEM and SYN).

Rubenstein-Goodenough dataset (RG) (Rubenstein & Goodenough, 1965) is also used in their works. However, we do not use it, because it fails the sanity check in Batchkarov et al. (2016) : ρ may increase when noise is added.

We use ukWaC to learn the word vectors, the same with Bollegala et al. (2016) . We also use the same parameters with the prior works: The vector-space dimension is set to 300; the context window size is set to 8; the number of negative samples is set to 25. Then we calculate the cosine similarity of the words and report 100 * ρ for Men. We use the add method described in section 3.2 and report the percentage of correct answers, for the word analogical reasoning task.

Table 3 shows the results of the experiments. The of MEN and SEM is 0.86 and 0.44 as shown in Table 1 . Therefore we see that our proposed method outperforms the prior works under these benchmarks. We consider our score for SYN is as good as Bollegala et al. (2016) achieved, and better than the others, because its margin of error is 1.79 as shown in Table 1 .

5. CONCLUSION & THE FUTURE WORKS:

We proposed a fuzzy approach to control the contamination caused by the polysemous words when a lexicon is used to improve the vector-space word representations. We annotate each paraphrase of a word with a degree of reliability, like the members of a fuzzy set with their memberships, on the basis of their multilingual similarities to the original ones. We use the fuzzy paraphrases to learn a corpus by jointly learning a generated text, in which the original words are randomly replaced by their paraphrases. A paraphrase is less likely to be put into the generated text if it has lower reliability than the others, and vice versa.

We tested the performance using different types of paraphrases in the lexicon PPDB2.0 and find that it is best to use the equivalence type and the entailment type. Using other related paraphrases deteriorates the performance.

We explored the effects of parameters. We find that the best parameter setting depends on the task.

We should tune the model carefully in practical use.

We evaluated the effectiveness of our approach by comparing it to the situations that simpler functions are used to control replacements: f (x) = 1 which accepts all, and f (x) = 0 which rejects all. We also repeated the experiments under a tiny, a medium sized, and a large corpus, to see the effects of the corpus size on the effectiveness. Our approach achieves the best in 3 of 4 benchmarks under the tiny corpus, and in all benchmarks under the medium sized and the large one. The results indicate that our approach is effective to improve the word vectors.

Our proposed method also achieved the top scores, compared with the prior works.

Unlike the previous works that solve the problems about polysemy by estimating a vector for each word sense or word type, our approach keeps one vector per word. It makes the word vectors easier to use in practical terms: it is neither necessary to disambiguate the word senses nor to tag the part-of-speeches before we use the word vectors.

The fuzzy paraphrases can also be employed for the other models with some changes. We are going to show it in the future. The proposed idea for the polysemy problem without word sense disambiguation is meaningful especially for practical use because it saves the effort of part-of-speech tagging and word sense disambiguation.

Besides, the control function may be more accurate if it considers all the context. We are also going to work on it in the future.

We have opened the source of a demo of the proposed method online4 .

http://mattmahoney.net/dc/enwiki9.zip

According to their homepage: http://clic.cimec.unitn.it/ elia.bruni/MEN.html.

http://mattmahoney.net/dc/text8.zip

https://github.com/huajianjiu/Bernoulli-CBOFP

Figures and Tables:

Figure 1: The process flow of the proposed method.

Figure 6: The comparison of using the proposed function described in section 2.3, f (x) = 0 (equivalent to CBOW) and f (x) = 1 (no drop-out) as the control function. They are compared under different corpora in varying size. The green bar (the left) indicates the scores of the proposed function; the blue bar (the middle) indicates the scores of f (x) = 0; the pink bar (the right) indicates the scores of f (x) = 1. We report 100 * ρ for SimLex and MEN, the percentage of correct answers for SEM and SYN. The vector-space dimension is set to 300. The context window size is set to 8. 25 negative samples are used in learning.

The results of 10 times repeated learning and test under each benchmark. The vector-space dimension is set to 100. Enwiki9 is used as the corpus. The maximum, minimum, and the margin of error are marked bold.

Table content: <table><tr><td colspan="4">Benchmark SimLex WS353 RW</td><td>MEN</td><td>SEM</td><td>SYN</td></tr><tr><td>1</td><td>29.41</td><td>62.02</td><td colspan="4">38.12 60.00 13.26 27.77</td></tr><tr><td>2</td><td>29.57</td><td>62.49</td><td colspan="3">38.26 60.39 12.70</td><td>27.27</td></tr><tr><td>3</td><td>29.48</td><td>61.04</td><td colspan="4">39.90 59.80 13.89 26.94</td></tr><tr><td>4</td><td>29.52</td><td>60.20</td><td colspan="3">39.68 59.81 14.02</td><td>27.11</td></tr><tr><td>5</td><td>28.69</td><td>63.45</td><td colspan="4">38.65 60.16 12.94 26.87</td></tr><tr><td>6</td><td>29.26</td><td>61.95</td><td colspan="4">39.13 59.73 13.75 26.60</td></tr><tr><td>7</td><td>29.46</td><td>62.90</td><td colspan="2">39.12 60.45</td><td colspan="2">13.42 25.98</td></tr><tr><td>8</td><td>28.51</td><td>62.96</td><td colspan="2">37.93 59.31</td><td colspan="2">13.58 27.10</td></tr><tr><td>9</td><td>29.13</td><td>62.44</td><td>39.91</td><td colspan="3">59.75 13.98 26.89</td></tr><tr><td>10</td><td>28.59</td><td>60.66</td><td colspan="4">38.67 60.24 13.66 26.98</td></tr><tr><td>Margin of Error</td><td>0.98</td><td>2.41</td><td>1.98</td><td>1.14</td><td>1.32</td><td>1.79</td></tr></table>

mutually exclusive with Y

Table content: <table><tr><td>OtherRelated</td><td>X is related in some other way to Y</td></tr><tr><td>Independent</td><td>X is not related to Y</td></tr><tr><td>37.10</td><td/></tr><tr><td>37.05</td><td/></tr><tr><td>37.00</td><td/></tr><tr><td>36.95</td><td/></tr><tr><td>36.90</td><td/></tr><tr><td>36.85</td><td/></tr><tr><td>36.80</td><td/></tr><tr><td>36.75</td><td/></tr><tr><td>36.70</td><td/></tr><tr><td>36.65</td><td/></tr></table>

Comparison to the prior Works. The scores of the prior works under ukWaC are fromBollegala et al. (2016). The SYN score of ours and Bollegala's are marked as best together because the margin of error is 1.79 as shown in Table1.

Table content: <table><tr><td>Method</td><td>MEN</td><td>SEM</td><td>SYN</td></tr><tr><td>Our Proposed Method</td><td colspan="3">76.99 67.48 67.89</td></tr><tr><td>Bollegala et al. (2016)</td><td colspan="3">70.90 61.46 69.33</td></tr><tr><td>Yu &amp; Dredze (2014)</td><td>50.10</td><td>-</td><td>29.90</td></tr><tr><td>R-Net (Xu et al., 2014)</td><td>-</td><td colspan="2">32.64 43.46</td></tr><tr><td>C-Net (Xu et al., 2014)</td><td>-</td><td colspan="2">37.07 40.06</td></tr><tr><td>RC-Net (Xu et al., 2014)</td><td>-</td><td colspan="2">34.36 44.42</td></tr><tr><td>Faruqui et al. (2015)(Pretrained by CBOW)</td><td colspan="3">60.50 36.65 52.50</td></tr><tr><td colspan="4">Faruqui et al. (2015)(Pretrained by Skipgram) 65.70 45.29 65.65</td></tr><tr><td>same corpus and benchmarks with Bollegala et al. (</td><td/><td/><td/></tr></table>

Peer review (iclr2017):
This paper proposes a method for estimating the context sensitivity of paraphrases and uses that to inform a word embedding learning model. The main idea and model are presented convincingly and seem plausible. The main weaknesses of the paper are shortcomings in the experimental evaluation and in the model exploration. The evaluation does not convincingly determine whether the model is a significant improvement over simpler methods (particularly those that do not require the paraphrase database!). Likewise, the model section did not convince me that this was the most obvious model formulation to try. The paper would be stronger if model choices were explained more convincingly or - better yet - alternatives were explored. On balance I lean towards rejecting the paper and encouraging the authors to submit a revised and improved version at a near point in the future. Detailed/minor points below: 1) While the paper is grammatically mostly correct, it would benefit from revision with the help of a native English speaker. In its current form long sections are very difficult to understand due to the unconventional sentence structure. 2) The tables need better and more descriptive labels. 3) The results are somewhat inconclusive. Particularly in the analogy task in Table 4 it is surprising that CBOW does better on the semantic aspect of the task than your embeddings which are specifically tailored to be good at this? 4) Why was *Enriched CBOW* not included in the analogy task? 5) In the related work section several papers are mentioned that learn embeddings from a combination of lexica and corpora, yet it is repeatedly said that this was the first work of such a kind / that there hasn*t been enough work on this. That feels a little misleading.