Title: Fuzzy paraphrases in learning word representations with a lexicon
Authors: Yuanzhi Ke
Masafumi Hagiwara
Venue: ICLR 2017
Keywords: Natural language processing
Unsupervised Learning

Abstract: A synonym of a polysemous word is usually only the paraphrase of one sense among many. When lexicons are used to improve vector-space word representations, such paraphrases are unreliable and bring noise to the vector-space. The prior works use a coefficient to adjust the overall learning of the lexicons. They regard the paraphrases equally. 
In this paper, we propose a novel approach that regards the paraphrases diversely to alleviate the adverse effects of polysemy. We annotate each paraphrase with a degree of reliability. The paraphrases are randomly eliminated according to the degrees when our model learns word representations. In this way, our approach drops the unreliable paraphrases, keeping more reliable paraphrases at the same time. The experimental results show that the proposed method improves the word vectors.
Our approach is an attempt to address the polysemy problem keeping one vector per word. It makes the approach easier to use than the conventional methods that estimate multiple vectors for a word. Our approach also outperforms the prior works in the experiments.

TL;DR: We propose a novel idea to address polysemy problem by annotating paraphrases with a degree of reliability like a member of a fuzzy set.

Review discussion:
Compared to JointREPs, your method achieves slightly better overall accuracy on the analogy task, but it performs much better on the semantic category than it does on the syntactic one. Do you have an explanation for this imbalance?
Dear AnonReviewer1:
Hello. Thanks for your comment.
Each question in the semantic part of the word analogical reasoning task contains two pairs of words sharing the same semantic relationship, for example, ("boy", "girl") and ("brother, sister"). Each question in the syntactic part contains two pairs of words sharing the syntactic relationship, for example, ("amazing", "amazingly") and ("calm", "calmly").
The JointReps by Bollegala et al. minimizes the distance of the target word and one of its context words in the vector space if they are paraphrases in the lexicon. For polysemous words, it leads to overfitting to a wrong vector of a wrong sense. Because it is an issue about representing the semantic relationship, it deteriorates the accuracy for the semantic part and much less for the syntactic part.
Our method alleviates the issue about overfitting to the wrong sense.
Therefore, we get an improvement in the semantic part.
As the previous works using lexicons report improvement in the syntactic sense, we can see that the paraphrases in the lexicons improve the accuracy in syntactic sense.
However, our method drops some of the paraphrases in learning the word embeddings.
That is one of the reasons that our method is less efficient in improving the accuracy of the syntactic part.
Besides, the reported results of JointReps are achieved using a different corpus called UkWaC. It is another possible reason of the difference.
This paper introduces the concept of fuzzy paraphrases to aid in the learning of distributed word representations from a corpus augmented by a lexicon or ontology. Sometimes polysemy is context-dependent, but prior approaches have neglected this fact when incorporating external paraphrase information during learning. The main idea is to introduce a function that essentially judges the context-sensitivity of paraphrase candidates, down-weighting those candidates that depend strongly on context. This function is inferred from bilingual translation agreement.
The main argumentation leading to the model selection is intuitive, and I believe that the inclusion of good paraphrases and the elimination of bad paraphrases during training should in principle improve word representation quality.
However, the main questions are how well the proposed method achieves this goal, and, even if it achieves it well, whether it makes much difference in practical terms.
Regarding the first question, I am not entirely convinced that the parameterization of the control function f(x_ij) is optimal. It would have been nice to see some experiments investigating different choices, in particular some baselines where the effect of f is diminished (so that it reduces to f=1 in the limit) would have been interesting. I also feel like there would be a lot to gain from having f be a function of the nearby word embeddings, though this would obvious incur a significant slowdown. (See for example 'Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space' by Neelakantan et al, which should probably be cited.) As it stands, the experimental results do not clearly distinguish the fuzzy paraphrase approach from prior work, i.e. tables 3 and 4 do not show major trends one way or the other.
Regarding the second question, it is hard to draw many conclusions from analogy tasks alone, especially when effects unrelated to good/bad paraphrasing such as corpus size/content, window size, vocabulary size, etc., can have an outsize effect on performance.
Overall, I think this is a good paper presenting a sensible idea, but I am not convinced by the experiments that the specific approach is achieving its goal. With some improved experiments and analysis, I would wholeheartedly recommend this paper for acceptance; as it stands, I am on the fence.
Thank you for your careful review, kindly comments and valuable advice.
We are glad that you like our idea. According to the issues you mentioned and your valuable advice, we will perform additional experiments add detailed and further discussions:
1. "how well the proposed method achieves this goal, and, even if it achieves it well, whether it makes much difference in practical terms"
Thank you for the comment. We apologize for the unclear discussion of the motivations and contributions in the paper.
We think the major contribution of this work is to propose an idea that alleviates the bad effects of polysemous words and improve the performance, but not involves word sense vectors and word disambiguation. Word disambiguation is therefore not necessary when we use the estimated word embeddings in other works.
We will revise the paper and give statements and discussions about the motivations and contributions.
2. "I am not entirely convinced that the parameterization of the control function f(x_ij) is optimal."
Thank for your valuable comments and advice.
As you say, it is interesting and we are going to investigate different choices of function f and compare the results.
3. "the experimental results do not clearly distinguish the fuzzy paraphrase approach from prior work"
Thank you for your kindly comments.
We think the word analogy is not enough to distinguish the proposed method from prior works. We are going to use more benchmarks including WordSim353 and SimLex999. After we collect the results, we will add them in the paper and give you a revision list.
4. "it is hard to draw many conclusions from analogy tasks alone, especially when effects unrelated to good/bad paraphrasing such as corpus size/content, window size, vocabulary size, etc., can have an outsize effect on performance. "
Thank you for your kindly comments and advice.
We will use more benchmarks including WordSim353 and SimLex999 to evaluate our method. We will also compare the results at different situations including different corpora, window sizes, vector size, numbers of samples, versions of the paraphrase database.
After we collect the results, we will revise the paper and report them. We will also post a revision list for you.
Thank you again for your kindly comments and valuable advice.
Thank you again for your kindly reviews.
We have revised our paper.
For the first question how well the proposed method achieves the goal, we added the comparison with simpler control function to evaluate the effectiveness of our approach in section 3.5. We also evaluate the methods using more benchmarks.
Besides, the benchmarks' margins of errors are tested in section 3.2; they are used to analyze whether the experimental results indicate the proposed method performs better than the prior works.
For the second question, we explored the effects of the corpus, paraphrase sets, the dimension of word vectors, the context window size and the number of negative samples. We show the results and give discussions in section 3.3, 3,4 and 3.5.
Please check the full list of revisions here:
1)　We rewrote the abstract and tried to avoid misleading utterances.
2)　We rewrote the parts that may mislead or confuse readers in section 1.
3)　We added lists of contributions and sections at the end of section 1.
4)　We deleted the "Related Works" section and put the introduction about them in section 1.
5)　We rewrote section 2  to make it more logical and easier to read. We introduce about the ideas of fuzzy paraphrases in a separated subsection in section 2.1 now.
6)　We replaced the figure of the architecture with a process flow chart in section 2, which is easier to understand.
7)　We moved the discussion about a control function considering all context from section 2 to section 5 as the future work.
8)　We performed some additional experiments to explore the effects of paraphrase types, parameters, and the control function. We discuss them in the new section 3 "Model Exploration."
9)　We added experiments about the benchmarks to see how they can evaluate the performance. We tested their margins of error when we repeat learning under the same conditions. We provide the results and discussion in section 3.2.
10)　We used an additional large corpus -- ukWaC for exploration about the effects of different corpus sizes to show the effects of the corpus size more clearly. We describe the experiments and discuss the results in section 3.5.
11)　 We added comparison with the situation that f(x)=1 (regards every paraphrase equally) and f(x)=0 (equivalent to CBOW) in  section 3.5.
12)　 We redid the comparative experiments for comparison with the prior works using the same corpus and benchmarks to make it fairer. We describe the experiment method and discuss the results in section 4.
13)　 We deleted the discussion of processing speed because it is less related to the major contributions and the paper is longer.
14)　We rewrote the conclusion in section 5. We avoided the misleading utterances such as "we figure out."
Thank you again for your review and kindly advice.
Hello. Thank you again for your reviews.
We have made some additional revisions as our final version before the decision.
1. We realized that the benchmarks are testing different aspects of the vectors. We do not think the inconsistency for best parameters is an issue now. We put the changes of the benchmark scores under different vector dimensions from section 3.2 to 3.3 as it is about the effect of vector dimensions. We also revised the title of section 3.2 from "Issues about benchmarks" to "Benchmarks."
2. We used our proposed model instead of fastText to test the margin of errors for the benchmarks and revised section 3.2.
3. We also revised the discussion on the parameters and gave a more detailed discussion in section 3.
4. We revised the y labels in the figures to keep the same float format.
5. We add an explanation of the transpose mark for Equation (5) in section 2.4.
6. We add the missed introduction of Figure 2 in section 3.3.
We recheck the misspellings and grammar mistakes.
Thank you again for spending the time to read our papers and the helpful advice.
This paper tries to leverage an external lexicon / knowledge base to improve corpus-based word representations by determining (in a fuzzy way) which potential paraphrase is the most appropriate in a particular context.
I think this paper is a bit lost in translation. The grammatical and storytelling styles made it really difficult for me to concentrate, and even unintelligible at times. One of the most important criteria in a conference paper is to communicate one's ideas clearly; unfortunately, I do not feel that this paper meets that standard.
In addition, the evaluation is rather lacking. There are many ways to evaluate word representations, and Google's analogy dataset has many issues (see, for example, Linzen's paper from RepEval 2016, as well as Drozd et al., COLING 2016).
Finally, this work does not provide any qualitative result or motivation. Why does this method work better? Where does it fail? What have we learned about word representations / lexicons / corpus-based methods in general?
Thank you for your comments and the issues pointed out.
We are sorry for our poor English. We will recheck the paper and try our best to make it clearer and easier to read and understand.
Thank you very much for pointing out the issues of Google's analogy dataset. We will use the other datasets including WordSim353 and SimLex999 to perform additional evaluations.
The motivations of this work are:
1. We noticed that the previous works (that use lexicons to improve word embeddings) do not improve the accuracy in the semantic part of Google's analogy dataset.
As you mentioned, the Google's analogy dataset has many issues.
But we think there are unsolved issues in the previous works too. We think one of them is the issue of polysemous words.
2. We find that although there are works in word disambiguation for learning word embeddings, they result in one vector per sense.
If we want to use such word embeddings in other works, we need to do word disambiguation first. The additional word disambiguation may bring additional error.
3. We think it is contributing to propose an idea that alleviates the bad effects of polysemous words, without estimating one vector per sense.
It is easier to use in other works when one vector per word because it is not necessary to do word disambiguation.
If we can alleviate the bad effects of polysemous words and keep one vector per word at the same time without word disambiguation. We think it is contributing.
Sorry for our poor English confused you and thank you for your comments again.
Thank you again for your comments.
We apologize again for the poor English in the old version. We wrote the most of the paper. I hope the current form is easier to read and understand.
We have added more benchmarks in evaluation experiments. We also thank for your advice that recommends papers in RepEval 2016 for us. Inspired by the articles, we tested the benchmarks themselves and their margins of error when the same experiment repeats. Benefitting from the results, we analyze the other experimental results more fairly now. We put the findings about the benchmarks in section 3.2 "Issues about Benchmarks".
We list the major contributions at the end of section 1 "Introduction", rewrote the discussion in section "Conclusion & The Future Works" to clarify the motivations and contributions.
The full list of revisions is here:
1)　We rewrote the abstract and tried to avoid misleading utterances.
2)　We rewrote the parts that may mislead or confuse readers in section 1.
3)　We added lists of contributions and sections at the end of section 1.
4)　We deleted the "Related Works" section and put the introduction about them in section 1.
5)　We rewrote section 2  to make it more logical and easier to read. We introduce about the ideas of fuzzy paraphrases in a separated subsection in section 2.1 now.
6)　We replaced the figure of the architecture with a process flow chart in section 2, which is easier to understand.
7)　We moved the discussion about a control function considering all context from section 2 to section 5 as the future work.
8)　We performed some additional experiments to explore the effects of paraphrase types, parameters, and the control function. We discuss them in the new section 3 "Model Exploration."
9)　We added experiments about the benchmarks to see how they can evaluate the performance. We tested their margins of error when we repeat learning under the same conditions. We provide the results and discussion in section 3.2.
10)　We used an additional large corpus -- ukWaC for exploration about the effects of different corpus sizes to show the effects of the corpus size more clearly. We describe the experiments and discuss the results in section 3.5.
11)　 We added comparison with the situation that f(x)=1 (regards every paraphrase equally) and f(x)=0 (equivalent to CBOW) in  section 3.5.
12)　 We redid the comparative experiments for comparison with the prior works using the same corpus and benchmarks to make it fairer. We describe the experiment method and discuss the results in section 4.
13)　 We deleted the discussion of processing speed because it is less related to the major contributions and the paper is longer.
14)　We rewrote the conclusion in section 5. We avoided the misleading utterances such as "we figure out."
We thank for your review and apologize again for our poor English and the confusing utterances in the old version.
Hello. Thank you again for your reviews.
We have made some additional revisions as our final version before the decision.
1. We realized that the benchmarks are testing different aspects of the vectors. We do not think the inconsistency for best parameters is an issue now. We put the changes of the benchmark scores under different vector dimensions from section 3.2 to 3.3 as it is about the effect of vector dimensions. We also revised the title of section 3.2 from "Issues about benchmarks" to "Benchmarks."
2. We used our proposed model instead of fastText to test the margin of errors for the benchmarks and revised section 3.2.
3. We also revised the discussion on the parameters and gave a more detailed discussion in section 3.
4. We revised the y labels in the figures to keep the same float format.
5. We add an explanation of the transpose mark for Equation (5) in section 2.4.
6. We add the missed introduction of Figure 2 in section 3.3.
We recheck the misspellings and grammar mistakes.
Thank you again for spending the time to read our papers and the helpful advice.
This paper proposes a method for estimating the context sensitivity of paraphrases and uses that to inform a word embedding learning model. The main idea and model are presented convincingly and seem plausible. The main weaknesses of the paper are shortcomings in the experimental evaluation and in the model exploration. The evaluation does not convincingly determine whether the model is a significant improvement over simpler methods (particularly those that do not require the paraphrase database!).
Likewise, the model section did not convince me that this was the most obvious model formulation to try. The paper would be stronger if model choices were explained more convincingly or - better yet - alternatives were explored.
On balance I lean towards rejecting the paper and encouraging the authors to submit a revised and improved version at a near point in the future.
Detailed/minor points below:
1) While the paper is grammatically mostly correct, it would benefit from revision with the help of a native English speaker. In its current form long sections are very difficult to understand due to the unconventional sentence structure.
2) The tables need better and more descriptive labels.
3) The results are somewhat inconclusive. Particularly in the analogy task in Table 4 it is surprising that CBOW does better on the semantic aspect of the task than your embeddings which are specifically tailored to be good at this?
4) Why was "Enriched CBOW" not included in the analogy task?
5) In the related work section several papers are mentioned that learn embeddings from a combination of lexica and corpora, yet it is repeatedly said that this was the first work of such a kind / that there hasn't been enough work on this. That feels a little misleading.
Thank you for your comments and valuable advice.
As you say, there are confusing and misleading statements in the paper, and the experiment results are not convincing enough.
We will rewrite the confusing and misleading parts. We will also perform further evaluations, compare the results under different parameters and difference f function.
After we get all done, we will post the revised version and the revision list.
1) While the paper is grammatically mostly correct, it would benefit from revision with the help of a native English speaker. In its current form long sections are very difficult to understand due to the unconventional sentence structure.
We are sorry for our poor English. We will recheck the paper and make it clearer to understand.
2) The tables need better and more descriptive labels.
Thank you for your advice. We will rewrite the labels and reform the tables.
3) The results are somewhat inconclusive. Particularly in the analogy task in Table 4 it is surprising that CBOW does better on the semantic aspect of the task than your embeddings which are specifically tailored to be good at this?
Thank you for pointing out the issue.
Because the negative sampling and the f function in our methods are stochastic, we think it is reasonable that the proposed method is less efficient when we use a smaller corpus.
But as you say, the experiments are not convincing enough and inconclusive. We are going to perform additional experiments using the other benchmarks like WordSim353 and SimLex999. We are also going to repeat the experiments several times to alleviate the randomness.
4) Why was "Enriched CBOW" not included in the analogy task?
We are sorry that our tables are confusing but we report "Enriched CBOW" in Table 2. We will rewrite the labels and reform the tables to make them look less similar and less confusing.
5) In the related work section several papers are mentioned that learn embeddings from a combination of lexica and corpora, yet it is repeatedly said that this was the first work of such a kind / that there hasn't been enough work on this. That feels a little misleading.
We are sorry for our poor English and the misleading statements.
We do not intend to say the proposed method is the first work of using lexicons or word disambiguation.
We want to express that our work  has the following features at the same time:
1. Use lexicons to improve the word embeddings
2. Alleviate the bad effects of polysemous words without additional word disambiguation
3. Keep one vector for one word.
So when we use the vector table, we do not need to do word disambiguation first.
We know that we are not first in using lexicons, nor working in word disambiguation for word embeddings, nor generate one vector per word.
We propose a new idea that alleviates the bad effects of polysemous words, keeping one vector per word without additional word disambiguation.
But as you say, some statements are misleading. We will rewrite the misleading parts.
After we revise the paper, we will post the revision list.
Thank you very much again for your kindly comments and valuable advice.
Thank you again for your comments.
We have revised the paper.
1)For the first point, "long sections are very difficult to understand due to the unconventional sentence structure," we have rewritten most parts of the paper. We use shorter paragraphs now and more lists now.
2)For the second point, "the tables need better and more descriptive labels," we have reformed the table of the comparative experiments with the prior works (now Table 3). We label each prior works with reference rather than abbreviation now.
3)For the third point about the inconclusive results, we have given more results under different benchmarks now. We also test the benchmarks themselves. We test their margin of errors when the experiments repeat and use the findings in the discussion of the other experimental results.
4)For the absence of "Enriched CBOW", we put it in the old version but now remove it because it is not an approach using lexicons to improve word vectors, a work in a different direction and less related to our works. Comparing with it is confusing. We also remove the results under GloVe for the same reason.
5)For the fifth point of the misleading utterances, we are sorry for them and have rewritten or removed the misleading sentences.
Please check the full list of revisions:
1)　We rewrote the abstract and tried to avoid misleading utterances.
2)　We rewrote the parts that may mislead or confuse readers in section 1.
3)　We added lists of contributions and sections at the end of section 1.
4)　We deleted the "Related Works" section and put the introduction about them in section 1.
5)　We rewrote section 2  to make it more logical and easier to read. We introduce about the ideas of fuzzy paraphrases in a separated subsection in section 2.1 now.
6)　We replaced the figure of the architecture with a process flow chart in section 2, which is easier to understand.
7)　We moved the discussion about a control function considering all context from section 2 to section 5 as the future work.
8)　We performed some additional experiments to explore the effects of paraphrase types, parameters, and the control function. We discuss them in the new section 3 "Model Exploration."
9)　We added experiments about the benchmarks to see how they can evaluate the performance. We tested their margins of error when we repeat learning under the same conditions. We provide the results and discussion in section 3.2.
10)　We used an additional large corpus -- ukWaC for exploration about the effects of different corpus sizes to show the effects of the corpus size more clearly. We describe the experiments and discuss the results in section 3.5.
11)　 We added comparison with the situation that f(x)=1 (regards every paraphrase equally) and f(x)=0 (equivalent to CBOW) in  section 3.5.
12)　 We redid the comparative experiments for comparison with the prior works using the same corpus and benchmarks to make it fairer. We describe the experiment method and discuss the results in section 4.
13)　 We deleted the discussion of processing speed because it is less related to the major contributions and the paper is longer.
14)　We rewrote the conclusion in section 5. We avoided the misleading utterances such as "we figure out."
We thank again for your review and apologize again for our poor English and the misleading sentences in the old version.
Hello. Thank you again for your reviews.
We have made some additional revisions as our final version before the decision.
1. We realized that the benchmarks are testing different aspects of the vectors. We do not think the inconsistency for best parameters is an issue now. We put the changes of the benchmark scores under different vector dimensions from section 3.2 to 3.3 as it is about the effect of vector dimensions. We also revised the title of section 3.2 from "Issues about benchmarks" to "Benchmarks."
2. We used our proposed model instead of fastText to test the margin of errors for the benchmarks and revised section 3.2.
3. We also revised the discussion on the parameters and gave a more detailed discussion in section 3.
4. We revised the y labels in the figures to keep the same float format.
5. We add an explanation of the transpose mark for Equation (5) in section 2.4.
6. We add the missed introduction of Figure 2 in section 3.3.
We recheck the misspellings and grammar mistakes.
Thank you again for spending the time to read our papers and the helpful advice.

Meta-review:
role: PC
data: comment: The reviewers agree that the paper's clarity and experimental evaluation can be improved.

Decision: Reject