Question: In what condition would distributional methods perform relatively poorly?
Evidence: Measures of semantic similarity and relatedness quantify the degree to which two concepts are similar (e.g., INLINEFORM0 – INLINEFORM1 ) or related (e.g., INLINEFORM2 – INLINEFORM3 ). Semantic similarity can be viewed as a special case of semantic relatedness – to be similar is one of many ways that a pair of concepts may be related. The automated discovery of groups of semantically similar or related terms is critical to improving the retrieval BIBREF0 and clustering BIBREF1 of biomedical and clinical documents, and the development of biomedical terminologies and ontologies BIBREF2 .
There is a long history in using distributional methods to discover semantic similarity and relatedness (e.g., BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 ). These methods are all based on the distributional hypothesis, which holds that two terms that are distributionally similar (i.e., used in the same context) will also be semantically similar BIBREF7 , BIBREF8 . Recently word embedding techniques such as word2vec BIBREF9 have become very popular. Despite the prominent role that neural networks play in many of these approaches, at their core they remain distributional techniques that typically start with a word by word co–occurrence matrix, much like many of the more traditional approaches.
However, despite these successes distributional methods do not perform well when data is very sparse (which is common). One possible solution is to use second–order co–occurrence vectors BIBREF10 , BIBREF11 . In this approach the similarity between two words is not strictly based on their co–occurrence frequencies, but rather on the frequencies of the other words which occur with both of them (i.e., second order co–occurrences). This approach has been shown to be successful in quantifying semantic relatedness BIBREF12 , BIBREF13 . However, while more robust in the face of sparsity, second–order methods can result in significant amounts of noise, where contextual information that is overly general is included and does not contribute to quantifying the semantic relatedness between the two concepts.
Our goal then is to discover methods that automatically reduce the amount of noise in a second–order co–occurrence vector. We achieve this by incorporating pairwise semantic similarity scores derived from a taxonomy into our second–order vectors, and then using these scores to select only the most semantically similar co–occurrences (thereby reducing noise).
We evaluate our method on two datasets that have been annotated in multiple ways. One has been annotated for both similarity and relatedness, and the other has been annotated for relatedness by two different types of experts (medical doctors and medical coders). Our results show that integrating second order co–occurrences with measures of semantic similarity increases correlation with our human reference standards. We also compare our result to a number of other studies which have applied various word embedding methods to the same reference standards we have used. We find that our method often performs at a comparable or higher level than these approaches. These results suggest that our methods of integrating semantic similarity and relatedness values have the potential to improve performance of purely distributional methods.
However, despite these successes distributional methods do not perform well when data is very sparse (which is common).
Answer: Distributional methods do not perform well when data is very sparse.