Learning Features from Co -occurrences: A
Theoretical Analysis
Yanpeng Li
IBM T. J. Watson Research Center
Yorktown Heights, New York 10598
liyanpeng.lyp@gmail.com

Abstract
Representing a word by its co-occurrences with other words in context is an
effective way to capture the meaning of the word. However, the theory
behind remains a challenge. In this work, taking the example of a word
classification task, we give a theoretical analysis of the approaches that
represent a word X by a function f(P(C|X)), where C is a context feature,
P(C|X) is the conditional probability estimated from a text corpus, and the
function f maps the co-occurrence measure to a prediction score. We
investigate the impact of context feature C and the function f. We also
explain the reasons why using the co-occurrences with multiple context
features may be better than just using a single one. In addition, some of the
results shed light on the theory of feature learning and machine learning in
general.

1

In trod u cti on

In natural language processing (NLP) and information retrieval (IR), representing a word by
its co-occurrences with contexts is an effective way to learn the meaning o f the word and
lead to significant improvement in many tasks [1][2][3]. The underlying intuition known as
distributional hypothesis [4] can be summarized as: "a word is characterized by the company
it keeps"[5]. However, there is a lack of theory to justify why it works, even for a simple
task such as word classification or clustering. For example, why do the features from cooccurrences improve the performance of word classification or clustering? What types of
context features or co-occurrence measures yield good performance? Why do the cooccurrences with multiple context terms perform better than a single one? The challenges lie
in: 1) the general theory of feature learning is still an unsolved problem. 2) It is difficult for
a theory to explain many different techniques that convert co-occurrences into prediction
scores.
In this work, we first show that in a word classification task, simple co -occurrence with a
single context feature can achieve perfect performance under some assumptions. Then we
investigate the impact of context features and different co-occurrence measures without the
assumptions. We also explain the reasons why using co-occurrences with multiple context
features, e.g., vector representation can be better than just using a single one.

2

Noti on s

We consider a word classification task that assigns each word in a text corpus one or more
class labels by the co-occurrences of the word with one or more context patterns. By going
through the occurrence of each word in the corpus, we define each example as a tuple
,
,
denoted by , 𝐜, . The component X is a discrete random variable
taking the values from the set { , … , }, where each element refers to a different word. The
component 𝐜 = 𝐶 , … , 𝐶 is a vector of features, where each component 𝐶 (k is from 1 to

n) is a random variable that takes the value 1 or 0, indicating if a word or pattern appears in
the context. Similarly we define =
,…,
as a vector of binary labels such as concepts.
We aim to investigate how well we can assign each example a set of labels y by the cooccurrence information of X and 𝐶 with insufficient or without y at training stage.

Note that for a label variable (t is from 1 to o), there are two cases. It may depend on both
the current word X and the context c or depend on X only. In the tasks such as word sense
disambiguation or part-of-speech tagging, the same word can be assigned to different labels
in different contexts. In some tasks that investigate the meaning of words independently of
contexts, e.g., building WordNet [6] or calculating word-word similarity [3], the same word
X can only be assigned to an unique label 0 or 1 for each . In other words, is function of
X, that is, for each X, 𝑃 | equals 0 or 1. We consider both cases in our analysis, but we
find that we can get much simpler results for the second case. Moreover, the word
classification task for the second case can be generalized to the task of annotating everything
with words. For example, describing an image with several words or a sentence is closely
related to the task of classifying words into the class of description for the given image.
In the following sections, when we aim to investigate the impact of a single context feature
𝐶 or a single task , for simplicity, we use a random variable C or Y to denote 𝐶 or
respectively. So the tuple
, 𝐜,
can be simplified as
, 𝐶,
in the setting of single
context feature and single task. In this work, we only consider the case of discrete random
variables because text data as well as almost all natural data are discrete.

3

Con d i ti on al in d ep en d en ce assu mp ti on

We show that in a special case if a context feature C is conditionally independent with the
word feature X on label Y, the correlation coefficient between P(Y=1|X) and P(C=1|X)
equals 1 or -1.
∈ { ,…,

Theorem 1. Given random variables
1. 𝑃 𝐶 = ,

=

≠𝑃 𝐶=

𝑃

=

}, 𝐶 ∈ { , } and

2. 𝑃 𝐶 = , = | =
=𝑃 𝐶= | = 𝑃 = | =
𝑃 𝐶= | = 𝑃 = | =
for every i from 1 to m

and 𝑃 𝐶 = ,

, we have:

𝐶
Proof. 𝑃 𝐶 = |
=

=

=

=

=

𝑃

𝑃 𝐶= | =

𝑃 𝐶= ,

𝑃

𝑃

=

𝑃 𝐶= ,

𝑃 𝐶= , =
𝑃 =

𝑃 𝐶= ,

𝑃

=

=
𝑃

=

−

𝑃

𝑃

| =

𝑃

𝑃

,

/𝑃

=

𝑃

= |

𝑃 𝐶= , | =

,

=

(𝑃 𝐶 =

−𝑃 𝐶 = 𝑃
= 𝑃 ̅̅̅̅̅̅̅̅
=

Since 𝑃 𝐶 = , =
so we have 𝐶
(𝑃

(𝑃

−𝑃 𝐶 =

= |

=

=

+

𝑃

𝑃

=

=

,𝑃 𝐶 = |

| =

=

) =

+𝑃 𝐶 = , | =

+𝑃 𝐶= | =
+𝑃 𝐶 = ,

(𝑃 𝐶 =

𝑃

=

=

𝑃

| =

𝑃

𝑃

=

=

(using the conditional independence assumption)

−𝑃 𝐶 = ,
𝑃 =
=

,𝑃 𝐶 = |

∈ { , }, if

=

= |

=

,

=

− 𝑃 𝐶 = , = )(𝑃
𝑃 =
)

𝑃

= |

+

+𝑃 𝐶 = | =

≠ ,𝑃 𝐶= |

) =

𝑃

/𝑃

(𝑃 𝐶 =

−𝑃

=
,

=

)

−𝑃 𝐶 = ,
𝑃 =

is a linear function of 𝑃

=

= |

)

,

This theorem indicates that a perfect context feature for a given task is not necessarily the

class label itself. Therefore, even if we have little information about Y, we still have the
chance to know the information of 𝑃 = | by 𝑃 𝐶 = | . Some previous works also
reported the impact of conditional independence assumption in different tasks [7][8][9][10].
However, in practice it is almost impossible to get the case with exact conditional
independence in different tasks. Even if it exists, it is difficult to find it because in order to
calculate the conditional dependence we still need to know the joint probability P(X,C,Y).
Therefore, we need a theory to describe the performance in the situation with certain degree
of conditional dependence. We also need to know the impact of different functions that
convert simple co-occurrence measures e.g., 𝑃 𝐶 = |
to feature values or prediction
scores. In the following sections, we will show the results for the cases.

4

Co-occu r ren c es w i th a si n gl e con text featu re

In this section, we study the case of co-occurrence with a single context feature for a single
task. Although it seems simple, there is still no systematic theoretical framework for it.
Based on the theory about the function of discrete random variables, we analyze the impact
of context features and the functions that convert co -occurrence measures to prediction
scores. Note that the cases discussed in this section are not under the conditional
independence assumption introduced in Section 3.
4.1

F u n c t i o n o f a d i s c re t e r a n d o m v a r i a b l e

Since the co-occurrence based learning actually converts the word variable ∈ { , … , }
to another variable 𝑆 ∈ { , … , } by certain co-occurrence measure such as 𝑃 , 𝐶 =
or
𝑃 𝐶 = | . In other words, there exists a function that maps { , … , } to { , … , }, or
equivalently, we can say S is a function of X. Our goal is to find the function that achieves
the best performance. Therefore, we need to investigate some general principles about the
function of a discrete random variable at first. In this work, we use correlation coefficient of
conditional probabilities as the measure of performance, because it tends to make the
complex analysis simpler.
Lemma 1. Let g be a function that maps
∈ { , … , } to another random variable 𝑆 ∈
{ , … , }, denoted by 𝑆 =
. For any binary random variable ∈ { , } and any function f
that assigns S a real number, we have the following results:
(1) 𝐶 (𝑃 = | , 𝑆 ) = 𝐶 (𝑃 = |𝑆 , 𝑆 )
(2) 𝐶 (𝑃 = | , 𝑃 = |𝑆 ) = 𝑉 (𝑃 = |𝑆 )

(3) 𝐶

(4) 𝐶

(𝑃

= |

(𝑃

Proof.
(1) 𝐸(𝑃

,𝑃

= |

,

= |𝑆 ) =

𝑆 )= 𝐶

)=∑= 𝑃

= |

Similarly, we can prove that 𝐸(𝑃
Therefore, we have 𝐸(𝑃 = |

𝐶

(𝑃

=∑= 𝑃

=∑

=∑

=∑

=

=

=

= |
=

,

𝑆 )=𝐶

𝑃

= | =

( ) − 𝐸( 𝑆 ) ∑𝑔

𝑥 =

( ) − 𝐸( 𝑆 ) 𝑃(𝑆 =

𝑃(𝑆 =

) 𝑃( = |𝑆 =

√𝑉𝑎 (𝑃

(𝑃

𝑃

=
𝑃

√𝑉𝑎 (𝑃

= |𝑆 )

= |

= |𝑆 ,

𝑃

=𝑥 , =

)

𝑆 )𝐶

(𝑃

=∑= 𝑃

=𝑥

= |𝑆 ) = 𝑃 = .
) = 𝐸(𝑃 = |𝑆 )
= |

− 𝐸(𝑃

𝑃

=

, (

= |
𝑃

) 𝑃( = |𝑆 =

) − 𝐸(𝑃

= |

=

,

,𝑃

=

=𝑃

)

ℎ

= | =

− 𝐸(𝑃

) ( (

= |𝑆 )

) − 𝐸(𝑃

)−𝐸

= |

𝑖 𝑖𝑖
(

)

( ) − 𝐸( 𝑆 )

𝐸(𝑃

= |

= |

) = 𝐸(𝑃

= |𝑆 )
=

) )
)

= |𝑆 )

=𝐶

(𝑃

= |𝑆 ,

𝑆 )

(2) Since 𝑃 = |𝑆 is a function of 𝑆, let 𝑆 be 𝑃 = |𝑆 and using (1) we have
𝐶 (𝑃 = | , 𝑃 = |𝑆 ) = 𝐶 (𝑃 = |𝑆 , 𝑃 = |𝑆 ) = 𝑉 (𝑃 = |𝑆 )

(3) Based on (2), we have
𝐶

(𝑃

= |

,𝑃
=

(4)
𝐶
=

(𝑃

√𝑉

=𝐶

= |

𝐶

(𝑃

(𝑃

(𝑃

√𝑉

,

(𝑃

𝑆 )=

= |𝑆 ,

= |𝑆 )√𝑉

= |𝑆 ,

𝐶

= |𝑆 ) =

√𝑉

𝑉

(𝑃

= |

√𝑉

𝐶

(𝑃

𝑆 )√𝑉

𝑆 )𝐶

(𝑃

(𝑃

(𝑃
(𝑃

= |

= |𝑆 )

)√𝑉

( 𝑆 )√𝑉
(𝑃

= |

(𝑃

)√𝑉

= |𝑆 )

= |

,

= |

)√𝑉

(𝑃

= |

= |

,𝑃

= |𝑆 )
,𝑃

= |𝑆 )
(𝑃

=

|𝑆 )

√𝑉

√𝑉

(𝑃

= |𝑆 )

(𝑃

= |

)

𝑆 )

( 𝑆 )
)

= |𝑆 )

3

The first equation plays a fundamental role, which indicates that if S is function of X, to
calculate the covariance between 𝑃 = | and 𝑆 , if S is available, we don’t need the
appearance of X but only S is enough, as if X was forgotten or safely hidden. It also
simplifies the analysis by avoiding explicit analysis of the divergence between 𝑃 = |
and
𝑆 such as |𝑃 = | − 𝑆 |or 𝑃 = |
𝑆 . Therefore, the correlation
coefficient can also be written in a much simpler form. In the word classification task, the
term
𝑆 can be viewed as a classifier based on the new features S. The 𝐶
(𝑃 =
| , 𝑆 ) measures its correlation with the best possible classifier 𝑃 = | based on the
word features X (single view) only. If Y is a function of X (the second case discussed in
Section 2), we have 𝐶
(𝑃 = | , 𝑆 ) = 𝐶
( , 𝑆 ), which measures almost
exactly the (regression) performance of the word classification task. Based on these results,
we are able to analyze the performance of the new features generated by co-occurrences.
4.2

An upper bound of any function
}, 𝐶 ∈ { , } and

Theorem 2. Given random variables ∈ { , … ,
that maps 𝑃 𝐶 = | to a real number, there is
𝐶

𝐶

𝑃

= |

, (𝑃 𝐶 = |

𝑃( = |𝑃 𝐶 = |

)

=

), (𝑃 𝐶 = |

)

𝐶

𝑃

= |

∈ { , }, for any function f
, 𝑃( = |𝑃 𝐶 = |

)

Poof. Since 𝑃 𝐶 = | is a function of X, let S be the random variable that takes values
𝑃 𝐶 = | and based on Lemma 1(4), we have:

𝐶
(𝑃 = | , 𝑆 ) = 𝐶
(𝑃 = |𝑆 ,
Equivalently, we can write the equation as:

𝑆 ) 𝐶

(𝑃

= |

,𝑃

= |𝑆 )

𝐶

𝑃

𝐶

= |

, (𝑃 𝐶 = |

𝑃( = |𝑃 𝐶 = |

)

=

), (𝑃 𝐶 = |

)

𝐶

𝑃

= |

, 𝑃( = |𝑃 𝐶 = |

)

It shows that the performance of the classifier (𝑃 𝐶 = | ) is determined by the product
of two parts. The first part depends on both the context feature C and the function f. The
second part depends on the context feature C only and not relevant to f. Therefore, if we fix
the context feature C to select the function f, using 𝐶

𝑃

= |

, (𝑃 𝐶 = |

)

(correlation with the gold standard) and 𝐶
𝑃( = |𝑃 𝐶 = | ), (𝑃 𝐶 = | ) (the
“silver” standard) produce the same result. Similar to Lemma 1(1), we don’t need the
appearance of 𝑃 = |
for function selection. In addition, since any correlation
coefficient ranges from -1 to 1, we have:

𝐶
𝑃 = | , (𝑃 𝐶 = | )
𝐶
𝑃 = | , 𝑃( = |𝑃 𝐶 = | )
It indicates that if we want to improve the performance by fixing a certain context feature C
and changing different functions f, e.g., from 𝑃 𝐶 = |

to log

𝑃 𝐶= ,

𝑃 𝐶=

𝑃

(the point wise

mutual information), the performance cannot be arbitrarily high but upper bounded by the
squared correlation coefficient between 𝑃 = | and 𝑃( = |𝑃 𝐶 = | ). Given fixed
word feature set X and label Y, the upper bound is determined only by the context feature C
but not relevant to the function f. Actually, the upper bound is a special case of maximum
correlation coefficient [11][12]. If we want to build a high performing classifier (𝑃 𝐶 =
| ), we must find a way to improve the upper bound, that is, to select a good context
feature C. However, it is not easy to do it from this formula directly. Therefore, we relate the
upper bound to the special case introduced in Section 3. Since 𝑃 𝐶 = | is a function of
itself, based on Theorem 2, we have:
𝐶

(𝑃

= |

,𝑃 𝐶 = |

)

𝐶

𝑃

= |

, 𝑃( = |𝑃 𝐶 = |

)

The correlation coefficient 𝐶
(𝑃 = | , 𝑃 𝐶 = | )
not only reflects the
performance of a special co-occurrence measure 𝑃 𝐶 = | , but also provides a way to
improve the upper bound (for other co-occurrence measures). Based on Theorem 1, under
the conditional independence assumption, 𝐶
(𝑃 = | , 𝑃 𝐶 = | ) equals 1, so that

𝐶
𝑃 = | , 𝑃( = |𝑃 𝐶 = | )
equals 1 as well. As what we discussed in
Section 3, it is important to know more about the cases out of the conditional independence
assumption, but it is difficult to analyze 𝐶
(𝑃 = | , 𝑃 𝐶 = | ) based on the
conditional dependence directly without any other assumption. In the following section, we
show that we are able to obtain much simpler results by assuming that the label variable
∈ { , } is a function of X.
4.3

Assu ming that Y is a function of X

In Section 2, we have pointed out that there are many cases in practice where Y is a function
of X. We believe that if we can explain everything about this task, we may naturally find the
cues to move to more advanced tasks such as sentence classification and question answering.
Theorem 3. If
have:
𝐶

(𝑃

∈ { , } is a function of

= |

,𝑃 𝐶 = |

=𝐶

(𝑃

)
= |

∈ { ,…,
,𝑃 𝐶 = |

}, for a context feature 𝐶 ∈ { , }, we
) 𝐶

(𝑃 𝐶 = |

,𝑃 𝐶 = |

)

Proof. Since Y and C are both binary random variables, using Lemma 1 (4) replace Y by C
without loss of generality, and we have:

𝐶

(𝑃 𝐶 = |

,

𝑆 ) = 𝐶

(𝑃 𝐶 = |𝑆 ,

𝑆 ) 𝐶

(𝑃 𝐶 = |

, 𝑃 𝐶 = |𝑆 )

Since Y is a function of X and 𝑃 = | is a function of Y, in the above equation replace S
by Y and 𝑆 by 𝑃 = | respectively, and we have:
𝐶
(𝑃 𝐶 = | , 𝑃 = | )
=𝐶

(𝑃 𝐶 = |

= |

,𝑃

(𝑃 𝐶 = | , 𝑃 𝐶 = |

)

= | ,𝑃 𝐶 = | )
= | ) 𝐶
(𝑃 𝐶 = | , 𝑃 𝐶 = |

)

∈ { , } is a function of X, there is 𝑃

Since
have
𝐶

= |

(𝑃

,𝑃 𝐶 = |

=𝐶

) =𝐶

(𝑃 𝐶 = |

,𝑃

) 𝐶

= |

(𝑃

=𝑃

= |

= . Therefore, we

This result describes what a good context feature is and can give guidance for selection of
context features for a particular task. It is much simpler and more practical than what we can
get by the direct analysis of the conditional dependence, since we just need to know the joint
distribution of
, 𝐶 and 𝐶,
rather than
, 𝐶, . This result can also be written as a
simpler form as follows.
∈ { , } is a function of

Corollary 1. If
have:

Proof.
𝐶
(𝑃

=𝐶

=

=
=

𝑉

𝐶

𝐶

(𝑃

= |

(𝑃

(𝑃

= |

= |

(𝑃 𝐶 = ,
𝑃

=

,𝑃 𝐶 = |

)

,𝑃 𝐶 = |

−𝑃 𝐶 =

=

)

𝑉

)𝑉

𝑃

(𝑃 𝐶 = ,
𝑃

=

(𝑃 𝐶 = |

)𝑉

,𝑃 𝐶 = |

)

)

)

)

)𝐸(𝑃 𝐶 = |

(𝑃 𝐶 = |

=

(𝑃 𝐶 = |

𝑃

)

)

−𝑃 𝐶 =
𝑉

(𝑃 𝐶 = |

= |

=

=

(𝑃 𝐶 = |

𝑉

) − 𝐸(𝑃

= |

(𝑃

) =

) 𝐶

(𝑃 𝐶 = |

)𝑉

𝑃 𝐶= |

𝑉

=

𝑃

,𝑃 𝐶 = |

,𝑃 𝐶 = |

= |

= |

𝐸(𝑃

= |

(𝑃

}, for a context feature 𝐶 ∈ { , }, we

∈ { ,…,

)

𝑃

=

(𝑃 𝐶 = |

)

)

(Theorem 3)
(𝐿

3 )

In the equation, the term (𝑃 𝐶 = , = − 𝑃 𝐶 = 𝑃 = ) describes the dependency
between context feature C and the class label Y. The term 𝑉 (𝑃 𝐶 = | ) addresses the
dependency between context feature C and each word feature X. We can see more clearly in
another equivalent form:

𝐶

(𝑃

= |

,𝑃 𝐶 = |

𝑃
) =
𝑃

=
=

𝑃 𝐶= , =
(
− )
𝑃 𝐶= 𝑃 =
𝑃 𝐶= , =
∑= (
− )
𝑃 𝐶= 𝑃 =

From this mutual information style form, we can conclude that a good c ontext feature for a
particular task should be a trade-off between high dependence with label Y and low
dependence with every word feature X. The finding is similar to our previous work [10], but
the result here is more general, under weaker assumptions, and more accurate (equation
rather than inequality).

5

Co-occu r ren c es w i th mu l ti p l e con text f eatu res

So far, we have investigated the co-occurrences with a single context feature only. In
practice, we usually find representing a word by a vector of co-occurrences with multiple
context features tends to perform better. In the following theorem, we show part of the
reason by analyzing the performance of the vector from multiple context features.
Theorem 4. If we represent each X by a r-dimensional vector = (𝑃 𝐶 = | , … , 𝑃 𝐶 =
| ), where 𝐶 , … , 𝐶 are r context features (
), for the label ∈ { , } and any function
f that maps each vector x to a real number, we have
(1) 𝐶

(2) 𝐶

(𝑃

= |

,

(𝑃

= |

,𝑃

𝐶

(𝑃

= |

) = 𝐶

(𝑃

= | ) =

= | ,

𝐶

𝑃

𝐶

𝑃

= |

) 𝐶

(𝑃

,𝑃( = |𝑃 𝐶 = |

= | ,𝑃( = |𝑃 𝐶 = |

)

)

2

2

= |

,𝑃

= | )

for every k from 1 to r

Proof. (1) Since X is a discrete random variable and = (𝑃 𝐶 = | , … , 𝑃 𝐶 = | ), we
know that x is a discrete random variable and a function of X as well. Therefore, based on
Lemma 1 (4), we come to the conclusion directly:
,

= 𝐶

)

(𝑃

= | ,

) 𝐶

(𝑃

= |

,𝑃

= | )

(2) Since for each = (𝑃 𝐶 = | , … , 𝑃 𝐶 = | ) with a different value, the value of
𝑃 𝐶 = |
is unique, the discrete random variable 𝑃 𝐶 = |
is a function of x. In
Lemma 1 (3), replace X by x, and S by 𝑃 𝐶 = | without loss of generality, and we have
𝑉
𝑃( = |𝑃 𝐶 = | )
𝐶
𝑃 = | , 𝑃( = |𝑃 𝐶 = | ) =
𝑉 (𝑃 = | )
Since x is a function of X, and 𝑃 𝐶 = | is a function of X, according to Lemma 1 (3), we
have
𝑉 (𝑃 = | )
𝐶
(𝑃 = | , 𝑃 = | ) =
𝑉 (𝑃 = | )
𝐶

𝑃

= |

, 𝑃( = |𝑃

|

)

By combining the above three equations, we have:
𝐶

(𝑃

= |

,𝑃

= | ) =

𝐶

𝐶

𝑃

=

𝑃

𝑉

𝑃( = |𝑃 𝐶 = |
𝑉

= |

(𝑃

= |

)

, 𝑃( = |𝑃 𝐶 = |

= | , 𝑃( = |𝑃 𝐶 |

)

)

)

Similar to Theorem 2, the performance of the vector-style representation can also be written
as the products of two parts. The second part 𝐶
(𝑃 = | , 𝑃 = | ) is an upper
bound of the performance of any function, which is determined only by the set of context
features 𝐶 , … , 𝐶 . The result (2) shows that the upper bound from the vector is always better
than the performance from any individual context feature, since we can prove that
𝐶

(𝑃

= |

,𝑃

= | )

𝐶

the proof of (2), it can be proved that 𝐶

𝑃

(𝑃

= |

= |

more context features are introduced. However, 𝐶

, 𝑃( = |𝑃 𝐶 = |
,𝑃

(𝑃

) . Similar to

= | ) always increases when

= | ,

) may decrease when

the vector become longer, so its final performance 𝐶
(𝑃 = | ,
) may increase or
decrease. Therefore, we need to find the factors that determine the combined performance of
the two terms in the future.

6

Con cl u si on s and f utu re w ork s

In this paper, we give a theoretical study of the approaches that learn the representation of
words by the approaches like (𝑃 𝐶 = | ) or (𝑃 𝐶 = | , … , 𝑃 𝐶 = | ). The
theoretical framework is general enough to explain a set of approaches based on
distributional semantics and is able to give guidance for algorithm design such as the
selection of context feature and the co-occurrence metrics. In the next steps, we are going to
give a deeper analysis of each formula that determines the performance, such as the diversity
between multiple context features, find more principles that are constructive to algorithm
design in practice and extend the theory to analyze other tasks in NLP and machine learning.
Moreover, there are many important fundamental questions, e.g., what is the nature of cooccurrence? Is there some more fundamental reason for learning from co -occurrences? Is the
creation of vector x related to the creation of the vector based instance representation in
machine learning in general?

Ref eren c es
[1] Deerwester, S., Dumais, S. T., Furnas, G. W., Landauer, T. K., & Harshman, R. (1990). Indexing
by latent semantic analysis. Journal of the American society for information science, 41(6), 391.
[2] Brown, P. F., Desouza, P. V., Mercer, R. L., Pietra, V. J. D., & Lai, J. C. (1992). Class -based ngram models of natural language. Computational linguistics, 18(4), 467-479.
[3] Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word
representations in vector space. arXiv preprint arXiv:1301.3781.
[4] Harris, Z. S. (1954). Distributional structure. Word, 10(2-3), 146-162.
[5] Firth, J. R. (1957). A synopsis of linguistic theory, 1930-1955. Studies in linguistic analysis.
[6] Miller, G. A. (1995). WordNet: a lexical database for English. Communications of the ACM,
38(11), 39-41.
[7] Blum, Avrim, and Tom Mitchell. "Combining labeled and unlabeled data with co-training."
Proceedings of the eleventh annual conference on Computational learning theory . ACM, 1998.
[8] S. Abney. 2002. Bootstrapping. In In Proceedings of the 40th Annual Meeting of the Association
for Computa- tional Linguistics, pages 360–367.
[9] Veeramachaneni, Sriharsha, and Ravi Kumar Kondadadi. "Surrogate learning: from feature
independence to semi-supervised classification." Proceedings of the NAACL HLT 2009 Workshop on
Semi-Supervised Learning for Natural Language Processing. Association for Computational
Linguistics, 2009.
[10] Li, Yanpeng. "Reference Distance Estimator." arXiv preprint arXiv:1308.3818 (2013).
[11] Rényi, Alfréd. "On measures of dependence." Acta mathematica hungarica 10.3-4 (1959): 441451.
[12] Hall, W. J. On Characterizing Dependence in Joint Distributions. University of North Carolina,
Department of Statistics, 1967. Pp11-12

