Repository: nltk/nltk

Issue:
WordNetCorpusReader.ic() does not add smoothing to N
The `WordNetCorpusReader.ic()` function, which computes the concept-frequency mapping used in information content calculations, does not add the smoothing value to what Resnik 1995 ([arXiv link](https://arxiv.org/pdf/cmp-lg/9511007.pdf)) calls *N*, i.e., the total number of occurrences of words in a category, which is used in the denominator of the concept probability. For example, here I build the concept-frequency mapping (*freq(c)* in Resnik 1995) with a corpus of 1 word, *word*:

```python
>>> from nltk.corpus import wordnet as wn
>>> class FakeCorpus:
...   def words(self): return ['word']
... 
>>> freq = wn.ic(FakeCorpus(), weight_senses_equally=False, smoothing=1.0)
```

This dictionary is structured `freq[pos][synset_offset] = count` (where `count` is not strictly a count if `weight_senses_equally` is False). The special offset `0` is used for counting the total, *N*. Since *word* occurs once and it has 11 senses (1 verb, 10 nouns), each synset will accumulate 1/11 == 0.0909... for that one occurrence, in addition to the 1.0 for smoothing:

```python
>>> word = wn.synset('word.n.01')  # one of the 11 synsets
>>> freq['n'][word._offset]
1.0909090909090908
```

But the total for nouns and verbs is less than that, which means the probability of a synset is > 1.0 and the information content is less than 0:

```python
>>> freq['v'][0]  # 0.0909... * 1
0.09090909090909091
>>> freq['n'][0]  # 0.0909... * 10
0.9090909090909093
>>> freq['n'][word._offset] / freq['n'][0]  # p(c) = freq(c) / N
1.1999999999999997
>>> from nltk.corpus.reader.wordnet import information_content
>>> information_content(wn.synset('word.n.01'), freq)
-0.1823215567939544
```

When `weight_senses_equally` is True, each synset gets the full counts and not the counts divided by the number of synsets, so this situation would only come up if the smoothing value were greater than the number of counts:

```python
>>> freq = wn.ic(FakeCorpus(), weight_senses_equally=True, smoothing=1.0)
>>> freq['n'][0]
10.0
>>> freq['n'][word._offset]
2.0
>>> freq = wn.ic(FakeCorpus(), weight_senses_equally=True, smoothing=10.0)
>>> freq['n'][0]
10.0
>>> freq['n'][word._offset]
11.0
```

In conclusion, I think `WordNetCorpusReader.ic()` should initialize the *N* values with the smoothing argument. This will change the values of the similarity functions for generated IC frequency mappings, but the `wordnet.doctest` tests still pass because the `smoothing` value is set to 0.0, so we might need a new test.

WordNetCorpusReader.ic() does not add smoothing to N
The `WordNetCorpusReader.ic()` function, which computes the concept-frequency mapping used in information content calculations, does not add the smoothing value to what Resnik 1995 ([arXiv link](https://arxiv.org/pdf/cmp-lg/9511007.pdf)) calls *N*, i.e., the total number of occurrences of words in a category, which is used in the denominator of the concept probability. For example, here I build the concept-frequency mapping (*freq(c)* in Resnik 1995) with a corpus of 1 word, *word*:

```python
>>> from nltk.corpus import wordnet as wn
>>> class FakeCorpus:
...   def words(self): return ['word']
... 
>>> freq = wn.ic(FakeCorpus(), weight_senses_equally=False, smoothing=1.0)
```

This dictionary is structured `freq[pos][synset_offset] = count` (where `count` is not strictly a count if `weight_senses_equally` is False). The special offset `0` is used for counting the total, *N*. Since *word* occurs once and it has 11 senses (1 verb, 10 nouns), each synset will accumulate 1/11 == 0.0909... for that one occurrence, in addition to the 1.0 for smoothing:

```python
>>> word = wn.synset('word.n.01')  # one of the 11 synsets
>>> freq['n'][word._offset]
1.0909090909090908
```

But the total for nouns and verbs is less than that, which means the probability of a synset is > 1.0 and the information content is less than 0:

```python
>>> freq['v'][0]  # 0.0909... * 1
0.09090909090909091
>>> freq['n'][0]  # 0.0909... * 10
0.9090909090909093
>>> freq['n'][word._offset] / freq['n'][0]  # p(c) = freq(c) / N
1.1999999999999997
>>> from nltk.corpus.reader.wordnet import information_content
>>> information_content(wn.synset('word.n.01'), freq)
-0.1823215567939544
```

When `weight_senses_equally` is True, each synset gets the full counts and not the counts divided by the number of synsets, so this situation would only come up if the smoothing value were greater than the number of counts:

```python
>>> freq = wn.ic(FakeCorpus(), weight_senses_equally=True, smoothing=1.0)
>>> freq['n'][0]
10.0
>>> freq['n'][word._offset]
2.0
>>> freq = wn.ic(FakeCorpus(), weight_senses_equally=True, smoothing=10.0)
>>> freq['n'][0]
10.0
>>> freq['n'][word._offset]
11.0
```

In conclusion, I think `WordNetCorpusReader.ic()` should initialize the *N* values with the smoothing argument. This will change the values of the similarity functions for generated IC frequency mappings, but the `wordnet.doctest` tests still pass because the `smoothing` value is set to 0.0, so we might need a new test.



Discussion:

Resolving patch:
diff --git a/nltk/corpus/reader/wordnet.py b/nltk/corpus/reader/wordnet.py
--- a/nltk/corpus/reader/wordnet.py
+++ b/nltk/corpus/reader/wordnet.py
@@ -1988,6 +1988,8 @@ def ic(self, corpus, weight_senses_equally=False, smoothing=1.0):
 
         # Initialize the counts with the smoothing value
         if smoothing > 0.0:
+            for pp in POS_LIST:
+                ic[pp][0] = smoothing
             for ss in self.all_synsets():
                 pos = ss._pos
                 if pos == ADJ_SAT:


Test patch:
diff --git a/nltk/test/wordnet.doctest b/nltk/test/wordnet.doctest
--- a/nltk/test/wordnet.doctest
+++ b/nltk/test/wordnet.doctest
@@ -605,6 +605,16 @@ Patch-1 https://github.com/nltk/nltk/pull/2065  Adding 3 functions (relations) t
     >>> wn.synsets("slang")[1].in_usage_domains()[18]
     Synset('can-do.s.01')
 
+Issue 2721: WordNetCorpusReader.ic() does not add smoothing to N
+
+    >>> class FakeCorpus:
+    ...     def words(self): return ['word']
+    ...
+    >>> fake_ic = wn.ic(FakeCorpus(), False, 1.0)
+    >>> word = wn.synset('word.n.01')
+    >>> information_content(word, fake_ic) > 0
+    True
+
 
 ------------------------------------------------
 Endlessness vs. intractability in relation trees