Repository: nltk/nltk

Issue:
IBM Models mix up target and source
The documentation IBM models (at least `IBMModel1`) mixes up the source and target languages:
1. The constructor accepts a list of `AlignedSent` objects, which state that `words: source language words ... mots: target language words`.
1. `IBMModel1` just eats it and generates the translation table, as well as generates `Alignment` objects for the input corpus
    - the docstring for `__init__` states that `Translation direction is from "AlignedSent.mots" to
        "AlignedSent.words"`
    - however, `translation_table` is actually a double `defaultdict {words: {mots: probability}}`, so it is definitely `words -> mots`
1. Then comes `IBMModel1.prob_t_a_given_s()`, which actually IS from target to source, and tries to get the probabilities `self.translation_table[trg_word][src_word]`. Which, of course, fails completely because that is not how the table was built.


Discussion:
Also, `defaultdict` should probably be replaced with `dict.get()` in `prob_t_a_given_s()`.
@DavidNemeskey Thanks for raising the issue. 

Could you further explain what you mean by:

> The documentation IBM models (at least IBMModel1) mixes up the source and target languages:
> 
> - The constructor accepts a list of AlignedSent objects, which state that words: source language words ... mots: target language words.
> - IBMModel1 just eats it and generates the translation table, as well as generates Alignment objects for the input corpus

Is `prob_t_a_given_s` used in the `IBMModel1` object? I haven't looked at the code closely but I can't find where it's been used. 

@hoontw could you help to look into this?
@alvations The formatting was messed up. I fixed it, and I hope it is clearer now. If not, just let me know what I should explain more.

`prob_t_a_given_s` is not used **in** `IBMModel1` (or maybe it is, I don't know), but it is a public method, so it can be called by client code. (And how else would I compute _prob_t_given_s_?)
It's been a while since I looked at the code, so I'll write a more detailed response later.

One easy thing I can clear up is that the translation table represents P(word|mot). For example, `translation_table["house"]["haus"]` is the probability that given the word "haus", "house" is the right translation. Therefore the translation direction is from `mot -> word`.
Any specific reason why you translate from _target_ to _source_? I mean, the semantics of those two words in English are very clear. What is the benefit of inverting them?

Also, the function is called _P(T,A|S)_ -- again indicating that the translation should be _S -> T_.

Finally, if you use the translation table as you say in the comment above, that also goes against any intuition one might have about a dict-of-dicts data structure. Take `numpy`, for example: `M[i, j]` is equivalent to `M[i][j]`, not `M[j][i]`. What if I am interested in all the possible translations of "haus"? Should I enumerate all entries in the table just to be able to get them (i.e. `translation_table[*]["haus"])? That's horribly ineffective.
tl;dr: documentation for `AlignedSent` is wrong. `words` is not target language, `mots` is not source language. We should swap them or use neutral terms like "first of a language pair" and "second of a language pair".

@DavidNemeskey Good catch! The documentation for `AlignedSent` says that `words` is source and `mots` is target, but the IBM models swap them. Either the documentation or the IBM model usage is wrong. To find out which one is correct, I looked at the history of these files. Unfortunately, this discrepancy has been there since the beginning in 2010: https://github.com/nltk/nltk/commit/699822356170667e8507d4efe344724ef0011b54 for `AlignedSent` and https://github.com/nltk/nltk/commit/2379d27da8493eb27f0f995695db73182965e5a7 for IBM Model.

So, I took another approach. I looked at comtrans, the bilingual corpus that the work of AlignedSent and IBMModel was based on.
```
>>> from nltk.corpus import comtrans
>>> comtrans.aligned_sents()[0].words
['Wiederaufnahme', 'der', 'Sitzungsperiode']
>>> comtrans.aligned_sents()[0].mots
['Resumption', 'of', 'the', 'session']
>>> comtrans.aligned_sents()[0].alignment
Alignment([(0, 0), (1, 1), (1, 2), (2, 3)])
```
The alignment pairs are in the format `(word_index, mot_index)`. This is easily seen because the German sentence has 3 words and thus a word index of 3 is not possible. Side note: the documentation example to initialize `AlignedSent` is wrong because it uses `(mot_index, word_index)`.

This still doesn't answer the question about whether `AlignedSent.words` should be source or target language. I see two solutions:
1.  `AlignedSent.words` as target language. Because this is how the IBM models treat it since the beginning, and changing how the IBM models process input will be a breaking functional change. On the other hand, changing the documentation on `AlignedSent` shouldn't be a big problem. The concept of source and target language is meaningless in `AlignedSent`. The alignments are bi-directional, and it doesn't matter which one is source or target. What matters is the format of alignment pairs: `(word_index, mot_index)` or `(mot_index, word_index)`. Which brings me to the second solution...

2. Don't mention source and target language at all in `AlignedSent`. Use neutral terms like language1 or language2. `AlignedSent` is just a data structure to store a sentence each from two languages and the mapping between words. It is up to the consumers of `AlignedSent` to decide which one should be source or target.

I am slightly inclined to the second solution, because it keeps `AlignedSent` agnostic of how it is used by higher applications.
Regarding the order of indices in `translation_table`, I agree that it is counter-intuitive and should be swapped. There isn't a good reason why P(a|b) is represented as p[a][b], other than historical reasons and was meant to be used internally by the IBM models, not for humans to examine.

I agree that it will be useful to examine the likely translations of a source word. Perhaps we could create a utility function to convert `translation_table[target_word][source_word]` to `translation_table_invert_access[source_word][target_word]`.

Also, @DavidNemeskey you mentioned that `defaultdict` should probably be replaced with `dict.get()` in `prob_t_a_given_s()`. Can you clarify what you mean because I do not see`defaultdict` being created in that function.
@alvations another option would be to add a parameter to the `IBMModel` constructor that specifies the direction (and provide a default value to maintain backward compatibility).

BTW _mots_ is not a very sensible name, but I guess it is too late to change that now...

@hoontw no, the whole `translation_table` is a `defaultdict` of `defaultdict`s. That's a problem because the user can call `prob_t_a_given_s` even with words not in the table. If that happens, the words are added to the table, increasing the model size with redundant (`EPSILON`) values. `dict.get` doesn't suffer from this problem. (`defaultdict.get` doesn't either, but it's better to be on the safe side, because user code could still call `[]` instead of `get`, even if you replace all occurrence of the former with the latter in the `ibm*.py` files).
Yet another way to switch the direction of translation is to use `AlignedSent.invert()` for the data. This is probably what the IBM models would do internally anyway if it supported switching.

Yes, _mots_ isn't a great name. It was originally _s1_ and _s2_, which wasn't great either because it is too cryptic. _target_sentence_ and _source_sentence_ can be confusing too because the relationship between two sentences in  `AlignedSent` is bi-directional. Moreover, in the noisy channel model of machine translation, _P(target|source) = (P(source|target) * P(target)) / P(source)_. To compute a translation from source to target, we actually need a translation table in the opposite direction from target to source. This is super confusing if the context of _target_ and _source_ is not made clear. Personally, I would just name it `sentence1` and `sentence2`, but like you said, it's too late for that.

Regarding the use of `defaultdict` vs `dict`, if I understand your concern correctly, it is the unnecessary increase in size of the table when unseen data are encountered. That is a valid concern and could be handled with a better data structure. The translation table has a practical problem of how to handle unseen data when queried. Basically, there are 4 options:
1. Throw an exception
2. Return 0
3. Return a "don't know" value like `None`
4. Return a smoothed value, usually very small

The first is not a practical option since it is common to encounter words that are not in the training data. That is why a plain `dict` is not sufficient; it should not bomb when an unknown key is referenced. For a 2-dimensional structure, `dict.get(key, dict()).get(key, default_val)` avoids `KeyError` but gets clunky to use all over the place.
 
The second and third options may be more reasonable, but it complicates the computation and leads to discontinuities (division by zero).

What we really need is a data structure like `defaultdict` but doesn't create new entries when reading unseen keys, and still returns a default value. Should be straightforward to do if someone has the time.

On the subject of optimization, a 2-dimensional `dict` or its variants isn't sufficient for real-world problems. The translation table can easily exceed memory capacity. We need a way to store the table to disk and efficiently read/write to it. Again, a task for someone who has the time and interest :smiley:

Resolving patch:
diff --git a/nltk/translate/api.py b/nltk/translate/api.py
--- a/nltk/translate/api.py
+++ b/nltk/translate/api.py
@@ -21,27 +21,30 @@ class AlignedSent(object):
     Return an aligned sentence object, which encapsulates two sentences
     along with an ``Alignment`` between them.
 
+    Typically used in machine translation to represent a sentence and
+    its translation.
+
         >>> from nltk.translate import AlignedSent, Alignment
         >>> algnsent = AlignedSent(['klein', 'ist', 'das', 'Haus'],
-        ...     ['the', 'house', 'is', 'small'], Alignment.fromstring('0-2 1-3 2-1 3-0'))
+        ...     ['the', 'house', 'is', 'small'], Alignment.fromstring('0-3 1-2 2-0 3-1'))
         >>> algnsent.words
         ['klein', 'ist', 'das', 'Haus']
         >>> algnsent.mots
         ['the', 'house', 'is', 'small']
         >>> algnsent.alignment
-        Alignment([(0, 2), (1, 3), (2, 1), (3, 0)])
+        Alignment([(0, 3), (1, 2), (2, 0), (3, 1)])
         >>> from nltk.corpus import comtrans
         >>> print(comtrans.aligned_sents()[54])
         <AlignedSent: 'Weshalb also sollten...' -> 'So why should EU arm...'>
         >>> print(comtrans.aligned_sents()[54].alignment)
         0-0 0-1 1-0 2-2 3-4 3-5 4-7 5-8 6-3 7-9 8-9 9-10 9-11 10-12 11-6 12-6 13-13
 
-    :param words: source language words
+    :param words: Words in the target language sentence
     :type words: list(str)
-    :param mots: target language words
+    :param mots: Words in the source language sentence
     :type mots: list(str)
-    :param alignment: the word-level alignments between the source
-        and target language
+    :param alignment: Word-level alignments between ``words`` and ``mots``.
+        Each alignment is represented as a 2-tuple (words_index, mots_index).
     :type alignment: Alignment
     """
 
diff --git a/nltk/translate/ibm1.py b/nltk/translate/ibm1.py
--- a/nltk/translate/ibm1.py
+++ b/nltk/translate/ibm1.py
@@ -15,22 +15,25 @@
 """
 Lexical translation model that ignores word order.
 
-In IBM Model 1, word order is ignored for simplicity. Thus, the
-following three alignments are equally likely. As long as the word
-alignments are equivalent, it doesn't matter where the word
-occurs in the source or target sentence.
+In IBM Model 1, word order is ignored for simplicity. As long as the
+word alignments are equivalent, it doesn't matter where the word occurs
+in the source or target sentence. Thus, the following three alignments
+are equally likely.
 
 Source: je mange du jambon
 Target: i eat some ham
-Alignment: (1,1) (2,2) (3,3) (4,4)
+Alignment: (0,0) (1,1) (2,2) (3,3)
 
 Source: je mange du jambon
 Target: some ham eat i
-Alignment: (1,4) (2,3) (3,1) (4,2)
+Alignment: (0,2) (1,3) (2,1) (3,1)
 
 Source: du jambon je mange
 Target: eat i some ham
-Alignment: (1,3) (2,4) (3,2) (4,1)
+Alignment: (0,3) (1,2) (2,0) (3,1)
+
+Note that an alignment is represented here as
+(word_index_in_target, word_index_in_source).
 
 The EM algorithm used in Model 1 is:
 E step - In the training data, count how many times a source language


Test patch: