gracaninja/lxmls-toolkit — pull request #18 by ramon-astudillo (2013-07-17 11:10:54+00:00)
Title: Log-domain bug solved in Day2, Sec 2.6. Some recently introduced bugs remain.

The use of log-domain representation leads to infinites being used in computations and causes to not-a-numbers in posterior decoding (and lots of Warnings). The results in Exercise 2.9 for posterior decoding are therefore wrong. This update solves this by imposing a finite, but very low, logzero() equal to -1e20 and using a new version of safe_log for ndarrays.

This is close to using a very very small smoothing constant. In principle it should not imply a difference in the exercises but the results are different from the originals, please have a look.

To reproduce the bug, use this code on the old version of the code (without this update)

# Preparation
corpus = pcc.PostagCorpus()
train_seq = corpus.read_sequence_list_conll("../data/train-02-21.conll",max_sent_len=15,max_nr_sent=1000)
test_seq = corpus.read_sequence_list_conll("../data/test-23.conll",max_sent_len=15, max_nr_sent=200)
dev_seq = corpus.read_sequence_list_conll("../data/dev-22.conll",max_sent_len=15, max_nr_sent=200)
hmm = hmmc.HMM(corpus.word_dict, corpus.tag_dict)
hmm.train_supervised(train_seq)
# Ex 2.9
viterbi_pred_train = hmm.viterbi_decode_corpus(train_seq)
pdb.set_trace()
posterior_pred_train = hmm.posterior_decode_corpus(train_seq)
eval_viterbi_train = hmm.evaluate_corpus(train_seq, viterbi_pred_train)
eval_posterior_train = hmm.evaluate_corpus(train_seq, posterior_pred_train)
print "Train Set Accuracy: Posterior Decode %.3f, Viterbi Decode: %.3f"%(eval_posterior_train,eval_viterbi_train)

Once in debug, break at line 193 of sequences/sequence_classifier.py

(Pdb) b sequences/sequence_classifier.py:193
(Pdb) c
(Pdb) state_posteriors

array([[ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan],
[ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan],
[ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan],
[ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan],
[ nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan]])

(Pdb) cl
(Pdb) c

This leads to the bogus result

Train Set Accuracy: Posterior Decode 0.322, Viterbi Decode: 0.981

Note that the Viterbi results seems to work, as it can deal with Infs. After this fix, the results should look like this

unknown tag po
unknown tag pr
unknown tag wd
unknown tag pd
unknown tag wr
unknown tag sy
unknown tag pr
unknown tag po
unknown tag wr
unknown tag wd
unknown tag pd
unknown tag wd
unknown tag wr
unknown tag po
unknown tag pr
unknown tag pd
Train Set Accuracy: Posterior Decode 0.981, Viterbi Decode: 0.981
Test Set Accuracy: Posterior Decode 0.862, Viterbi Decode: 0.867
Smoothing 10.000000 -- Train Set Accuracy: Posterior Decode 0.731, Viterbi Decode: 0.709
Smoothing 10.000000 -- Test Set Accuracy: Posterior Decode 0.710, Viterbi Decode: 0.687
Smoothing 1.000000 -- Train Set Accuracy: Posterior Decode 0.884, Viterbi Decode: 0.878
Smoothing 1.000000 -- Test Set Accuracy: Posterior Decode 0.820, Viterbi Decode: 0.810
Smoothing 0.100000 -- Train Set Accuracy: Posterior Decode 0.969, Viterbi Decode: 0.965
Smoothing 0.100000 -- Test Set Accuracy: Posterior Decode 0.860, Viterbi Decode: 0.848
Smoothing 0.000000 -- Train Set Accuracy: Posterior Decode 0.981, Viterbi Decode: 0.981
Smoothing 0.000000 -- Test Set Accuracy: Posterior Decode 0.852, Viterbi Decode: 0.857
Best Smoothing 0.100000 -- Test Set Accuracy: Posterior Decode 0.865, Viterbi Decode: 0.848

Note the small improvement attained by playing around with smoothing.