Repository: nltk/nltk

Issue:
Maybe use non-uniform distribution for Baum-Welch by default?
See https://github.com/nltk/nltk/issues/100. 

I encountered a suggestion for slightly non-uniform initial probability distributions (uniform + a bit of noize) in some paper (maybe about unsupervised POS tagging), but can't find the reference now. Most papers that employ Baum-Welch use either uniform distributions or distributions learned from  corpora using carefully crafted heuristics.



Discussion:
So as I understand it, and this is based on my advisor's [thesis](http://brl.ee.washington.edu/BRL_Pubs/Pdfs/Th037.pdf) (p.99 especially), a totally uniform distribution where each probable transition begins equivalent to every other results in a local minimum from which the BW algo will not escape. There is no probability gradient on which to update the matrices. Thus, some random jitter in the initialization is necessary. 

Ideally, yes - you would initialize the transition matrix with initial guesses based on learned distributions. However, most algorithms that require unlabeled training algorithms like BW do not have such heuristics in mind, but instead are willing to train a corpus of HMMs using randomly initiated probabilities and then select the HMM that shows best discrimination against some known dataset (this is how I use and evaluate HMMs). 

The most general solution to this problem is to assign the A and B (transition and output) matrices with a random set of probabilities where any value between 0 and 1 is equally probable (a uniform random distribution). Of course you could select a gaussian or a weighted or band-limited, but uniform is the most general. 

Seeing as I can't find much literature discussing the pros/cons of different types of random distributions, I'm happy to submit a pull request with a uniform random distribution function initializing the `HiddenMarkovModelTagger`'s `_transitions` and `_outputs`. 


Resolving patch:
diff --git a/nltk/probability.py b/nltk/probability.py
--- a/nltk/probability.py
+++ b/nltk/probability.py
@@ -636,6 +636,50 @@ def samples(self):
     def __repr__(self):
         return '<UniformProbDist with %d samples>' % len(self._sampleset)
 
+@compat.python_2_unicode_compatible
+class RandomProbDist(ProbDistI):
+    """
+    Generates a random probability distribution whereby each sample
+    will be between 0 and 1 with equal probability (uniform random distribution.
+    Also called a continuous uniform distribution).
+    """
+    def __init__(self, samples):
+        if len(samples) == 0: 
+            raise ValueError('A probability distribution must '+
+                             'have at least one sample.')
+        self._probs = self.unirand(samples)
+        self._samples = list(self._probs.keys())
+
+    @classmethod
+    def unirand(cls, samples):
+        """
+        The key function that creates a randomized initial distribution 
+        that still sums to 1. Set as a dictionary of prob values so that 
+        it can still be passed to MutableProbDist and called with identical
+        syntax to UniformProbDist
+        """
+        randrow = [random.random() for i in range(len(samples))]
+        total = sum(randrow)
+        for i, x in enumerate(randrow):
+            randrow[i] = x/total
+
+        total = sum(randrow)
+        if total != 1:
+            #this difference, if present, is so small (near NINF) that it 
+            #can be subtracted from any element without risking probs not (0 1)
+            randrow[-1] -= total - 1
+
+        return dict((s, randrow[i]) for i, s in enumerate(samples))
+
+    def prob(self, sample):
+        return self._probs.get(sample, 0)
+
+    def samples(self):
+        return self._samples
+
+    def __repr__(self):
+        return '<RandomUniformProbDist with %d samples>' %len(self._probs)
+
 
 @compat.python_2_unicode_compatible
 class DictionaryProbDist(ProbDistI):
diff --git a/nltk/tag/hmm.py b/nltk/tag/hmm.py
--- a/nltk/tag/hmm.py
+++ b/nltk/tag/hmm.py
@@ -83,7 +83,7 @@
                               ConditionalProbDist, DictionaryProbDist,
                               DictionaryConditionalProbDist,
                               LidstoneProbDist, MutableProbDist,
-                              MLEProbDist, UniformProbDist)
+                              MLEProbDist, RandomProbDist)
 from nltk.metrics import accuracy
 from nltk.util import LazyMap
 from nltk.compat import python_2_unicode_compatible, izip, imap
@@ -870,12 +870,12 @@ def train_unsupervised(self, unlabeled_sequences, **kwargs):
         # given an existing model
         model = kwargs.get('model')
         if not model:
-            priors = UniformProbDist(self._states)
+            priors = RandomProbDist(self._states)
             transitions = DictionaryConditionalProbDist(
-                            dict((state, UniformProbDist(self._states))
+                            dict((state, RandomProbDist(self._states))
                                   for state in self._states))
             outputs = DictionaryConditionalProbDist(
-                            dict((state, UniformProbDist(self._symbols))
+                            dict((state, RandomProbDist(self._symbols))
                                   for state in self._states))
             model = HiddenMarkovModelTagger(self._symbols, self._states,
                             transitions, outputs, priors)


Test patch: