I don't see why this should speed things up (by any noticeable amount).  Here's my reasoning:  Lookup in a set is just as fast as lookup in a dictionary, so using a dict is not faster than a set.  You do save time by creating the dict once (during initialization) rather than every time predict is called.  However, I expect bottleneck in `predict()` to be the E-M step which follows.  Can you explain your test methodology (results of `line_profiler` showing the line-by-line breakdown would be the most convincing)?  Perhaps you are testing this with a trivial sentence (resulting in very fast E-M step) and a very long `self.tokens`.

There is also a danger in using your scheme (as is).  Both `self.tokendict` and `self.tokens` are public attributes, and they are not set up to track one another.  This could be made safe by setting up `getter` and `setter` methods (using the `property` and `setter` decorators (see [this post](http://stackoverflow.com/questions/6618002/python-property-versus-getters-and-setters) and subsequent answers).  The idea is to change `self.tokendict` into `self._tokendict` (making it private) and then use the `setter` on `self.tokens` to ensure that every time `self.tokens` is modified, `self._tokendict` is reset.  Actually, it would be better to use `self._tokenset = set(self.tokens)` since all we really need is a set.