It must be the pre-formation of the dict that is speeding things up, as you
suggest.  I was running text content of random webpages through it and
profiling using something like 'python -m cProfile -s time myscript.py >
profile.out'.  The counts formation was limiting.  Once I changed
it, _dirichlet_expectation() becomes the slowest step, as you expected.
 Your suggestion about switching to a set and hiding it seems good.  I'll
implement it.

I'd love to see predict() parallelized.  Any tips?  I'm looking at the
parallel module now, but I don't really understand what's available yet.

best,

-Louis


On Wed, May 28, 2014 at 4:55 PM, Ian Langmore <notifications@github.com>wrote:

> In rosetta/text/vw_helpers.py:
>
> > @@ -468,7 +469,7 @@ def predict(
> >
> >          counts = Counter(tokenized_text)
> >          counts = pd.Series(
> > -            {k: counts[k] for k in counts if k in set(self.tokens)}
>
> I don't see why this should speed things up (by any noticeable amount).
> Here's my reasoning: Lookup in a set is just as fast as lookup in a
> dictionary, so using a dict is not faster than a set. You do save time by
> creating the dict once (during initialization) rather than every time
> predict is called. However, I expect bottleneck in predict() to be the
> E-M step which follows. Can you explain your test methodology (results of
> line_profiler showing the line-by-line breakdown would be the most
> convincing)? Perhaps you are testing this with a trivial sentence
> (resulting in very fast E-M step) and a very long self.tokens.
>
> There is also a danger in using your scheme (as is). Both self.tokendictand
> self.tokens are public attributes, and they are not set up to track one
> another. This could be made safe by setting up getter and setter methods
> (using the property and setter decorators (see this post<http://stackoverflow.com/questions/6618002/python-property-versus-getters-and-setters>and subsequent answers). The idea is to change
> self.tokendict into self._tokendict (making it private) and then use the
> setter on self.tokens to ensure that every time self.tokens is modified,
> self._tokendict is reset. Actually, it would be better to use self._tokenset
> = set(self.tokens) since all we really need is a set.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/columbia-applied-data-science/rosetta/pull/33/files#r13163067>
> .
>