ok, I hide both tokens and tokenset so I could update tokenset whenever
tokens is changed.  Much cleaner.  thanks


On Wed, May 28, 2014 at 5:52 PM, Louis Clark <sfgypsyboy@gmail.com> wrote:

> It must be the pre-formation of the dict that is speeding things up, as
> you suggest.  I was running text content of random webpages through it and
> profiling using something like 'python -m cProfile -s time myscript.py >
> profile.out'.  The counts formation was limiting.  Once I changed
> it, _dirichlet_expectation() becomes the slowest step, as you expected.
>  Your suggestion about switching to a set and hiding it seems good.  I'll
> implement it.
>
> I'd love to see predict() parallelized.  Any tips?  I'm looking at the
> parallel module now, but I don't really understand what's available yet.
>
> best,
>
> -Louis
>
>
> On Wed, May 28, 2014 at 4:55 PM, Ian Langmore <notifications@github.com>wrote:
>
>> In rosetta/text/vw_helpers.py:
>>
>> > @@ -468,7 +469,7 @@ def predict(
>> >
>> >          counts = Counter(tokenized_text)
>> >          counts = pd.Series(
>> > -            {k: counts[k] for k in counts if k in set(self.tokens)}
>>
>> I don't see why this should speed things up (by any noticeable amount).
>> Here's my reasoning: Lookup in a set is just as fast as lookup in a
>> dictionary, so using a dict is not faster than a set. You do save time by
>> creating the dict once (during initialization) rather than every time
>> predict is called. However, I expect bottleneck in predict() to be the
>> E-M step which follows. Can you explain your test methodology (results of
>> line_profiler showing the line-by-line breakdown would be the most
>> convincing)? Perhaps you are testing this with a trivial sentence
>> (resulting in very fast E-M step) and a very long self.tokens.
>>
>> There is also a danger in using your scheme (as is). Both self.tokendictand
>> self.tokens are public attributes, and they are not set up to track one
>> another. This could be made safe by setting up getter and setter methods
>> (using the property and setter decorators (see this post<http://stackoverflow.com/questions/6618002/python-property-versus-getters-and-setters>and subsequent answers). The idea is to change
>> self.tokendict into self._tokendict (making it private) and then use the
>> setter on self.tokens to ensure that every time self.tokens is modified,
>> self._tokendict is reset. Actually, it would be better to use self._tokenset
>> = set(self.tokens) since all we really need is a set.
>>
>> —
>> Reply to this email directly or view it on GitHub<https://github.com/columbia-applied-data-science/rosetta/pull/33/files#r13163067>
>> .
>>
>
>