mimno/Mallet — pull request #10 by steveash (2014-08-18 23:55:38+00:00)
Title: Issue9 alphabet pipe thread safety

The alphabet instances when deserialized are cached in a static map. Same is true for the Pipe instances. Also alphabets insert new symbols on-the-fly when they haven't seen them before. Thus if you are doing lots of sequence tagging on many threads there is a problem when rehashing occurs in the alphabet. Since the underlying trove hashmap isn't threadsafe we can just use read/write locks. Writing is relatively rare and uncontended read lock acquisition on java6+ is fast. Some of the locks inside of loops will surely be hoisted up during inlining anyways. So performance should be ok.

* wrapped element and trove map in alphabet with read/write locks
* made pipe and alphabet read resolves use a concurrent map as well
* the semisupervised learner was the only place using the iterator() method which now returns a copy to keep thread safety. it probably didn't matter from a thread safety perspective but I went ahead and made it use the size() and lookupObject(i) methods which are safe even during modification -- since new symbols are only ever added to the end of entries array.

I've been running this successfully on a 32-core, 256gb ram server doing lots of sequence tagging whereas before it would almost always fail during rehashing. So at least in CRF sequence tagging -- i think the rest is thread safe _if_ you create CRF instances per-thread and use them in a thread confined way (I've got mine in a ThreadLocal)