Get rid of all the locking

By making selected fields final and using AtmoicReference and
ConcurrentHashMap, all the synchronization can be removed while
(hopefully) preserving the semantics.

Also adds TODO comments for some parts of the code that might cause
problems