Cleanups in tiered storage and apply semaphores.

This commit adds semaphores for memory and disk storage.  It also
notes a few places where illegal state access can occur but doesn't
fix it due to compounding changes in the tests.