Simplify mutexes

We now only have one, which increases contention, but
avoids deadlocks