Refactor controller manager.

I also added a controller manager test, and fixed a locking problem and a potential race. Integration & e2e tests still pass.