Most of the complexity is introduced by the requirement that we need to run these observers serially, and at the same time some observers need to run asynchronously. 
running_observers_ caches observer pointer we just notified, then in the finish callback we can run the next observer after it;
removing_observer_ caches removed observer pointer, then in the finish callback we can tell it should be removed.