Add blocking in unit tests for more determinism

We should reimplement SerializedTaskInvoker, so it does not do blocking. Once it is done, we can provide Scheduler constructor parameter, which will allow us to inject Schedulers.test implementation. Once it is done we can get rid of latches/blocking calls in unit tests.
I am looking at this right now.