Fix for #864 - revert "fixing #563 by using delayed removal for a layer ...

...that's loading"

The original bug #563 is no longer apparent, even with this revert.