Fix various issues with prematurely aborting VE load

Add checks to make sure parts of the target are acutally
active before trying to tear them down or use them.

This fixes a couple of issues which appeared when the VE was
closed before it had a chance to finish loading.

Bug: 48520
Bug: 47813
Change-Id: Ide5def0e983bab49108b40008fc170957c7fc2a2
Merge "Fix various issues with prematurely aborting VE load"