Fix over zealous active view removal - Fixes #1657