Wasn't checking if we used layout before removing screen objects from layout.  This speeds up certain cases.