Don't move lower siblings up when we are being destroyed because our parent is.