Rather than clearing grid children, we want to selective remove children so we don't blow out adorners.