Remove the sublayers when switching modes.