Do not invalidate group rect on only children change