Remove deleted children.
Delete child from group_id_to_child if necessary.