Ensures that collections with allow_remove don't re-create child elements if they all were removed