Actually remove the entity when remove is called