Avoid double free when erasing at destructor