also delete the waiting_since hash if we destroy the totem