Cleanup the active_writes set too.