Fix being unable to remove hat in Hat Hunting mode.