Unset the associated object when unsetting the delegate