Maybe lazily initialize handle_sym, it's not used in the common case.

Also, maybe clear the ref afterwards?