Remove byref_vars from the tree earlier during resolution