Don't remove ref temps for PRIM_SET_MEMBER

Ref temps for PRIM_SET_MEMBER are always needed.