Remove Ref template

This is dead code, and `opImplicitCastTo` is not an actual operator.