Revert "in unsafe_ref, special case Any to fix #2458"

This reverts commit e8d6ba3a7200f7a2093a60b5b585b2253d35450e.