Only perform expensive check if the cast is unsafe