Fix performance regression getting format class for Unit

Discovered by asv.
Looks fine, except I think this normally would be `is not` instead of `!=`