More Unit conversion speedup

See comments in https://github.com/astropy/astropy/pull/1715