Fix inf and NaN cosimulation cases

- Make simulator use a common format for NaN, which matches hardware.
- (unrelated) clamp reciprocal estimate at appropriate precision