RF: refactor check for nan_fill value

Take estimated error into account when checking whether nan2zero nan_fill
value is compatible with correct scaling to shared range of integer and
float.