Don't cache float pixels if contrast map is not used
Use less precise remapping if images are posterized

Avoids edge case where all pixels take slow search path