Fix bug in colormap limit inference when input has nans

Closes #336