rgb to planar:  check for out of bounds coordinates