Don't force nearest for colortest against black

Should fix #4217 and be relatively safe.  Probably it's a common colortest ref color that's safe?

-[Unknown]