Don't force nearest for colortest against black.

Should fix #4217 and be relatively safe.  Probably it's a common colortest
ref color that's safe?