Avoid forcing nearest/disabling linear if possible

If we know that the test is trivially true, we don't need to worry about the test.  May help #4405.

-[Unknown]