Reduce the cost of an expensive loop in bounds inference

Optimizing the comparison in this loop in bounds inference reduced the compile time of the FFT by 5-10%. Not huge, but still significant (IMHO).

This could probably be optimized even further to avoid the O(N) algorithm, but profiling/experiments show this loop to have negligible cost with these commits so I didn't spend too much more time on this.