Optimization: when doing Intersect, only loop over the smaller of the two sets