Skip multiple downconvs when building integer-to-integer Convs.