improve the speed of "second order" -> contacts

to bring them back in line with how they were before the
first order special-casing happened