SI-5900 Fix pattern inference regression

Deets in the commits, review by @adriaanm

/cc @paulp (we'll take another swing at SI-7886 in 2.12)