[nomaster] SI-8152 Backport variance validator performance fix

	% time qbin/scalac test/files/pos/t8146-performance.scala

	real	0m2.015s
	user	0m2.892s
	sys	0m0.215s

	% time scalac-hash v2.10.3 test/files/pos/t8146-performance.scala

	real	1m13.652s
	user	1m14.245s
	sys	0m0.508s

Cherry-picks one hunk from 882f8e64.
Merge pull request #3364 from retronym/ticket/8152

[nomaster]  Backport variance validator performance fix