Use a more sophisticated trick to catch recursive instantiated calls

Though theoretically occasionally useful, they are a major graph-blowup
hazard. The old trick stopped working reliably when we switched to worklists.

Issue #142