Make the bridge remember which pushers it's seen

The remap process can take a second or two.  We should remember which
pushers we've seen, and not remap them unless we really need to.