rustc: Drop the visitor object from the visitor glue

Recent demoding makes the visitor glue leak. It hasn't shown up in tests
because the box annihilator deletes the leaked boxes. This affects the
new scheduler though which does not yet have a box annihilator.

I don't think there's any great way to test this besides setting up
a task that doesn't run the box annihilator and I don't know that that's
a capability we want tasks to have.