Fixes for setInteractionDelegate

See #1264 for original issue.
Mostly a refactor to make it a matched pair (`addEvents` + `removeEvents`). They now do the exact opposite of each other, with the same checks and guards.