[BUGFIX] Avoid non-gc-able call-controllers/players

Circular references make trouble for GC. This also happens to fix #471 by accident. I don't expect the extra allocations to be a large performance burden.