Reduce memory allocation in the main loop

A few final tweaks for reducing memory allocations in the main game loop. This is the last of the easy stuff on the memory side of things. I don't think there are any more particularly easy wins to be had just from tweaking individual functions sadly.

On the plus side, things have improved enough that the GC is doing a fairly light amount of work now, so that's nice.
The idea here was to only render the actor if at least one other player can see it. This (in theory) should have significantly cut down the number of actors that need rendering.  In practice this never quite worked, since we aren't filtering the owner!