Reduce memory allocation in the main loop

A few final tweaks for reducing memory allocations in the main game loop. This is the last of the easy stuff on the memory side of things. I don't think there are any more particularly easy wins to be had just from tweaking individual functions sadly.

On the plus side, things have improved enough that the GC is doing a fairly light amount of work now, so that's nice.
Maybe move the predicate into the `Any()`?