Play nicely with List<>.remove()
Further optimisation of chunk unload algorithm

Reduce fallback to expensive logic by implementing simple exemption and
popularity contest logic.