Doh. More chunk-unloading NPE fixes

Only do chunk stuff on the server side