Deal with partial updates correctly

When chunks are skipped because they’re not visible, ensure that they
will still be updated even if the world hasn’t been updated