Only refresh neighbor chunks when block neighbors chunks