Repository: wenet-e2e/wenet

Issue:
tarfile.ReadError: unexpected end of data on uio
Hi, I use the uio scheme, and the data is published with nginx. Whenever I finish training the first Epoch, the following error occurs, and the running k8s multi-machine environment.

---

2022/03/21 15:13:18 2022-03-21 15:13:18,614 DEBUG CV Batch 0/3900 loss 86.512817 loss_att 80.519363 loss_ctc 100.497559 history loss 84.197869 rank 5
2022/03/21 15:13:29 2022-03-21 15:13:29,898 DEBUG CV Batch 0/4000 loss 10.645514 loss_att 10.246677 loss_ctc 11.576130 history loss 84.150134 rank 5
2022/03/21 15:13:30 2022-03-21 15:13:30,921 INFO Epoch 0 CV info cv_loss 84.02436912753343
2022/03/21 15:13:30 2022-03-21 15:13:30,921 INFO Epoch 1 TRAIN info lr 7.256e-05
2022/03/21 15:13:30 2022-03-21 15:13:30,923 INFO using accumulate grad, new batch size is 4 times larger than before
2022/03/21 15:15:55 2022-03-21 15:15:55,011 DEBUG TRAIN Batch 1/0 loss 11.397388 loss_att 10.426254 loss_ctc 13.663368 lr 0.00007264 rank 5
2022/03/21 15:16:27 2022-03-21 15:16:27,494 WARNING error to parse 2-169-1_371.wav
2022/03/21 15:16:35 2022-03-21 15:16:35,489 DEBUG TRAIN Batch 1/100 loss 48.813408 loss_att 45.244461 loss_ctc 57.140949 lr 0.00007464 rank 5
2022/03/21 15:17:15 2022-03-21 15:17:15,154 DEBUG TRAIN Batch 1/200 loss 112.027512 loss_att 104.039352 loss_ctc 130.666565 lr 0.00007664 rank 5
2022/03/21 15:17:47 Traceback (most recent call last):
2022/03/21 15:17:47   File "wenet/bin/train.py", line 268, in main
2022/03/21 15:17:47     for batch_idx, batch in enumerate(data_loader):
2022/03/21 15:17:47   File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1199, in _next_data
2022/03/21 15:17:47   File "wenet/bin/train.py", line 295, in <module>
2022/03/21 15:17:47     executor.train(model, optimizer, scheduler, train_data_loader, device,
2022/03/21 15:17:47   File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 517, in __next__
2022/03/21 15:17:47     return self._process_data(data)
2022/03/21 15:17:47     data.reraise()
2022/03/21 15:17:47     data = self._next_data()
2022/03/21 15:17:47     main()
2022/03/21 15:17:47   File "/wenet/wenet/utils/executor.py", line 41, in train
2022/03/21 15:17:47   File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
2022/03/21 15:17:47     data = next(self.dataset_iter)
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 509, in static_batch
2022/03/21 15:17:47   File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
2022/03/21 15:17:47     raise self.exc_type(msg)
2022/03/21 15:17:47   File "/usr/local/lib/python3.8/dist-packages/torch/_utils.py", line 429, in reraise
2022/03/21 15:17:47 Original Traceback (most recent call last):
2022/03/21 15:17:47     data = fetcher.fetch(index)
2022/03/21 15:17:47 tarfile.ReadError: Caught ReadError in DataLoader worker process 0.
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 567, in padding
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 457, in shuffle
2022/03/21 15:17:47   File "/usr/local/lib/python3.8/dist-packages/torch/utils/data/_utils/fetch.py", line 34, in fetch
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 485, in sort
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 238, in speed_perturb
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 423, in spec_aug
2022/03/21 15:17:47     for tarinfo in stream:
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 372, in tokenize
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 266, in compute_fbank
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 180, in filter
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 78, in tar_file_and_group
2022/03/21 15:17:47   File "/wenet/wenet/dataset/processor.py", line 213, in resample
2022/03/21 15:17:47     for sample in data:
2022/03/21 15:17:47   File "/usr/lib/python3.8/tarfile.py", line 2434, in __iter__
2022/03/21 15:17:47   File "/usr/lib/python3.8/tarfile.py", line 2312, in next
2022/03/21 15:17:47 tarfile.ReadError: unexpected end of data
2022/03/21 15:17:47     tarinfo = self.next()
2022/03/21 15:17:47     raise ReadError("unexpected end of data")


Discussion:
which is caused by incomplete download. you can add try/catch exception to avoid the problem.
I solved it temporarily by using shard-local mode, thanks! @robin1001 

Resolving patch:
diff --git a/wenet/dataset/processor.py b/wenet/dataset/processor.py
--- a/wenet/dataset/processor.py
+++ b/wenet/dataset/processor.py
@@ -49,7 +49,7 @@ def url_opener(data):
                 stream = open(url, 'rb')
             # network file, such as HTTP(HDFS/OSS/S3)/HTTPS/SCP
             else:
-                cmd = f'curl -s -L {url}'
+                cmd = f'wget -q -O - {url}'
                 process = Popen(cmd, shell=True, stdout=PIPE)
                 sample.update(process=process)
                 stream = process.stdout


Test patch: