Repository: horovod/horovod

Issue:
Spark Lightning MNIST fails on GPU
From buildkite.

```
/horovod/examples/spark/pytorch/pytorch_lightning_spark_mnist.py --num-proc 2 --work-dir /work --data-dir /data --epochs 3"' in service test-gpu-gloo-py3_8-tf2_4_3-keras2_3_1-torch1_7_1-mxnet1_6_0_p0-pyspark3_1_2 | 2m 46s
-- | --
  | $ docker-compose -f docker-compose.test.yml -p buildkite27b20b242900470484e62a9743c390b7 -f docker-compose.buildkite-6387-override.yml run --name buildkite27b20b242900470484e62a9743c390b7_test-gpu-gloo-py3_8-tf2_4_3-keras2_3_1-torch1_7_1-mxnet1_6_0_p0-pyspark3_1_2_build_6387 -v /var/lib/buildkite-agent/builds/buildkite-2x-gpu-v510-i-0c5beeec60df27818-2/horovod/horovod/artifacts:/artifacts --rm test-gpu-gloo-py3_8-tf2_4_3-keras2_3_1-torch1_7_1-mxnet1_6_0_p0-pyspark3_1_2 /bin/sh -e -c 'bash -c "OMP_NUM_THREADS=1 /spark_env.sh python /horovod/examples/spark/pytorch/pytorch_lightning_spark_mnist.py --num-proc 2 --work-dir /work --data-dir /data --epochs 3"'
  | Creating buildkite27b20b242900470484e62a9743c390b7_test-gpu-gloo-py3_8-tf2_4_3-keras2_3_1-torch1_7_1-mxnet1_6_0_p0-pyspark3_1_2_run ... done
  | 21/08/18 15:13:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  | Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  | Setting default log level to "WARN".
  | To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
  | num_partitions=20
  | writing dataframes
  | train_data_path=file:///work/intermediate_train_data.0
  | val_data_path=file:///work/intermediate_val_data.0
  | train_partitions=18
  | val_partitions=2
  | /usr/local/lib/python3.8/dist-packages/horovod/spark/common/util.py:509: FutureWarning: The 'field_by_name' method is deprecated, use 'field' instead
  | metadata, avg_row_size = make_metadata_dictionary(train_data_schema)
  | train_rows=48721
  | val_rows=5384
  | 2021-08-18 15:14:26.091848: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
  | 2021-08-18 15:14:26.091875: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  | 2021-08-18 15:14:27.292047: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
  | 2021-08-18 15:14:27.292079: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  | 2021-08-18 15:14:28.372275: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
  | 2021-08-18 15:14:28.372304: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  | Wed Aug 18 15:14:31 2021[1]<stdout>:Training data of rank[1]: train_rows:48721, batch_size:64, _train_steps_per_epoch:380.
  | Wed Aug 18 15:14:34 2021[1]<stdout>:Creating trainer with:
  | Wed Aug 18 15:14:34 2021[1]<stdout>:  {'accelerator': 'horovod', 'gpus': 1, 'callbacks': [<__main__.MyDummyCallback object at 0x7f62c12d1fd0>, <pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint object at 0x7f6261f33490>, <pytorch_lightning.callbacks.early_stopping.EarlyStopping object at 0x7f6261b9a1f0>], 'max_epochs': 3, 'logger': <pytorch_lightning.loggers.tensorboard.TensorBoardLogger object at 0x7f62c12d1790>, 'log_every_n_steps': 50, 'resume_from_checkpoint': None, 'checkpoint_callback': True, 'num_sanity_val_steps': 0, 'reload_dataloaders_every_epoch': False, 'progress_bar_refresh_rate': 38, 'terminate_on_nan': False}
  | Wed Aug 18 15:14:34 2021[1]<stdout>:Starting to init trainer!
  | Wed Aug 18 15:14:34 2021[1]<stdout>:Trainer is initialized.
  | Wed Aug 18 15:14:34 2021[1]<stdout>:pytorch_lightning version=1.3.8
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:192.168.48.2<0>
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  | Wed Aug 18 15:14:34 2021[1]<stdout>:
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:192.168.48.2<0>
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Using network Socket
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO threadThresholds 8/8/64 \| 16/8/64 \| 8/8/64
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Trees [0] -1/-1/-1->1->0\|0->1->-1/-1/-1 [1] -1/-1/-1->1->0\|0->1->-1/-1/-1
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Could not enable P2P between dev 1(=1e0) and dev 0(=1d0)
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Could not enable P2P between dev 1(=1e0) and dev 0(=1d0)
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Channel 00 : 1[1e0] -> 0[1d0] via direct shared memory
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Could not enable P2P between dev 1(=1e0) and dev 0(=1d0)
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Could not enable P2P between dev 1(=1e0) and dev 0(=1d0)
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO Channel 01 : 1[1e0] -> 0[1d0] via direct shared memory
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
  | Wed Aug 18 15:14:34 2021[1]<stdout>:b436722175a2:337:369 [1] NCCL INFO comm 0x7f62ac01c6c0 rank 1 nranks 2 cudaDev 1 busId 1e0 - Init COMPLETE
  | Wed Aug 18 15:14:35 2021[1]<stdout>:Setup train dataloader
  | Wed Aug 18 15:14:35 2021[1]<stdout>:[train dataloader]: Initializing petastorm dataloader with batch_size=64shuffling_queue_capacity=24360, limit_step_per_epoch=380
  | Wed Aug 18 15:14:35 2021[1]<stdout>:Apply the AsyncDataLoaderMixin on top of the data loader, async_loader_queue_size=64.
  | Wed Aug 18 15:14:35 2021[1]<stdout>:setup val dataloader
  | Wed Aug 18 15:14:35 2021[1]<stdout>:[val dataloader]: Initializing petastorm dataloader with batch_size=64shuffling_queue_capacity=0, limit_step_per_epoch=42
  | Wed Aug 18 15:14:35 2021[1]<stdout>:Apply the AsyncDataLoaderMixin on top of the data loader, async_loader_queue_size=64.
  | Wed Aug 18 15:14:35 2021[1]<stdout>:Start generating batches from async data loader.
  | Wed Aug 18 15:14:35 2021[1]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:36 2021[1]<stdout>:training data batch size: torch.Size([64])
  | Wed Aug 18 15:14:38 2021[1]<stdout>:[train dataloader]: Reach limit_step_per_epoch. Stop at step 380.
  | Wed Aug 18 15:14:38 2021[1]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:38 2021[1]<stdout>:A train epoch ended.
  | Wed Aug 18 15:14:38 2021[1]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:38 2021[1]<stdout>:Start generating batches from async data loader.
  | Wed Aug 18 15:14:38 2021[1]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:39 2021[1]<stdout>:validation data batch size: torch.Size([64])
  | Wed Aug 18 15:14:39 2021[1]<stdout>:[val dataloader]: Reach limit_step_per_epoch. Stop at step 42.
  | Wed Aug 18 15:14:39 2021[1]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:39 2021[1]<stdout>:A val epoch ended.
  | Wed Aug 18 15:14:39 2021[1]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:40 2021[1]<stdout>:Start generating batches from async data loader.
  | Wed Aug 18 15:14:40 2021[1]<stdout>:training data batch size: torch.Size([64])
  | Wed Aug 18 15:14:40 2021[1]<stdout>:[val dataloader]: Reach limit_step_per_epoch. Stop at step 42.
  | Wed Aug 18 15:14:40 2021[1]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:42 2021[1]<stdout>:[train dataloader]: Reach limit_step_per_epoch. Stop at step 380.
  | Wed Aug 18 15:14:42 2021[1]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:43 2021[1]<stdout>:A train epoch ended.
  | Wed Aug 18 15:14:43 2021[1]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:43 2021[1]<stdout>:Start generating batches from async data loader.
  | Wed Aug 18 15:14:43 2021[1]<stdout>:validation data batch size: torch.Size([64])
  | Wed Aug 18 15:14:43 2021[1]<stdout>:A val epoch ended.
  | Wed Aug 18 15:14:43 2021[1]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:43 2021[1]<stdout>:Start generating batches from async data loader.
  | Wed Aug 18 15:14:43 2021[1]<stdout>:training data batch size: torch.Size([64])
  | Wed Aug 18 15:14:43 2021[1]<stdout>:[val dataloader]: Reach limit_step_per_epoch. Stop at step 42.
  | Wed Aug 18 15:14:43 2021[1]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:45 2021[1]<stdout>:[train dataloader]: Reach limit_step_per_epoch. Stop at step 380.
  | Wed Aug 18 15:14:45 2021[1]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:46 2021[1]<stdout>:A train epoch ended.
  | Wed Aug 18 15:14:46 2021[1]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:46 2021[1]<stdout>:Start generating batches from async data loader.
  | Wed Aug 18 15:14:46 2021[1]<stdout>:validation data batch size: torch.Size([64])
  | Wed Aug 18 15:14:46 2021[1]<stdout>:[val dataloader]: Reach limit_step_per_epoch. Stop at step 42.
  | Wed Aug 18 15:14:46 2021[1]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:46 2021[1]<stdout>:A val epoch ended.
  | Wed Aug 18 15:14:46 2021[1]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:46 2021[1]<stdout>:Training ends:epcoh_end_counter=6, train_epcoh_end_counter=3, validation_epoch_end_counter=3
  | Wed Aug 18 15:14:46 2021[1]<stdout>:
  | Wed Aug 18 15:14:46 2021[1]<stdout>:Tear down petastorm readers
  | Wed Aug 18 15:14:34 2021[1]<stderr>:GPU available: True, used: True
  | Wed Aug 18 15:14:34 2021[1]<stderr>:TPU available: False, using: 0 TPU cores
  | Wed Aug 18 15:14:34 2021[1]<stderr>:/usr/local/lib/python3.8/dist-packages/petastorm/fs_utils.py:88: FutureWarning: pyarrow.localfs is deprecated as of 2.0.0, please use pyarrow.fs.LocalFileSystem instead.
  | Wed Aug 18 15:14:34 2021[1]<stderr>:  self._filesystem = pyarrow.localfs
  | Wed Aug 18 15:14:34 2021[1]<stderr>:LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [2,3]
  | Wed Aug 18 15:14:34 2021[1]<stderr>:Missing logger folder: /tmp/tmpwmgicp1r/logs/default
  | Wed Aug 18 15:14:36 2021[1]<stderr>:/usr/local/lib/python3.8/dist-packages/petastorm/pytorch.py:339: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  | Wed Aug 18 15:14:36 2021[1]<stderr>:  row_as_dict[k] = self.transform_fn(v)
  | Wed Aug 18 15:14:40 2021[1]<stderr>:[rank: 1] Metric val_loss improved. New best score: 0.500
  | Wed Aug 18 15:14:46 2021[1]<stderr>:terminate called without an active exception
  | Wed Aug 18 15:14:50 2021[1]<stderr>:Aborted (core dumped)
  | Wed Aug 18 15:14:31 2021[0]<stdout>:Training data of rank[0]: train_rows:48721, batch_size:64, _train_steps_per_epoch:380.
  | Wed Aug 18 15:14:34 2021[0]<stdout>:Creating trainer with:
  | Wed Aug 18 15:14:34 2021[0]<stdout>:  {'accelerator': 'horovod', 'gpus': 1, 'callbacks': [<__main__.MyDummyCallback object at 0x7f00ac791fd0>, <pytorch_lightning.callbacks.model_checkpoint.ModelCheckpoint object at 0x7f004d418490>, <pytorch_lightning.callbacks.early_stopping.EarlyStopping object at 0x7f004d07f1f0>], 'max_epochs': 3, 'logger': <pytorch_lightning.loggers.tensorboard.TensorBoardLogger object at 0x7f00ac791790>, 'log_every_n_steps': 50, 'resume_from_checkpoint': None, 'checkpoint_callback': True, 'num_sanity_val_steps': 0, 'reload_dataloaders_every_epoch': False, 'progress_bar_refresh_rate': 38, 'terminate_on_nan': False}
  | Wed Aug 18 15:14:34 2021[0]<stdout>:Starting to init trainer!
  | Wed Aug 18 15:14:34 2021[0]<stdout>:Trainer is initialized.
  | Wed Aug 18 15:14:34 2021[0]<stdout>:pytorch_lightning version=1.3.8
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:192.168.48.2<0>
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
  | Wed Aug 18 15:14:34 2021[0]<stdout>:
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:192.168.48.2<0>
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Using network Socket
  | Wed Aug 18 15:14:34 2021[0]<stdout>:NCCL version 2.7.8+cuda10.1
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Channel 00/02 :    0   1
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Channel 01/02 :    0   1
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO threadThresholds 8/8/64 \| 16/8/64 \| 8/8/64
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1\|-1->0->1/-1/-1 [1] 1/-1/-1->0->-1\|-1->0->1/-1/-1
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Could not enable P2P between dev 0(=1d0) and dev 1(=1e0)
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Could not enable P2P between dev 0(=1d0) and dev 1(=1e0)
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Channel 00 : 0[1d0] -> 1[1e0] via direct shared memory
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Could not enable P2P between dev 0(=1d0) and dev 1(=1e0)
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Could not enable P2P between dev 0(=1d0) and dev 1(=1e0)
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Channel 01 : 0[1d0] -> 1[1e0] via direct shared memory
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO comm 0x7f009801d090 rank 0 nranks 2 cudaDev 0 busId 1d0 - Init COMPLETE
  | Wed Aug 18 15:14:34 2021[0]<stdout>:b436722175a2:333:379 [0] NCCL INFO Launch mode Parallel
  | Wed Aug 18 15:14:35 2021[0]<stdout>:Setup train dataloader
  | Wed Aug 18 15:14:35 2021[0]<stdout>:[train dataloader]: Initializing petastorm dataloader with batch_size=64shuffling_queue_capacity=24360, limit_step_per_epoch=380
  | Wed Aug 18 15:14:35 2021[0]<stdout>:Apply the AsyncDataLoaderMixin on top of the data loader, async_loader_queue_size=64.
  | Wed Aug 18 15:14:35 2021[0]<stdout>:setup val dataloader
  | Wed Aug 18 15:14:35 2021[0]<stdout>:[val dataloader]: Initializing petastorm dataloader with batch_size=64shuffling_queue_capacity=0, limit_step_per_epoch=42
  | Wed Aug 18 15:14:35 2021[0]<stdout>:Apply the AsyncDataLoaderMixin on top of the data loader, async_loader_queue_size=64.
  | Wed Aug 18 15:14:35 2021[0]<stdout>:Epoch 0:   0%\|          \| 0/422 [00:00<?, ?it/s] Start generating batches from async data loader.
  | Wed Aug 18 15:14:35 2021[0]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:36 2021[0]<stdout>:training data batch size: torch.Size([64])
  | Wed Aug 18 15:14:38 2021[0]<stdout>:Epoch 0:  72%\|███████▏  \| 304/422 [00:02<00:00, 118.47it/s, loss=1.2, v_num=0] [train dataloader]: Reach limit_step_per_epoch. Stop at step 380.
  | Wed Aug 18 15:14:38 2021[0]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:38 2021[0]<stdout>:Epoch 0:  90%\|█████████ \| 380/422 [00:03<00:00, 125.42it/s, loss=1.05, v_num=0]A train epoch ended.
  | Wed Aug 18 15:14:38 2021[0]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:38 2021[0]<stdout>:                                                  Start generating batches from async data loader.
  | Wed Aug 18 15:14:38 2021[0]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:40 2021[0]<stdout>:[val dataloader]: Reach limit_step_per_epoch. Stop at step 42.
  | Wed Aug 18 15:14:40 2021[0]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:40 2021[0]<stdout>:validation data batch size: torch.Size([64])
  | Wed Aug 18 15:14:40 2021[0]<stdout>:Epoch 0: 100%\|██████████\| 422/422 [00:04<00:00, 91.34it/s, loss=1.05, v_num=0] A val epoch ended.
  | Wed Aug 18 15:14:40 2021[0]<stdout>:A train or eval epoch ended. 38/42 [00:01<00:00, 24.30it/s]
  | Wed Aug 18 15:14:40 2021[0]<stdout>:Epoch 1:   0%\|          \| 0/422 [00:00<?, ?it/s, loss=1.05, v_num=0]Start generating batches from async data loader.
  | Wed Aug 18 15:14:40 2021[0]<stdout>:training data batch size: torch.Size([64])
  | Wed Aug 18 15:14:40 2021[0]<stdout>:[val dataloader]: Reach limit_step_per_epoch. Stop at step 42.
  | Wed Aug 18 15:14:40 2021[0]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:42 2021[0]<stdout>:Epoch 1:  72%\|███████▏  \| 304/422 [00:02<00:00, 138.95it/s, loss=0.939, v_num=0][train dataloader]: Reach limit_step_per_epoch. Stop at step 380.
  | Wed Aug 18 15:14:42 2021[0]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:43 2021[0]<stdout>:Epoch 1:  90%\|█████████ \| 380/422 [00:02<00:00, 138.97it/s, loss=0.774, v_num=0]A train epoch ended.
  | Wed Aug 18 15:14:43 2021[0]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:43 2021[0]<stdout>:                                                  Start generating batches from async data loader.
  | Wed Aug 18 15:14:43 2021[0]<stdout>:validation data batch size: torch.Size([64])?it/s]
  | Wed Aug 18 15:14:43 2021[0]<stdout>:A val epoch ended.
  | Wed Aug 18 15:14:43 2021[0]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:43 2021[0]<stdout>:Epoch 2:   0%\|          \| 0/422 [00:00<?, ?it/s, loss=0.774, v_num=0]Start generating batches from async data loader.
  | Wed Aug 18 15:14:43 2021[0]<stdout>:training data batch size: torch.Size([64])
  | Wed Aug 18 15:14:43 2021[0]<stdout>:[val dataloader]: Reach limit_step_per_epoch. Stop at step 42.
  | Wed Aug 18 15:14:43 2021[0]<stdout>:[val dataloader]: Start to generate batch data. limit_step_per_epoch=42
  | Wed Aug 18 15:14:45 2021[0]<stdout>:Epoch 2:  72%\|███████▏  \| 304/422 [00:02<00:00, 135.23it/s, loss=0.639, v_num=0][train dataloader]: Reach limit_step_per_epoch. Stop at step 380.
  | Wed Aug 18 15:14:45 2021[0]<stdout>:[train dataloader]: Start to generate batch data. limit_step_per_epoch=380
  | Wed Aug 18 15:14:46 2021[0]<stdout>:Epoch 2:  90%\|█████████ \| 380/422 [00:02<00:00, 134.50it/s, loss=0.656, v_num=0]A train epoch ended.
  | Wed Aug 18 15:14:46 2021[0]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:46 2021[0]<stdout>:                                                  Start generating batches from async data loader.
  | Wed Aug 18 15:14:46 2021[0]<stdout>:validation data batch size: torch.Size([64])?it/s]
  | Wed Aug 18 15:14:46 2021[0]<stdout>:A val epoch ended.
  | Wed Aug 18 15:14:46 2021[0]<stdout>:A train or eval epoch ended.
  | Wed Aug 18 15:14:46 2021[0]<stdout>:Epoch 2: 100%\|██████████\| 422/422 [00:02<00:00, 14Training ends:epcoh_end_counter=6, train_epcoh_end_counter=3, validation_epoch_end_counter=3
  | Wed Aug 18 15:14:46 2021[0]<stdout>:
  | Wed Aug 18 15:14:46 2021[0]<stdout>:Epoch 2: 100%\|██████████\| 422/422 [00:02<00:00, 143.65it/s, loss=0.656, v_num=0]
  | Wed Aug 18 15:14:46 2021[0]<stdout>:Tear down petastorm readers
  | Wed Aug 18 15:14:34 2021[0]<stderr>:GPU available: True, used: True
  | Wed Aug 18 15:14:34 2021[0]<stderr>:TPU available: False, using: 0 TPU cores
  | Wed Aug 18 15:14:34 2021[0]<stderr>:/usr/local/lib/python3.8/dist-packages/petastorm/fs_utils.py:88: FutureWarning: pyarrow.localfs is deprecated as of 2.0.0, please use pyarrow.fs.LocalFileSystem instead.
  | Wed Aug 18 15:14:34 2021[0]<stderr>:  self._filesystem = pyarrow.localfs
  | Wed Aug 18 15:14:34 2021[0]<stderr>:LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [2,3]
  | Wed Aug 18 15:14:34 2021[0]<stderr>:2021-08-18 15:14:34.868268: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/nvidia/lib:/usr/local/nvidia/lib64
  | Wed Aug 18 15:14:34 2021[0]<stderr>:2021-08-18 15:14:34.868287: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
  | Wed Aug 18 15:14:35 2021[0]<stderr>:
  | Wed Aug 18 15:14:35 2021[0]<stderr>:  \| Name       \| Type      \| Params
  | Wed Aug 18 15:14:35 2021[0]<stderr>:-----------------------------------------
  | Wed Aug 18 15:14:35 2021[0]<stderr>:0 \| conv1      \| Conv2d    \| 260
  | Wed Aug 18 15:14:35 2021[0]<stderr>:1 \| conv2      \| Conv2d    \| 5.0 K
  | Wed Aug 18 15:14:35 2021[0]<stderr>:2 \| conv2_drop \| Dropout2d \| 0
  | Wed Aug 18 15:14:35 2021[0]<stderr>:3 \| fc1        \| Linear    \| 16.1 K
  | Wed Aug 18 15:14:35 2021[0]<stderr>:4 \| fc2        \| Linear    \| 510
  | Wed Aug 18 15:14:35 2021[0]<stderr>:-----------------------------------------
  | Wed Aug 18 15:14:35 2021[0]<stderr>:21.8 K    Trainable params
  | Wed Aug 18 15:14:35 2021[0]<stderr>:0         Non-trainable params
  | Wed Aug 18 15:14:35 2021[0]<stderr>:21.8 K    Total params
  | Wed Aug 18 15:14:35 2021[0]<stderr>:0.087     Total estimated model params size (MB)
  | Wed Aug 18 15:14:35 2021[0]<stderr>:/usr/local/lib/python3.8/dist-packages/petastorm/pytorch.py:339: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
  | Wed Aug 18 15:14:35 2021[0]<stderr>:  row_as_dict[k] = self.transform_fn(v)
  | Wed Aug 18 15:14:40 2021[0]<stderr>:[rank: 0] Metric val_loss improved. New best score: 0.479
  | Wed Aug 18 15:14:40 2021[0]<stderr>:/usr/local/lib/python3.8/dist-packages/pytorch_lightning/callbacks/model_checkpoint.py:610: LightningDeprecationWarning: Relying on `self.log('val_loss', ...)` to set the ModelCheckpoint monitor is deprecated in v1.2 and will be removed in v1.4. Please, create your own `mc = ModelCheckpoint(monitor='your_monitor')` and use it as `Trainer(callbacks=[mc])`.
  | Wed Aug 18 15:14:40 2021[0]<stderr>:  warning_cache.deprecation(
  | Wed Aug 18 15:14:46 2021[0]<stderr>:terminate called without an active exception
  | Wed Aug 18 15:14:51 2021[0]<stderr>:Aborted (core dumped)
  | Exception in thread Thread-3:
  | Traceback (most recent call last):
  | File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
  | self.run()
  | File "/usr/lib/python3.8/threading.py", line 870, in run
  | self._target(*self._args, **self._kwargs)
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/runner.py", line 141, in run_spark
  | result = procs.mapPartitionsWithIndex(mapper).collect()
  | File "/usr/local/lib/python3.8/dist-packages/pyspark/rdd.py", line 949, in collect
  | sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
  | File "/usr/local/lib/python3.8/dist-packages/py4j/java_gateway.py", line 1304, in __call__
  | return_value = get_return_value(
  | File "/usr/local/lib/python3.8/dist-packages/pyspark/sql/utils.py", line 111, in deco
  | return f(*a, **kw)
  | File "/usr/local/lib/python3.8/dist-packages/py4j/protocol.py", line 326, in get_return_value
  | raise Py4JJavaError(
  | py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
  | : org.apache.spark.SparkException: Job 4 cancelled part of cancelled job group horovod.spark.run.0
  | at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2258)
  | at org.apache.spark.scheduler.DAGScheduler.handleJobCancellation(DAGScheduler.scala:2154)
  | at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleJobGroupCancelled$4(DAGScheduler.scala:1048)
  | at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
  | at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
  | at org.apache.spark.scheduler.DAGScheduler.handleJobGroupCancelled(DAGScheduler.scala:1047)
  | at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2407)
  | at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2387)
  | at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2376)
  | at org.apache.spark.util.EventLoop$anon$1.run(EventLoop.scala:49)
  | at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:868)
  | at org.apache.spark.SparkContext.runJob(SparkContext.scala:2196)
  | at org.apache.spark.SparkContext.runJob(SparkContext.scala:2217)
  | at org.apache.spark.SparkContext.runJob(SparkContext.scala:2236)
  | at org.apache.spark.SparkContext.runJob(SparkContext.scala:2261)
  | at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
  | at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  | at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  | at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
  | at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
  | at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:180)
  | at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
  | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  | at java.lang.reflect.Method.invoke(Method.java:498)
  | at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
  | at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
  | at py4j.Gateway.invoke(Gateway.java:282)
  | at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
  | at py4j.commands.CallCommand.execute(CallCommand.java:79)
  | at py4j.GatewayConnection.run(GatewayConnection.java:238)
  | at java.lang.Thread.run(Thread.java:748)
  |  
  | Traceback (most recent call last):
  | File "/horovod/examples/spark/pytorch/pytorch_lightning_spark_mnist.py", line 214, in <module>
  | train_model(args)
  | File "/horovod/examples/spark/pytorch/pytorch_lightning_spark_mnist.py", line 199, in train_model
  | torch_model = torch_estimator.fit(train_df).setOutputCols(['label_prob'])
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/common/estimator.py", line 35, in fit
  | return super(HorovodEstimator, self).fit(df, params)
  | File "/usr/local/lib/python3.8/dist-packages/pyspark/ml/base.py", line 161, in fit
  | return self._fit(dataset)
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/common/estimator.py", line 80, in _fit
  | return self._fit_on_prepared_data(
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/lightning/estimator.py", line 406, in _fit_on_prepared_data
  | handle = backend.run(trainer, args=(serialized_model,), env={})
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/common/backend.py", line 83, in run
  | return horovod.spark.run(fn, args=args, kwargs=kwargs,
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/runner.py", line 287, in run
  | _launch_job(use_mpi, use_gloo, settings, driver, env, stdout, stderr, executable)
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/runner.py", line 154, in _launch_job
  | run_controller(use_gloo, lambda: gloo_run(executable, settings, nics, driver, env, stdout, stderr),
  | File "/usr/local/lib/python3.8/dist-packages/horovod/runner/launch.py", line 706, in run_controller
  | gloo_run()
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/runner.py", line 154, in <lambda>
  | run_controller(use_gloo, lambda: gloo_run(executable, settings, nics, driver, env, stdout, stderr),
  | File "/usr/local/lib/python3.8/dist-packages/horovod/spark/gloo_run.py", line 68, in gloo_run
  | launch_gloo(command, exec_command, settings, nics, {}, server_ip)
  | File "/usr/local/lib/python3.8/dist-packages/horovod/runner/gloo_run.py", line 282, in launch_gloo
  | raise RuntimeError('Horovod detected that one or more processes exited with non-zero '
  | RuntimeError: Horovod detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was:
  | Process name: 1
  | Exit code: 134


```



Discussion:
Can you reproduce this locally? There is probably some useful error message in the Spark worker logs.
> Can you reproduce this locally? There is probably some useful error message in the Spark worker logs.

I'm going to get a GPU machine to check.

Resolving patch:
diff --git a/horovod/data/data_loader_base.py b/horovod/data/data_loader_base.py
--- a/horovod/data/data_loader_base.py
+++ b/horovod/data/data_loader_base.py
@@ -71,24 +71,19 @@ def __init__(self, async_loader_queue_size=64, *args, **kwargs):
             self.thread.daemon = True
             self.started = False
 
-    def __del__(self):
-        self._close_async_loader()
-        s = super()
-        if hasattr(s, "__del__"):
-            s.__del__(self)
-
-    def _close_async_loader(self):
+    def close_async_loader(self):
         """
         Close the async data loader.
         """
         print("Closing the AsyncDataLoaderMixin.")
         if self.async_loader_queue_size > 0 and self.started:
             self.finished_event.set()
-            try:
-                # Free buffer to allow worker to retry
-                self.queue.get_nowait()
-            except Empty:
-                pass
+            while True:
+                try:
+                    # Drain buffer
+                    self.queue.get_nowait()
+                except Empty:
+                    break
             self.thread.join()
 
     def _async_worker(self):
diff --git a/horovod/spark/lightning/datamodule.py b/horovod/spark/lightning/datamodule.py
--- a/horovod/spark/lightning/datamodule.py
+++ b/horovod/spark/lightning/datamodule.py
@@ -65,9 +65,14 @@ def setup(self, stage=None):
     def teardown(self, stage=None):
         if stage == "fit" or stage is None:
             if self.verbose:
-                print("Tear down petastorm readers")
+                print("Tear down: closing async dataloaders")
+            self.train_dl.close_async_loader()
+            if self.has_val:
+                self.val_dl.close_async_loader()
             if not self.inmemory_cache_all:
                 # Reader was loaded once and stopped for inmemory datalaoder.
+                if self.verbose:
+                    print("Tear down: closing petastorm readers")
                 self.train_reader.stop()
                 self.train_reader.join()
                 if self.has_val:
@@ -90,7 +95,8 @@ def train_dataloader(self):
             dataloader_class = PytorchInfiniteAsyncDataLoader
             kwargs['shuffling_queue_capacity'] = self.shuffle_size
 
-        return dataloader_class(**kwargs)
+        self.train_dl = dataloader_class(**kwargs)
+        return self.train_dl
 
     def val_dataloader(self):
         if not self.has_val:
@@ -110,4 +116,5 @@ def val_dataloader(self):
             dataloader_class = PytorchInfiniteAsyncDataLoader
             kwargs['shuffling_queue_capacity'] = 0
 
-        return dataloader_class(**kwargs)
+        self.val_dl = dataloader_class(**kwargs)
+        return self.val_dl


Test patch: