Repository: dmlc/dgl

Issue:
Fail to train graphsage model with ogbn-papers100M dataset and 8 GPU
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

1. Open examples/pytorch/graphsage/lightning/node_classification.py
2. Replace `dataset = DglNodePropPredDataset('ogbn-products')`  to  `dataset = DglNodePropPredDataset("ogbn-papers100M")`
3. Replace `gpus=[0, 1, 2, 3]`  to  `gpus=[0, 1, 2, 3, 4, 5, 6, 7]`
4. Run `python examples/pytorch/graphsage/lightning/node_classification.py`

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
**error log：**
> /opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
/opt/conda/lib/python3.7/multiprocessing/semaphore_tracker.py:144: UserWarning: semaphore_tracker: There appear to be 1 leaked semaphores to clean up at shutdown
  len(cache))
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/notebook/data/group/graph/frame/dglx-v1.0.0/ogp-compute/dglx/python/node_classification.py", line 194, in <module>
    trainer.fit(model, datamodule=datamodule)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 769, in fit
    self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 719, in _call_and_handle_interrupt
    return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 82, in launch
    start_method=self._start_method,
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 136, in join
    signal_name=name
torch.multiprocessing.spawn.ProcessExitedException: process 6 terminated with signal SIGKILL
## Expected behavior
<!-- A clear and concise description of what you expected to happen. -->
The case should support large graph with 8 gpu

## Environment

 - DGL Version (e.g., 1.0): 0.8.1
 - Backend Library & Version (e.g., PyTorch 0.4.1, MXNet/Gluon 1.3): PyTorch 
 - OS (e.g., Linux): Linux
 - How you installed DGL (`conda`, `pip`, source): pip
 - Build command you used (if compiling from source):
 - Python version: 3.7
 - CUDA/cuDNN version (if applicable): 10.2
 - GPU models and configuration (e.g. V100): V100 * 8
 - Memory: 512GB
 - CPU: Intel(R) Xeon(R) Gold 6242 CPU

## Additional context
<!-- Add any other context about the problem here. -->




Discussion:

Resolving patch:
diff --git a/examples/pytorch/graphsage/lightning/node_classification.py b/examples/pytorch/graphsage/lightning/node_classification.py
--- a/examples/pytorch/graphsage/lightning/node_classification.py
+++ b/examples/pytorch/graphsage/lightning/node_classification.py
@@ -116,6 +116,7 @@ def val_dataloader(self):
     dataset = DglNodePropPredDataset('ogbn-products')
     graph, labels = dataset[0]
     graph.ndata['label'] = labels.squeeze()
+    graph.create_formats_()
     split_idx = dataset.get_idx_split()
     train_idx, val_idx, test_idx = split_idx['train'], split_idx['valid'], split_idx['test']
     datamodule = DataModule(graph, train_idx, val_idx, [15, 10, 5], 1024, dataset.num_classes)


Test patch: