Training with DDP ended error and process in GPU died
🐛 Bug

Training process ended incorrectly with DDP. It seems that the last checkpoint saved failed, causing one process in gpu died and cannot kill.

Code sample

import argparse

import pytorch_lightning as pl
from pytorch_lightning.loggers import TensorBoardLogger
from pytorch_lightning.callbacks import ModelCheckpoint
from src.ADFA_stereo import Adfa_stereo16

check_point_callback = ModelCheckpoint(
    save_top_k = 1,
    save_last = True,
    monitor='avg_val_loss',
    mode='max'
)

parser = argparse.ArgumentParser()
parser.add_argument('--model_name',type=str,default='msc_ADFA_debug',help='model name')
parser.add_argument('--train_info',type=str,default=None,help="training information")
parser.add_argument('--num_workers',type=int ,default=8,help='Number of workers for dataloader')
parser.add_argument('--batchsize',type=int ,default=1,help='batch size during training')
parser.add_argument('--num_epochs',type=int ,default=60,help='number of epochs of training ')
parser.add_argument('--start_epoch',type=int,default=0,help="start epoch number,must be int")

parser = Adfa_stereo16.add_model_specific_args(parser)
parser = pl.Trainer.add_argparse_args(parser)

arg = parser.parse_args()
print('training option',arg)

logger = TensorBoardLogger('logs',name=arg.model_name)
cycstereo=Adfa_stereo16(arg)

trainer = pl.Trainer.from_argparse_args(arg,max_epochs=arg.num_epochs,logger=logger,default_root_dir='logs',checkpoint_callback=check_point_callback)
print("---------------------\n** %s **\n---------------------\n"%cycstereo.hparams.model_name)
trainer.fit(cycstereo)

Environment

CUDA:

GPU:

GeForce RTX 2080 Ti
GeForce RTX 2080 Ti
GeForce RTX 2080 Ti
GeForce RTX 2080 Ti


available:         True
version:           10.1.243


Packages:

numpy:             1.16.4
pyTorch_debug:     False
pyTorch_version:   1.3.0
pytorch-lightning: 0.8.4
tqdm:              4.46.1


System:

OS:                Linux
architecture:

64bit



processor:         x86_64
python:            3.7.3
version:           #217-Ubuntu SMP Tue Jul 21 04:18:15 UTC 2020



Additional context
Information about died process(PID=10007)
died process info.txt