Training stuck at Epoch 3

username: duddal
cooked: <p>Hello,</p>
<p>I am training a custom Encoder-Decoder network but the training gets stuck at Epoch 3. Nothing happens for about 2 hours. I will share the Dataset class and the DataLoader object. The version if CUDA and GPU can be seen in the pic below.</p>
<p>Training stuck here:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6c90ef152101f3df8e5e47fbab9038c3536ec40.png" data-download-href="https://discuss.pytorch.org/uploads/default/f6c90ef152101f3df8e5e47fbab9038c3536ec40" title="error_training"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6c90ef152101f3df8e5e47fbab9038c3536ec40.png" alt="error_training" data-base62-sha1="zdadQmSI9iiQKQlKun0S0FJeU4E" width="690" height="253" data-dominant-color="151515"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error_training</span><span class="informations">1077×396 13 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>nvidia-smi output looks like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70effde8067bc3a1d5399360ef885aaadaef2dfa.png" data-download-href="https://discuss.pytorch.org/uploads/default/70effde8067bc3a1d5399360ef885aaadaef2dfa" title="nvidia_smi_during_training_stuck"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70effde8067bc3a1d5399360ef885aaadaef2dfa.png" alt="nvidia_smi_during_training_stuck" data-base62-sha1="g75EIJ55QGQRxzTj5zMRDvTDVDc" width="690" height="353" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvidia_smi_during_training_stuck</span><span class="informations">706×362 8.84 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The <code>__getitem__</code> method of the dataset class looks like this:</p>
<pre><code class="lang-auto">    def __init__(self,
                 images_dir,
                 annots_dir,
                 train=True,
                 img_size=(512, 1536),
                 stride=4,
                 model='custom',
                 transforms=None):
        """
        :param root: dataset directory
        :param filenames: filenames inside the root directory
        :param labels: Object Detection Labels
        """
        super(CustomDataset).__init__()
        self.images_dir = images_dir
        self.annots_dir = annots_dir
        self.train = train
        self.image_size = img_size
        self.stride = stride
        self.transforms = transforms
        self.model = model

        # Load the image and annotation files from the dataset
        # self.image_files, self.annot_files = self._load_image_and_annot_files()

        self.image_files = [os.path.join(self.images_dir, idx) for idx in os.listdir(self.images_dir)]
        self.annot_files = [os.path.join(self.annots_dir, idx) for idx in os.listdir(self.annots_dir)]

    def __getitem__(self, index):
        """
        :param index: index...0 to N
        :return: tensor_image and tensor_label
        """
        # Image filename from _load_image_files()
        # Load Image with _read_matrix() and label
        curr_image_filename = self.image_files[index]
        curr_annot_filename = self.annot_files[index]
        # curr_image_filename = self.image_files[index]
        # curr_annot_filename = self.annot_files[index]
        np_image = self._read_matrix(raw_img=curr_image_filename)
        np_image_normalized = np.squeeze(self._normalize_raw_img(np_image))

        # label = self.labels[index]
        boxes, classes, depths, tgts = self._load_annotations(curr_annot_filename)

        # Normalize bounding boxes: range [0, 1]
        targets_normalized = self._normalize_bbox(np_image_normalized, tgts)

        # image and the corresponding label should be a tensor
        torch_image = torch.from_numpy(np_image).reshape(1, 512, 1536).float()  # dtype: torch.float64
        torch_boxes = torch.from_numpy(boxes).type(torch.FloatTensor)
        torch_depths = torch.from_numpy(depths)

        if self.model == 'fasterrcnn':
            # For FasterRCNN: As COCO format
            area = (torch_boxes[:, 3] - torch_boxes[:, 1]) * (torch_boxes[:, 2] - torch_boxes[:, 0])
            iscrowd = torch.zeros((boxes.shape[0],), dtype=torch.int64)
            image_id = torch.Tensor([index])
            torch_classes = torch.from_numpy(classes)

            target = {'boxes': torch_boxes, 'labels': torch_classes.long(),
                      'area': area, 'iscrowd': iscrowd, 'image_id': image_id}

            return torch_image, target

        elif self.model == 'custom':
            if self.train:
                if self.transforms:
                    try:
                        tr = self.transforms()
                        transform_image, transform_boxes, labels = tr.__call__(np_image, tgts, tgts[:, :4], tgts[:, 4:])
                        transform_targets = np.hstack((np.array(transform_boxes), labels))
                        gt_tensor = gt_creator(img_size=self.image_size,
                                               stride=self.stride,
                                               num_classes=8,
                                               label_lists=transform_targets)
                        return torch.from_numpy(transform_image).float(), gt_tensor
                    except IndexError:
                        pass
                else:
                    gt_tensor = gt_creator(img_size=self.image_size,
                                           stride=self.stride,
                                           num_classes=8,
                                           label_lists=targets_normalized)
                    return torch_image, gt_tensor
            else:
                return torch_image, targets_normalized
</code></pre>
<p>And in the train.py script the DataLoader object is:</p>
<pre><code class="lang-auto">    train_loader = torch.utils.data.DataLoader(dataset=dataset,
                                               shuffle=True,
                                               batch_size=batch_size,
                                               num_workers=0,
                                               collate_fn=detection_collate,
                                               pin_memory=True)
</code></pre>
<p>Why does the training get stuck? Is there an issue in the <code>__getitem__</code> method? Or the <code>DataLoader</code>?</p>
<p>Thank You.</p>

username: duddal
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Sorry to tag you here. But could you help me out a bit?<br>
Thanks a ton.</p>

username: duddal
cooked: <p>Is this a problem with the GPU memory, i.e. just not big enough or the DataLoader <code>__getitem__</code> method?</p>
<p>Can someone help?</p>