Pytorch lightning model freezes after some epoch

mrjarvis: <p>I’m training a model on image and text input pairs from Flickr30k. Nothing special, just a Resnet18 for image and an Embedding + GRU network for text.</p>
<p>I’m training the model with Pytorch Lightning running on two GPUs with a DDP strategy, 16-bit precision, 512 batch size, and 8 workers in total. I defined a ModelCheckpoint that saves the 5 best iterations and an EarlyStopping callback. Both callbacks monitor the <em>val_loss</em>.</p>
<pre><code class="lang-auto">def train_dataloader(self) -&gt; DataLoader:
        return DataLoader(
            self.train_data,
            batch_size=self.batch_size,
            num_workers=self.conf.data.num_workers,
            pin_memory=True,
            shuffle=True,
            drop_last=True,
            collate_fn=collate_fn,
        )

    def val_dataloader(self) -&gt; DataLoader:
        return DataLoader(
            self.val_data,
            batch_size=self.batch_size,
            num_workers=self.conf.data.num_workers,
            pin_memory=True,
            drop_last=True,
            collate_fn=collate_fn,
        )
</code></pre>
<p>The problem is that after starting the sixth epoch, the model appears to have entered an infinite loop. There is a 100% used GPU and CPU usage has dropped dramatically. I can’t understand where the problem may be. Some idea?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed80ff1b5b40f5301d7a3987d0b21b70fdec9b12.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ed80ff1b5b40f5301d7a3987d0b21b70fdec9b12" title="training"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed80ff1b5b40f5301d7a3987d0b21b70fdec9b12_2_690x138.jpeg" alt="training" data-base62-sha1="xT3x7E1acbpgpxD85c13BEjKIka" width="690" height="138" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed80ff1b5b40f5301d7a3987d0b21b70fdec9b12_2_690x138.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed80ff1b5b40f5301d7a3987d0b21b70fdec9b12_2_1035x207.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed80ff1b5b40f5301d7a3987d0b21b70fdec9b12_2_1380x276.jpeg 2x" data-dominant-color="69495F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">training</span><span class="informations">1920×385 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/1/2115855f192f849dbc04a724f2d21c4a05949bbf.png" data-download-href="https://discuss.pytorch.org/uploads/default/2115855f192f849dbc04a724f2d21c4a05949bbf" title="cpu utilization"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/2115855f192f849dbc04a724f2d21c4a05949bbf_2_690x283.png" alt="cpu utilization" data-base62-sha1="4IFRD7JeyoBc29z3x9Q3Ld5ziaP" width="690" height="283" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/2115855f192f849dbc04a724f2d21c4a05949bbf_2_690x283.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/2115855f192f849dbc04a724f2d21c4a05949bbf_2_1035x424.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/1/2115855f192f849dbc04a724f2d21c4a05949bbf_2_1380x566.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cpu utilization</span><span class="informations">2479×1020 127 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81cae2b3de806814d5fde28e9cd5d3efb89e9d9f.png" data-download-href="https://discuss.pytorch.org/uploads/default/81cae2b3de806814d5fde28e9cd5d3efb89e9d9f" title="gpu utilization"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81cae2b3de806814d5fde28e9cd5d3efb89e9d9f_2_690x361.png" alt="gpu utilization" data-base62-sha1="iwcgZdjlXrjIe1bPs0MH47H0MeH" width="690" height="361" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81cae2b3de806814d5fde28e9cd5d3efb89e9d9f_2_690x361.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81cae2b3de806814d5fde28e9cd5d3efb89e9d9f_2_1035x541.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/81cae2b3de806814d5fde28e9cd5d3efb89e9d9f_2_1380x722.png 2x" data-dominant-color="F8F8F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu utilization</span><span class="informations">2502×1311 256 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf1927b20580e7f43a8267f8f80ad9fcb575b745.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cf1927b20580e7f43a8267f8f80ad9fcb575b745" title="htop"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1927b20580e7f43a8267f8f80ad9fcb575b745_2_690x167.jpeg" alt="htop" data-base62-sha1="ty4JjocfVhscmRjn6I1JuPB8SIl" width="690" height="167" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1927b20580e7f43a8267f8f80ad9fcb575b745_2_690x167.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1927b20580e7f43a8267f8f80ad9fcb575b745_2_1035x250.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf1927b20580e7f43a8267f8f80ad9fcb575b745_2_1380x334.jpeg 2x" data-dominant-color="361D2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">htop</span><span class="informations">1920×465 98.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55167d5c2efcea302acb5089811bc6b3b0f39cc4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/55167d5c2efcea302acb5089811bc6b3b0f39cc4" title="nvtop"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55167d5c2efcea302acb5089811bc6b3b0f39cc4_2_690x322.jpeg" alt="nvtop" data-base62-sha1="c8ILFJGzWS6oakiM1WPWY6QHFoE" width="690" height="322" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55167d5c2efcea302acb5089811bc6b3b0f39cc4_2_690x322.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55167d5c2efcea302acb5089811bc6b3b0f39cc4_2_1035x483.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55167d5c2efcea302acb5089811bc6b3b0f39cc4_2_1380x644.jpeg 2x" data-dominant-color="38142D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nvtop</span><span class="informations">1920×896 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mohbat_Tharani: <p>Make sure you have enough number of CPU cores to handle all the workers. You could try to reduce number of worker to half number of CPUs.</p>

Sumner_Magruder: <p><a class="mention" href="/u/mohbat_tharani">@Mohbat_Tharani</a> I have the same issue on GPU and CPU and I hard set everything to</p>
<pre><code class="lang-auto"># create our PyLightning trainer to actualy train the model
trainer = pl.Trainer(
    max_epochs=10,
    gradient_clip_val=100,
    log_every_n_steps=5,
    accelerator='cpu',
    num_processes=1,
    max_time={'minutes': 2},
    limit_train_batches=2
)
</code></pre>
<p>Not only does it freeze after 3 epochs, it doesn’t time out after 2 minutes!</p>

Mohbat_Tharani: <p>Maybe Empty coda cache after some steps/epochs: torch.cuda.empty_cache()</p>