Dataloader leads to a slow backward process?

username: MTandHJ
cooked: <p>I use the following code to train a model:</p>
<pre><code class="lang-auto">    def train_per_epoch(self, epoch: int):
        with torch.profiler.profile(
            activities=[
                torch.profiler.ProfilerActivity.CPU,
                torch.profiler.ProfilerActivity.CUDA,
            ],
            schedule=torch.profiler.schedule(wait=1, warmup=1, active=2, repeat=1),
            on_trace_ready=torch.profiler.tensorboard_trace_handler('./plog'),
            record_shapes=True,
            profile_memory=True,
            with_stack=True
        ) as profiler:
            for data in self.dataloader:
                data = self.dict_to_device(data)
                loss = self.model(data)

                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

                profiler.step()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97644f9fc1f5118c57727b3512f3116c7e25fa54.png" data-download-href="https://discuss.pytorch.org/uploads/default/97644f9fc1f5118c57727b3512f3116c7e25fa54" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97644f9fc1f5118c57727b3512f3116c7e25fa54_2_690x354.png" alt="image" data-base62-sha1="lBh0PD6KUQBz13Z2bg7XCiH9eVC" width="690" height="354" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97644f9fc1f5118c57727b3512f3116c7e25fa54_2_690x354.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97644f9fc1f5118c57727b3512f3116c7e25fa54_2_1035x531.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/97644f9fc1f5118c57727b3512f3116c7e25fa54_2_1380x708.png 2x" data-dominant-color="D4DFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2001×1028 114 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Each epoch takes about <strong>94s</strong>. If I remove the backward process, it only takes <strong>26s</strong> per epoch.</p>
<pre><code class="lang-auto">            for data in self.dataloader:
                data = self.dict_to_device(data)
                loss = self.model(data)

                # self.optimizer.zero_grad()
                # loss.backward()
                # self.optimizer.step()

                profiler.step()
</code></pre>
<p>It seems that the backward pass is much slow.<br>
However, when I replace the dataloader with a type of random data, each epoch (with the same number of iterations) takes only <strong>35s</strong>:</p>
<pre><code class="lang-auto">            # for data in self.dataloader:
            for _ in range(712): # 712 is the number of iterations
                data = {
                    self.model.ISeq: torch.randint(0, 1000, (1024, 50)),
                    self.model.IPos: torch.randint(0, 1000, (1024, 50)),
                    self.model.INeg: torch.randint(0, 1000, (1024, 50)),
                }
                data = self.dict_to_device(data)
                loss = self.model(data)

                self.optimizer.zero_grad()
                loss.backward()
                self.optimizer.step()

                profiler.step()
</code></pre>

username: MTandHJ
cooked: <p>Emmm.</p>
<p>It seems that the way to look up embeddings makes the differences.</p>
<pre><code class="lang-auto">            # 1
            posEmbds = self.Item.embeddings(data[self.IPos] + self.NUM_PADS)
            negEmbds = self.Item.embeddings(data[self.INeg] + self.NUM_PADS)

            # 2
            # posEmbds = self.Item.embeddings.weight[self.NUM_PADS:][data[self.IPos]]
            # negEmbds = self.Item.embeddings.weight[self.NUM_PADS:][data[self.INeg]]
</code></pre>
<h1><a name="p-453615-h-2-is-much-slower-than-1-1" class="anchor" href="#p-453615-h-2-is-much-slower-than-1-1"></a>2 is much slower than # 1.</h1>

username: Soumya_Kundu
cooked: <p>I am not sure about this – but essentially in the random sequence, you are <em>not loading</em> the data and just passing it?</p>