Training time keeps increasing

nuriaa: <p>Hi,<br>
I am training an actor-critic algorithm using my own code and I noticed the training time keeps increasing at every epoch.<br>
I thought it could be a problem of continuously increasing the Computational Graph, but I think I am not doing anything wrong and when keeping track of the actor_loss and critic_loss backward times it doesn’t seem to be the issue, since their respective training times don’t increase drastically.</p>
<p>This is my training function, and the graph shown below, shows time of training every 500 training epochs (1 epoch goes once through the function  ‘train’ below).</p>
<p>Any hints?<br>
Thanks a lot in advance!</p>
<pre><code class="lang-auto">def train(self, episode_num=None):
       
        self.memory.shuffle()
        for num_batch, obs in enumerate(self.data_loader):
            state, action, reward, next_state, done = obs[0]
            state = state.to(torch.float32, copy=False)
            action = action.to(torch.float32, copy=False)
            reward = reward.to(torch.float32, copy=False)
            next_state = next_state.to(torch.float32, copy=False)
            done = done.to(torch.float32, copy=False)
            current_Z, target_Z, tau_k = self.td(
                state, action, reward, next_state, done)
            
            self.critic_loss = self.criterion_critic(target_Z, current_Z, tau_k)
            self.optimizer_critic.zero_grad()
            self.critic_loss.backward()
            self.optimizer_critic.step()
            if self.num_train_steps % self.policy_update_freq == 0: 
                # Update policy network:
                actor_loss_ = self.compute_actor_loss(state)
                self.actor_loss = self.to_min*actor_loss_

                self.optimizer_actor.zero_grad()
                self.actor_loss.backward()

                self.optimizer_actor.step()
            
                # call @parameters.setter in NNQFunction
                self.target_Q_function.parameters = \
                    self.Q_function.parameters
                # call @parameters.setter in NNPolicyFunction
                self.target_policy_function.parameters = \
                    self.policy_function.parameters
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50d968a4ddc5fd0ccd61f3af7c676c824ad90257.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/50d968a4ddc5fd0ccd61f3af7c676c824ad90257" title="train_time"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50d968a4ddc5fd0ccd61f3af7c676c824ad90257_2_690x322.jpeg" alt="train_time" data-base62-sha1="bxdZC2LGhprvwHHY1pbtSsi7UJF" width="690" height="322" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50d968a4ddc5fd0ccd61f3af7c676c824ad90257_2_690x322.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50d968a4ddc5fd0ccd61f3af7c676c824ad90257_2_1035x483.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/0/50d968a4ddc5fd0ccd61f3af7c676c824ad90257_2_1380x644.jpeg 2x" data-dominant-color="FDFDFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train_time</span><span class="informations">1560×728 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Could it be that you keep increasing the size of self.memory there?<br>
Or that you save on <code>self</code> more and more stuff?</p>

nuriaa: <p>Thanks so much for your reply,<br>
self.memory keeps increasing indeed. self.memory is an Experience Replay buffer with a maximum size of 1e6, so I keep adding environment “observations” in it till it is full and then I keep updating it in a queue basis.</p>
<p>I finally found out that the step slowing time at every training epoch is when sampling the batch using the DataLoader. So I am know getting the data sampling directly from the buffer without using the DataLoader and things work far faster.<br>
This slow performance by the DataLoader happens also when having a Dataset directly loaded from an hdf5 file (so its size doesn’t keep increasing with time).<br>
Any insights what I am doing wrong?</p>
<p>Increasing num_workers doesn’t solve the issue since no part of my code is parallelized, so increasing num_workers won’t make any difference I guess.</p>

albanD: <p>The data loader is built with loading from disk in mind.<br>
In your case here, you don’t want multiple workers for sure. But <code>0</code>. Otherwise, it will try to use multiprocessing to load data which will increase the memory usage by a lot.</p>

nuriaa: <p>Okay, so if my data fits well in memory and ( hence don’t need to load from disk), it will be better in terms of speed to avoid the DataLoader part for getting the batch data.<br>
Am I correct?</p>

albanD: <p>Using the dataloader is fine for sampling but you should make sure to use 0 workers.</p>

nuriaa: <p>The plot above showing the increase in training time was when using the DataLoader with num_workers = 0.<br>
As soon as I removed the DataLoader for sampling, the speed increased a lot.</p>