Pytorch CPU Training Optimization Freezes

aromero: <p>Hi. I’m executing the following code in an AzureML virtual machine: <a href="http://mcr.microsoft.com/azureml/base:openmpi3.1.2-ubuntu18.04" rel="noopener nofollow ugc">mcr.microsoft.com/azureml/base:openmpi3.1.2-ubuntu18.04</a>. The following Pytorch Adam Optimization of a mathematical formula in CPU mode. At a certain point, the process slows down and it takes 100 times more to finish. The point where it starts slowing down seems to be random and it depends on the capabilities of the virtual machine that I execute the code with. What could be the issue here?. Has anybody experienced a similar problem?. In my laptop and othervirtual  machines it works well, it seems to be related to the virtual machine of azureml but I’m not sure.</p>
<p>Thanks a lot in advance!</p>
<p>optimizer_d[key_dict_d] = optim.Adam(list(qt_d.values()) + [t_background], lr=learning_rate)<br>
loss_fn = torch.nn.MSELoss(reduction=‘mean’)</p>
<p>errors = []<br>
cnt_stop = -1<br>
error_optim = np.inf<br>
qt_d_optim = dict()<br>
t_background_optim = 0.0</p>
<p>for t in range(n_iters):<br>
t_pred = torch.full((df_source_points_d_shape,), 0.0, device=device, dtype=dtype)<br>
if t % 1000 == 0:<br>
logger.info(f"CURRENT ITERATION: {t}“)<br>
logger.info(f"ERROR OPTIM: {error_optim}”)<br>
logger.info(f"T BACKGROUND OPTIM: {t_background_optim}")</p>
<p>t_pred += t_background</p>
<p>for key in key_dict_l:<br>
sigma_y = torch.pow(torch.abs(xt_d[key] / x0 + 1e-6), b)<br>
enum = qt_d[key] * torch.exp(-0.5 * torch.pow(yt_d[key] + 1e-6, 2) / torch.pow(<br>
a_d[key] * sigma_y, 2)) * m_a<br>
denom = u_d[key] * math.sqrt(2 * math.pi) * a_d[key] * sigma_y * m_ch4 * omega_a * ppb</p>
<h1><a name="h-1" class="anchor" href="#h-1"></a></h1>
<p>t_pred_aux = enum / denom<br>
t_pred_aux = torch.where(torch.isnan(t_pred_aux), torch.zeros_like(t_pred_aux), t_pred_aux)<br>
t_pred_aux = t_pred_aux.type(dtype)<br>
t_pred_aux = torch.where(xt_d[key] &lt;= 0.0, torch.tensor(0.0, dtype=dtype, device=device),<br>
t_pred_aux)<br>
t_pred += t_pred_aux<br>
loss = loss_fn(t_pred, t_target)<br>
optimizer_d[key_dict_d].zero_grad(set_to_none=True)<br>
current_error = loss.item()</p>
<p>if current_error &lt; error_optim:<br>
for key in key_dict_l:<br>
qt_d_optim[key] = qt_d[key].item()</p>
<p>t_background_optim = t_background.item()</p>
<p>error_optim = current_error<br>
cnt_stop = -1<br>
else:<br>
cnt_stop += 1</p>
<p>errors.append(current_error)<br>
loss.backward()<br>
optimizer_d[key_dict_d].step()</p>
<p>for key in key_dict_l:<br>
qt_d[key].data = qt_d[key].data.clamp(0.0, np.inf)</p>
<p>if cnt_stop &gt;= early_stopping:<br>
break</p>
<p>if t_background.data &gt;= df_source_points_d[“methane”].mean():<br>
t_background.data = t_background.data.clamp(0.0, df_source_points_d[self.col_methane].mean())</p>

ptrblck: <p>Your code is unfortunately not formatted properly so it’s a bit unclear which parts are in a loop etc.<br>
In any case, check if you are increasing the computation graph by adding tensors inplace, e.g. here:</p>
<pre><code class="lang-python">t_pred += t_background
</code></pre>
<p>or by appending it a <code>list</code> etc.<br>
If that’s not the case (you should also see an increasing memory usage) I would check the machine’s health status and see if it’s reducing the clocks etc.</p>