Normalization [0,1] during training

MalumaDev: <p>Hi,<br>
I need to perform a normalization [0,1] over each channel of a tensor [shape(BxCxWxH)] as a part of the model and I wrote this code:</p>
<pre><code class="lang-auto">def normalize_channels(_x, every=True):
    out = torch.ones_like(_x)
    for i in range(_x.shape[0]):
        if every:
            for j in range(_x[i].shape[0]):
                min_ = _x[i][j].min()
                max_ = _x[i][j].max()
                out[i][j].copy_((_x[i][j] - min_) * (1 / (max_ - min_)))
        else:
            min_ = _x[i].min()
            max_ = _x[i].max()
            out.copy_((_x[i] - min_) * (1 / (max_ - min_)))
    return out
</code></pre>
<p>but the computational time increases too much. Someone has any ideas on how to improve the performance and how to remove the indexing?</p>

KFrank: <p>Hello Maluma!</p>
<aside class="quote no-group quote-modified" data-username="MalumaDev" data-post="1" data-topic="60403" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malumadev/48/17463_2.png" class="avatar"> MalumaDev:</div>
<blockquote>
<p>I need to perform a normalization [0,1] over each channel of a tensor [shape(BxCxWxH)] as a part of the model and I wrote this code:<br>
…<br>
but the computational time increases too much. Someone has any ideas on how to improve the performance and how to remove the indexing?</p>
</blockquote>
</aside>
<p>As a general rule, computations on tensors in pytorch run faster if<br>
you use built-in tensor operations, rather than looping over indicies.</p>
<p>Try running <a href="https://pytorch.org/docs/stable/torch.html#torch.flatten" rel="noopener nofollow ugc">torch.flatten()</a> on the last two dimensions of your tensor<br>
to get a tensor of shape(BxCxL), where L = W*H.</p>
<p>Then do your computations with things like <a href="https://pytorch.org/docs/stable/torch.html#torch.max" rel="noopener nofollow ugc">torch.max()</a>, using its<br>
<code>dim</code> argument to specify that you want to take the <code>max</code> over your<br>
tensor’s last dimension (W*H).</p>
<p>Give it a try, and if you have issues, post your code (working or<br>
not) together with any errors.</p>
<p>(If you do get this working, it might be nice to post a follow-up with<br>
before-and-after timings so we can see how much it helped.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

MalumaDev: <p>Thanks for the tip. I wrote this code:</p>
<pre><code class="lang-auto">def normalize_channels( _x, inplace=False):
        tmp = torch.flatten(_x, start_dim=2)
        _min = tmp.min(dim=-1)[0]
        _max = tmp.max(dim=-1)[0]
        del tmp
        if inplace:
            return _x.add_(_min.unsqueeze(-1).unsqueeze(-1), alpha=-1).mul_(
                1 / _max.add(_min, alpha=-1).unsqueeze(-1).unsqueeze(-1))
        else:
            return _x.add(_min.unsqueeze(-1).unsqueeze(-1), alpha=-1).mul(
                1 / _max.add(_min, alpha=-1).unsqueeze(-1).unsqueeze(-1))
</code></pre>
<p>And it performs better:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:center">ITERATIONS</th>
<th style="text-align:center">OLD (s)</th>
<th style="text-align:center">NEW (s)</th>
<th style="text-align:center">OLD-NEW (s)</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">0</td>
<td style="text-align:center">2,42E-01</td>
<td style="text-align:center">7,80E-03</td>
<td style="text-align:center">2,34E-01</td>
</tr>
<tr>
<td style="text-align:center">1</td>
<td style="text-align:center">2,38E-01</td>
<td style="text-align:center">8,23E-03</td>
<td style="text-align:center">2,30E-01</td>
</tr>
<tr>
<td style="text-align:center">2</td>
<td style="text-align:center">2,34E-01</td>
<td style="text-align:center">8,17E-03</td>
<td style="text-align:center">2,26E-01</td>
</tr>
<tr>
<td style="text-align:center">3</td>
<td style="text-align:center">2,26E-01</td>
<td style="text-align:center">8,22E-03</td>
<td style="text-align:center">2,18E-01</td>
</tr>
<tr>
<td style="text-align:center">4</td>
<td style="text-align:center">2,24E-01</td>
<td style="text-align:center">8,21E-03</td>
<td style="text-align:center">2,16E-01</td>
</tr>
<tr>
<td style="text-align:center">5</td>
<td style="text-align:center">2,54E-01</td>
<td style="text-align:center">8,19E-03</td>
<td style="text-align:center">2,46E-01</td>
</tr>
<tr>
<td style="text-align:center">mean</td>
<td style="text-align:center">2,36E-01</td>
<td style="text-align:center">8,14E-03</td>
<td style="text-align:center">2,28E-01</td>
</tr>
</tbody>
</table>
</div>