How to efficiently normalize a batch of tensor to [0, 1]

lcy: <p>Hi,</p>
<p>I have a batch of tensor. How can I efficiently normalize it to the range of [0, 1].</p>
<p>For example,<br>
The tensor is A with dimension [batch=25, height=3, width=3]. I can use for-loop to finish this normalization like</p>
<pre><code class="lang-auto"># batchwise normalize to [0, 1] along with height and width
for i in range(batch):          
    min_ele = torch.min(A[i])
    A[i] -= min_ele
    A[i] /= torch.max(A[i])
</code></pre>
<p>However, this solution is low. Is there any efficient way?</p>
<p>Thanks!</p>

ptrblck: <p>You could calculate the min and max values directly for all samples in the batch and apply the normalization:</p>
<pre><code class="lang-python">A -= A.min(1, keepdim=True)[0]
A /= A.max(1, keepdim=True)[0]
</code></pre>

lcy: <p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Thanks for your reply. However, I want to calculate the minimum and maximum element along with both height and width dimension.</p>
<p>For example, we have a tensor a=[[1,2],[3,4]], the min/max element should be 1 and 4</p>
<pre><code class="lang-auto">a = torch.Tensor([[1,2],[3,4]])
torch.min(a)  # this function will return 1
torch.max(a) # return 4
</code></pre>
<p>I have tried your solution, it gives a vector.</p>
<pre><code class="lang-auto">a = torch.Tensor([[1,2],[3,4]])
a.min(1, keepdim=True)[0] # this gives [1, 2]
a.max(1, keepdim=True)[0] # this gives [3, 4]
</code></pre>

ptrblck: <p>Based on your code snippet I assumed <code>batch</code> would correspond to the batch size, and my code snippet would yield the same result as yours.<br>
Could you post the shape of <code>A</code> and an executable code snippet (using random values for <code>A</code>)?</p>

lcy: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Here it is</p>
<pre><code class="lang-auto">batch_size = 3
height = 2
width = 2

A = torch.randint(2, 11, (batch_size, height, width)).float()
AA = A.clone()
print(A)

# I can get what I want from below for-loop solution
for i in range(batch_size):
    A[i] -= torch.min(A[i])
    A[i] /= torch.max(A[i])

# Your solution
AA -= AA.min(1, keepdim=True)[0]
AA /= AA.max(1, keepdim=True)[0]

print(A)  # A and AA are different
print(AA)

</code></pre>

ptrblck: <p>Thanks for the code.<br>
This should work:</p>
<pre><code class="lang-python">AA = AA.view(A.size(0), -1)
AA -= AA.min(1, keepdim=True)[0]
AA /= AA.max(1, keepdim=True)[0]
AA = AA.view(batch_size, height, width)
</code></pre>

lcy: <p>Got you. Thanks!</p>
<p>Is your solution much faster than for-loop?</p>

ptrblck: <p>For your specified sizes, I get these numbers on a CPU:</p>
<pre><code class="lang-python"># your method
402 µs ± 26.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

# my suggestion
115 µs ± 7.22 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
</code></pre>

Red-Eyed: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> !</p>
<p>strange, but your approach with view’s is very slow.</p>
<p>It is faster than loop approach when I use <code>timeit</code>, but inference pipeline got slower in 10 times (with for loop is about 50 FPS, with views about 5 FPS)</p>
<p>EDIT 1:<br>
Just added <code>torch.cuda.synchronize()</code></p>
<ol>
<li>for loop: 0.5 ms</li>
<li>view approach: 150 ms</li>
</ol>
<p>I don’t understand what happens, view shouldn’t change tensor itself (from continuous to non continuous)</p>
<p>Do you have any thoughts?</p>
<p>Additional info:<br>
I use CUDA tensor with shape [B, 3, 1024, 1024]<br>
torch version: 1.2.0<br>
cuda version: 10.0.130<br>
GPU: NVIDIA QUADRO GV100<br>
OS: linux</p>

ptrblck: <p>The <code>view</code> operation should be really cheap, as it only changes the meta-data, i.e. no copy will be performed and you would get an error, if a copy is necessary.</p>
<p>Could you post your profiling code so that I could take a look, please?</p>

Red-Eyed: <p>my profiling code is here:</p>
<pre><code class="lang-auto">    @contextmanager
    def timeit(msg):
        start = time.time_ns()
        yield
        end = time.time_ns()

        result = round((end - start) * 1e-6, 2)
        msg = f"timeit: {result:&lt;10} ms " + msg

        print(msg)
</code></pre>
<p>then I use it like so:</p>
<pre><code class="lang-auto">with timeit("normalization"):
    # code here
    torch.cuda.synchronize()
</code></pre>
<p>But overall speed (on 1k images) is really different, it is notable w/o any measurements.</p>

Red-Eyed: <p>The performance drop in is in <code>min()</code> function<br>
I read that it calls <code>item()</code>, and it is slow.</p><aside class="quote" data-post="3" data-topic="32847">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/e5b9ba/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/speed-of-torch-max/32847/3">Speed of torch.max</a> 
  </div>
  <blockquote>
    Thanks for clarification! Seems like it is synchronization thing resulting inaccurate timing of .item() operation. 
Since torch.max() return a int number, does that mean it will call .item() in the end and that is the reason why torch.max seems “slow”?
  </blockquote>
</aside>


ptrblck: <p>The <code>min</code> and <code>max</code> operations return a tensor in my example, so no synchronizing <code>item()</code> operation is performed.</p>
<p>I used this code snippet:</p>
<pre><code class="lang-python">
batch_size = 3
height = 2
width = 2

A = torch.randint(2, 11, (batch_size, height, width)).float().cuda()
AA = A.clone()
print(A)

def fun1(A):
    for i in range(batch_size):
        A[i] -= torch.min(A[i])
        A[i] /= torch.max(A[i])

# Your solution
def fun2(AA):
    AA = AA.view(AA.size(0), -1)
    AA -= AA.min(1, keepdim=True)[0]
    AA /= AA.max(1, keepdim=True)[0]
    AA = AA.view(batch_size, height, width)


nb_iters = 100
torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    fun1(A)
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)


torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    fun2(AA)
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
</code></pre>
<p>And get these results:</p>
<ul>
<li>CPU: original: <code>9.6135e-05 sec/iter</code>, mine: <code>2.3348e-05 s/iter</code>
</li>
<li>GPU: original: <code>0.0004302668 s/iter</code>, mine: <code>7.0183e-05 s/iter</code>
</li>
</ul>
<p>Note that the original workload is really small, so using:</p>
<pre><code class="lang-python">batch_size = 300
height = 200
width = 200
</code></pre>
<p>I get:</p>
<ul>
<li>CPU: <code>0.0646844482421875 s/iter</code> vs. <code>0.01887556552886963 s/iter</code>
</li>
<li>GPU: <code>0.040595355033874514 s/iter</code> vs. <code>0.0023879456520080567 s/iter</code>
</li>
</ul>

Red-Eyed: <p>Thank you for checking it out!<br>
Just run your code as is (w/o any modifications, so it’s cuda tensor) and:<br>
on sizes 1:</p>
<pre><code class="lang-auto">batch_size = 300
height = 200 
width = 200 
</code></pre>
<p>got</p>
<pre><code class="lang-auto">0.028152191638946535
0.0026354169845581054
</code></pre>
<p>on sizes 2:</p>
<pre><code class="lang-auto">batch_size = 8 
height = 1024 
width = 1024 
</code></pre>
<p>got</p>
<pre><code class="lang-auto">0.0010659313201904297
0.05072437047958374
</code></pre>
<p>and on sizes 3:</p>
<pre><code class="lang-auto">batch_size = 2
height = 4096 
width = 4096 
</code></pre>
<p>got</p>
<pre><code class="lang-auto">0.0024710512161254883
0.7927418446540833
</code></pre>
<p>So, it gets slower when <code>h</code> and <code>w</code> is increasing, isn’t it weird?</p>
<p>Update 1:<br>
same situation on torch version 1.5.0</p>

ptrblck: <p>I think the overhead is created by the <code>torch.max</code> call with a dimension keyword, as it’ll also return the indices.<br>
In the last use case the batch dimension can more or less be ignored, as it’s much smaller than the flattened <code>height*width</code>.<br>
If I’m not mistaken, there was recently a feature request to add a <code>return_indices</code> argument to <code>torch.max</code>.</p>

Red-Eyed: <p>Got it.<br>
I’m confirming that overhead is due to dimension keyword: just tried <code>torch.min(AA)</code> (view tensor) and didn’t noticed any overhead.</p>
<p>Thank you for the help!</p>

synapse: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="65122">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">AA = AA.view(A.size(0), -1)
AA -= AA.min(1, keepdim=True)[0]
AA /= AA.max(1, keepdim=True)[0]
AA = AA.view(batch_size, height, width)
</code></pre>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> , say I have a <code>dataloader</code> and want to normalize my whole dataset using the min_max scaling solution above. Would it be a good approach to do it on every batch (where the mins and max would differ)? Or what would be a better approach?</p>

ptrblck: <p>Usually you wouldn’t normalize with the batch statistics directly, since you would also need to do the same during your inference/deployment code.<br>
Depending on your use case it could work (e.g. if your test use case is also using the same batch size) or it could change the behavior of your model a bit, since the <code>min</code> and <code>max</code> values depend on the actual noise (the effect might be small, so it could still work, but you would need to verify it).</p>

synapse: <p>I see, thanks for the reply.<br>
How could your snippet be used for normalizing the samples in a <code>dataloader</code> then?<br>
Say I got the following, is this the correct approach?:</p>
<pre><code class="lang-auto">for i_batch, batch in enumerate(iter(dataloader), start=1):
     mixed_audio_ba, voice_audio_ba, _ = batch

     mixed_audio_ba -= mixed_audio_ba.min(1, keepdim=True)[0]
     mixed_audio_ba /= mixed_audio_ba.max(1, keepdim=True)[0]

     voice_audio_ba-= voice_audio_ba.min(1, keepdim=True)[0]
     voice_audio_ba/= voice_audio_ba.max(1, keepdim=True)[0]
     
     ...

     model_out = model(mixed_audio_ba, voice_audio_ba)
</code></pre>

ptrblck: <p>The posted approach would also normalize the entire batch. As I previously mentioned, it could work but note that this normalization now depends on the batch size and could thus change the behavior of the model if the batch size is changed e.g. during inference. The change might be minimal and the model might still perform well, so you would need to check it in your use case.</p>