F.relu(torch.cat()) VS. torch.cat()

solsol: <p>Hi,</p>
<p>Assume a, b are two tensors, what does F.relu(torch.cat((a, b), dim=1)) do in compare with  torch.cat((a, b), dim=1)?</p>
<p>Thanks!</p>

Nikronic: <p>Hi,<br>
<code>torch.cat()</code> is a function to concatenate tensors on a specified channel.<br>
<code>F.relu()</code> is a activation function which you can use on any tensor to get the corresponding values.</p>
<p>Here is an example:</p>
<pre data-code-wrap="python"><code class="lang-python">
a = torch.randn(3, 2, 1, 1)
b = torch.randn(3, 2, 1, 1)
b = -1*b
ab = torch.cat((a, b), dim=1)
print(ab)

tensor([[[[ 1.2391]],
         [[ 0.0619]],
         [[ 0.6406]],
         [[ 0.1950]]],
        [[[-0.5557]],
         [[-0.9740]],
         [[ 0.3454]],
         [[-1.1767]]],
        [[[-0.1938]],
         [[ 0.8195]],
         [[-1.3665]],
         [[ 1.7226]]]])

ab_relu = nn.functional.relu(ab, inplace=False)
print(ab_relu)

tensor([[[[1.2391]],
         [[0.0619]],
         [[0.6406]],
         [[0.1950]]],
        [[[0.0000]],
         [[0.0000]],
         [[0.3454]],
         [[0.0000]]],
        [[[0.0000]],
         [[0.8195]],
         [[0.0000]],
         [[1.7226]]]])
</code></pre>
<p>As you can see, all negative numbers are now zero after applying <code>Relu</code>.</p>
<p>Here is the reason based on <code>Relu</code> function definition:<br>
<a href="https://pytorch.org/docs/stable/nn.html#relu" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/nn.html#relu</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/a76ced805ec52f1febe1254f47ab3d55669e24ab.png" data-download-href="https://discuss.pytorch.org/uploads/default/a76ced805ec52f1febe1254f47ab3d55669e24ab" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/a/a76ced805ec52f1febe1254f47ab3d55669e24ab.png" alt="image" data-base62-sha1="nT774SiYBvuTnqZBNbAKZSfCMrx" width="690" height="84" data-dominant-color="1E2020"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">729×89 4.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/8/8a0ef4e610f6a547370e8f386b397bf18c17963e.png" data-download-href="https://discuss.pytorch.org/uploads/default/8a0ef4e610f6a547370e8f386b397bf18c17963e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8a0ef4e610f6a547370e8f386b397bf18c17963e_2_332x250.png" alt="image" data-base62-sha1="jHjVZxRuTU7HwzBuR20L8dVia98" width="332" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/8/8a0ef4e610f6a547370e8f386b397bf18c17963e_2_332x250.png, https://discuss.pytorch.org/uploads/default/optimized/2X/8/8a0ef4e610f6a547370e8f386b397bf18c17963e_2_498x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/8/8a0ef4e610f6a547370e8f386b397bf18c17963e_2_664x500.png 2x" data-dominant-color="FAFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">683×514 38.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

solsol: <p>Thanks! it’s now completely clear to me!</p>