Layer's output is inconsistent with next layer's input_value

Forceless: <p>Steps to reproduce:</p>
<ol>
<li>Load an alexnet from torchvision</li>
<li>Change model’s first weight to torch.tensor(float(‘inf’))</li>
<li>Register forward hook and check the output of first conv layer and first ReLu layer<br>
I speculate this bug was caused by torch’s measure to inf value<br>
Screen Shots:<br>
Conv Layer’s output and another parameter:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a0bcd8c6085b2d967596ac94114657c3c82ff9e.png" data-download-href="https://discuss.pytorch.org/uploads/default/9a0bcd8c6085b2d967596ac94114657c3c82ff9e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a0bcd8c6085b2d967596ac94114657c3c82ff9e_2_690x298.png" alt="image" data-base62-sha1="lYKOIL4Ple4Q7mZg5umE3KsQa1o" width="690" height="298" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a0bcd8c6085b2d967596ac94114657c3c82ff9e_2_690x298.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/a/9a0bcd8c6085b2d967596ac94114657c3c82ff9e_2_1035x447.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/a/9a0bcd8c6085b2d967596ac94114657c3c82ff9e.png 2x" data-dominant-color="2A2A2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1096×474 38.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
ReLu Layer’s output and another parameter:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e2a0b24664762597faa03811b598daf9c5db865.png" data-download-href="https://discuss.pytorch.org/uploads/default/0e2a0b24664762597faa03811b598daf9c5db865" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/e/0e2a0b24664762597faa03811b598daf9c5db865.png" alt="image" data-base62-sha1="21iLgKmlCcXXK8AnuyE7PiHsW2N" width="690" height="339" data-dominant-color="2A292B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1031×508 33.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Torchinfo’s summary of AlexNet<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a25a2e429855971d00921122ade9ec2e5aed829.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a25a2e429855971d00921122ade9ec2e5aed829" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a25a2e429855971d00921122ade9ec2e5aed829_2_673x500.png" alt="image" data-base62-sha1="60QGJ4Z0Rf8uwpwbe76UPjVJJ3P" width="673" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a25a2e429855971d00921122ade9ec2e5aed829_2_673x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a25a2e429855971d00921122ade9ec2e5aed829_2_1009x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2a25a2e429855971d00921122ade9ec2e5aed829_2_1346x1000.png 2x" data-dominant-color="262627"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1378×1023 82.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Note: input_val was a tuple</li>
</ol>

Forceless: <p>PS: Here’s what happened when I change first weight to 999999<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33cf41c483d54c83a882b203cde523d2f0f53677.png" data-download-href="https://discuss.pytorch.org/uploads/default/33cf41c483d54c83a882b203cde523d2f0f53677" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/33cf41c483d54c83a882b203cde523d2f0f53677.png" alt="image" data-base62-sha1="7okoBxHEGNsCKbJv8Zy0UAniICP" width="538" height="500" data-dominant-color="2C2B2D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">612×568 34.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e06aca1a52df20458fe906e984a3f820c8e86ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e06aca1a52df20458fe906e984a3f820c8e86ff" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e06aca1a52df20458fe906e984a3f820c8e86ff.png" alt="image" data-base62-sha1="4hCCFYM6BjFSlUIkHSrnFkNAGd1" width="690" height="383" data-dominant-color="2D3135"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1021×567 37.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>OK this seems not caused because of inf value</p>
<p>So I’m really confused</p>

Forceless: <p>Update: I find that all layer’s input data don’t value less than 0<br>
It’s like conv’s output was relued before relu layer</p>

ptrblck: <aside class="quote no-group" data-username="Forceless" data-post="3" data-topic="138651">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/forceless/48/43294_2.png" class="avatar"> Forceless:</div>
<blockquote>
<p>It’s like conv’s output was relued before relu layer</p>
</blockquote>
</aside>
<p><code>AlexNet</code> uses inplace <code>nn.ReLU</code> layers as seen <a href="https://github.com/pytorch/vision/blob/93b26da3623c9134d64f026a234d97caab0675f2/torchvision/models/alexnet.py#L18-L53">here</a>, which would explain your observation.</p>

Forceless: <p>Woo, it means output and input of relu layer should be the same if I hook after this layer was completed its work.<br>
You were so nice.<br>
Thank you!</p>