Different input for the same network for pytorch

username: AndrewSoul
cooked: <p>Hello.<br>
Three different input: images1, images1_sub(a subset of images1), images2,.<br>
One network: model.</p>
<pre><code class="lang-auto">... ...
optimizer.zero_grad()

output1，intermediate_feature_map_1  = model(images1)
# get output and intermediate feature map
output1_sub, intermediate_feature_map_sub1 = model(images1_sub)
# get output and intermediate feature map
output2, intermediate_feature_map2 = model(images2)

# calculate the cross_entropy ce_loss1 for output1
ce_loss1 = cross_entropy(output1, label)
# calculate the cross_entropy ce_loss2 for output2
ce_loss2 = cross_entropy(output2, label)  # the same label
# the distance of intermediate feature map for images_sub1 and images2 
#(the shape of images_sub1 is the same as the shape of images2  )
loss_d =  distance(intermediate_feature_map_sub1, intermediate_feature_map2 )
loss =  ce_loss1 + ce_loss2 + loss_d 
loss.backward()
optimizer.step()
</code></pre>
<p>I’d like to sum the loss and backward. Then update parameter for network.<br>
Is there anything wrong with my code?</p>
<p>Appreciated for your help!</p>

username: ptrblck
cooked: <p>The code looks alright.<br>
Do you see any strange things happening?</p>

username: AndrewSoul
cooked: <p>Hi, thanks for your reply!<br>
I have not seen anything strange happening so far. But I have a question:<br>
images1_sub in the code above is a subset of images1.  If <code>images1_sub = images1[:half]</code><br>
Compared to the code:<br>
<code>output1_sub, intermediate_feature_map_sub1 = model(images1_sub)</code><br>
will the following code achieve the same goal and save some calculation resouces?<br>
<code>output1_sub, intermediate_feature_map_sub1 = output1[:half], intermediate_feature_map1[:half]</code><br>
Thank you.</p>

username: ptrblck
cooked: <p>This won’t necessarily give the same output.<br>
E.g. if you are using conv layers with some padding, the outputs might differ:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 1, 3, 1, 1)
x = torch.randn(1, 1, 6, 6)
output = conv(x)
output_half = conv(x[:, :, 3:, 3:])

print(output[:, :, 3:, 3:] == output_half)
&gt; tensor([[[[0, 0, 0],
            [0, 1, 1],
            [0, 1, 1]]]], dtype=torch.uint8)
</code></pre>

username: AndrewSoul
cooked: <p>I got it.  Thanks for your reply!</p>

username: AndrewSoul
cooked: <p>Yeah, <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"> but, I think my original question is : about half batch of images ~</p>
<pre><code class="lang-auto">images1_sub = images1[:half]
output1_sub, intermediate_feature_map_sub1 = model(images1_sub)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">outptut, inter = model(images)
output_half, inter_half = outptut[:half], inter[:half]
</code></pre>
<p>Is output1_sub different from output_half ? I have tested it, maybe same?</p>

username: ptrblck
cooked: <p>Thanks for the information. I though you would like to split the spatial size.<br>
If you split the batch, you would have to take care of e.g. batch norm layers, as the running stats will differ.<br>
Besides that it should yield the same output.</p>
<p>What results did your test yield?</p>

username: AndrewSoul
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="50113">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If you split the batch, you would have to take care of e.g. batch norm layers, as the running stats will differ.</p>
</blockquote>
</aside>
<p>Yes, with batch norm layers, I tested dummy data. The output is different. Appreciate for your reply!</p>