Why the function defined to take in a [784] tensor and output a [10] tensor can take in [100, 28, 28] tensor and output [100, 10] tensor?

SouthTorch: <p>Here is the myNet, defined as:</p>
<pre><code class="lang-auto">class myMLP(torch.nn.Module):

    def __init__(self):
        super(myMLP, self).__init__()

        self.model=torch.nn.Sequential(
            torch.nn.Linear(784, 200),
            torch.nn.Dropout(0.3), #drop 30%
            torch.nn.LeakyReLU(inplace=True),
            torch.nn.Linear(200, 200),
            torch.nn.Dropout(0.4), #drop 40%
            torch.nn.LeakyReLU(inplace=True),
            torch.nn.Linear(200, 10),
        )
    
    def forward(self, x):
        x=self.model(x)

        return x
</code></pre>
<p>which is defined to take in [784] tensor and output [10] tensor.<br>
while when I use it in the following way, and successfully:</p>
<pre><code class="lang-auto">for epoch in range(maxepoch):
    myNet.train()
    for batch_idx, (data, target) in enumerate(train_loader):

        data=data.view(-1, 28*28)
        print(data.size())
        data, target=data.to(device), target.to(device)
        logits=myNet(data)
        print(logits.size())
        loss=loss_function(logits, target)
</code></pre>
<p>It took in a [100, 28, 28] data each time without reporting error, and output [100, 10] logits.</p>
<p>How does that work? Thanks.</p>

ptrblck: <aside class="quote no-group" data-username="SouthTorch" data-post="1" data-topic="154637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c89c15/48.png" class="avatar"> SouthTorch:</div>
<blockquote>
<p>It took in a [100, 28, 28] data each time without reporting error, and output [100, 10] logits.</p>
</blockquote>
</aside>
<p>Additionally you are flattening the inputs to <code>[-1, 28*28=784]</code> in this line of code:</p>
<pre data-code-wrap="python"><code class="lang-python">data=data.view(-1, 28*28)
</code></pre>
<p>which passes the input to the model in the expected shape.</p>

SouthTorch: <p>Thanks.</p>
<p>But, how about the 100 in [100, 784], is that a built-in automatic extension? the broadcasting principle? I didn’t expect broadcasting can go that far!</p>

ptrblck: <p>The <code>-1</code> will indicate to the <code>view</code> operation to move “everything else” to this dimension and broadcasting is not used.</p>
<aside class="quote no-group" data-username="SouthTorch" data-post="3" data-topic="154637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c89c15/48.png" class="avatar"> SouthTorch:</div>
<blockquote>
<p>I didn’t expect broadcasting can go that far!</p>
</blockquote>
</aside>
<p>I don’t know how to interpret this statement.</p>

SouthTorch: <p>Look, the class is defined to take in [784], while I sent in a batch of 100 of [784] by using [-1, 784], and the class calculated the whole batch, which is not a behavior defined by me. I thought it is broadcasting working here but in a confusing way.</p>

ptrblck: <aside class="quote no-group" data-username="SouthTorch" data-post="5" data-topic="154637">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c89c15/48.png" class="avatar"> SouthTorch:</div>
<blockquote>
<p>which is not a behavior defined by me</p>
</blockquote>
</aside>
<p>It is exactly the defined behavior.<br>
The linear layer expects inputs in the shape <code>[batch_size, in_features]</code>. Your input has 100 samples (<code>batch_size=100</code>) and you are flattening the input to <code>[batch_size=100, in_features=784]</code> so the layer will process this batch as specified. No broadcasting is used in this case as the batch dimension is expected (in all layers).</p>

SouthTorch: <p>Yes, you are right, it is i confused myself. Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>