Learning - Why does .unsqueeze(-1) magically work?

btb: <p>Getting the hang of PyTorch by doing a simple linear regression.</p>
<p>When I use my own “custom” linear regression formula, I get fine results without changing my y_tensor (from a numpy array).</p>
<p>When I use the built in PyTorch linear regression class, I have to pass “y_tensor = y_tensor.unsqueeze(-1)” other wise I get the error:</p>
<pre><code class="lang-auto">/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py:432: UserWarning: Using a target size (torch.Size([824])) that is different to the input size (torch.Size([824, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  return F.mse_loss(input, target, reduction=self.reduction)
</code></pre>
<p>Code for not having to unsqueeze…</p>
<pre><code class="lang-auto">X_tensor = torch.from_numpy(X_train)
X_tensor = X_tensor.float()
y_tensor = torch.tensor(y_train.values,dtype=torch.float)

dep_vars = X_tensor.shape[1]

weights = torch.rand(dep_vars,requires_grad=True,dtype=torch.float)
bias = torch.rand(1,requires_grad=True,dtype=torch.float)

num_iterations = 1000
learning_rate = 0.1

def model(X): # Model function of the form y = w*X + b
    return torch.mv(X_tensor, weights) + bias 

loss_fn = torch.nn.MSELoss()  # Loss function MSE from PyTorch
# Optimizer using my weights and bias Tensor from PyTorch
optimizer = torch.optim.SGD([weights,bias],lr=learning_rate)  

for epoch in range(0,num_iterations): # Loop Through
    predicted = model(X_tensor)  # Compute predicted values
    loss = loss_fn(predicted, y_tensor) # Compute the loss
    if epoch % (num_iterations/10) == 0:
        print('loss at step ', ': ', loss)
    loss.backward() # Gradient of loss through backward propogation
    optimizer.step() # Optimizer update the weights
    optimizer.zero_grad() # Optimizer zero out the gradient for next step

print('weights: ', weights) # Print final weights and biases
print('bias: ', bias)
</code></pre>
<p>Code requiring unsqueeze…</p>
<pre><code class="lang-auto">X_tensor = torch.from_numpy(X_train)
X_tensor = X_tensor.float()
y_tensor = torch.tensor(y_train.values,dtype=torch.float)
y_tensor = y_tensor.unsqueeze(-1)

dep_vars = X_tensor.shape[1]

learning_rate = .1
num_iterations = 1000

model = torch.nn.Linear(dep_vars, 1, bias=True)
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)

for epoch in range(0,num_iterations):

    pred = model(X_tensor) # Make predictions of Linear Model
    loss = loss_fn(pred, y_tensor) # Calcualte Loss
    loss.backward() # Backward pass
    optimizer.step() # Update weights 
    optimizer.zero_grad() # Zero out gradient

    if epoch % (num_iterations / 10) == 0:
        print(f'{epoch} Loss: {loss}')

for param in model.parameters():
    print(param)
</code></pre>
<p>Any ideas?</p>

harsha_g: <p>There’s nothing magic about this. What you got is a warning and not an error. If you look at the documentation for MSELoss it says the target should have the same shape as the input which is what the <code>unsqueeze</code> operation ensures by adding an extra dimension. Hope this helps.</p>

btb: <p>Ok - so thanks for writing that out, it helps.</p>
<p>Basically my model is specified output as [y,1] (pred here) so I have to specify the other input to the loss function also as a [y,1] instead of just [y].</p>
<p>Is that a fair explanation?</p>

harsha_g: <p>Yes. And vice versa too. If your targets are shaped [y], you shape the output from the model as [y].</p>