Using Linear Layer and Manually Implementing Linear Layer gives different results

Rachit: <p>So pretty much what the title says. Using nn.Linear and manually implementing it keeps giving me different results. The loss for nn.Linear is much higher than the loss I see when I manually implement it (to the point where results from nn.Linear is unusable). Here is the code that I have.</p>
<p>Manual Implementation:</p>
<pre><code class="lang-auto">num_cols = 3  # Postive, Negative, Neutral

A = torch.randn((1, num_cols), requires_grad=True)
b = torch.randn(1, requires_grad=True)

def model(X):
    return A.mm(x_data) + b

def loss(y_predicted, y_target):
    return torch.sqrt(torch.mean((y_predicted - y_target) ** 2))

optimizer = torch.optim.Adam([A, b], lr=10)
num_epochs = 10000

with trange(num_epochs, desc="Training Multi-variable Linear Regression") as progress_bar:
    for _ in progress_bar:
        optimizer.zero_grad()
        y_pred = model(x_data)
        curr_loss = loss(y_pred, y_data)
        curr_loss.backward()
        optimizer.step()
        progress_bar.set_postfix(loss=curr_loss.data)
        # print(f"Epoch: {curr_epoch}, Loss: {curr_loss}")
</code></pre>
<p>Graph Produced:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/984232f1889383da9c81b8d30eb958ade918c2fc.png" alt="image" data-base62-sha1="lIWpbsADypkUwkV4kfeCLrXjB80" width="399" height="248"><br>
Weights after training: <code>tensor([[ 165.0836,  307.2172, -130.3902]], requires_grad=True)</code></p>
<p>PyTorch’s Linear Layer:</p>
<pre><code class="lang-auto">class PlainSentimentCaster(nn.Module):
    def __init__(self, DIM_FEATURES=3):
        super().__init__()
        self.DIM_FEATURES = DIM_FEATURES
        self.lin_layer1 = nn.Linear(self.DIM_FEATURES, 1)
        self.lin_layer1.weight = nn.Parameter(torch.randn((1, num_cols), requires_grad=True))
    
    def forward(self, X):
        return self.lin_layer1(X)


def train(model: nn.Module, X, Y_target, num_epochs=10000):
    optimizer = torch.optim.Adam(model.parameters(), lr=10)
    with trange(num_epochs, desc="Training TF-IDF Sentiment Linear Layer Model") as progress_bar:
        for _ in progress_bar:
            optimizer.zero_grad()
            y_pred = model.forward(X)
            curr_loss = loss(y_pred, Y_target)
            curr_loss.backward()
            optimizer.step()
            progress_bar.set_postfix(loss=curr_loss.data)


baseline_model_3 = PlainSentimentCaster(DIM_FEATURES=3)
train(baseline_model_3, x_data.float().t(), y_data.float().t())
</code></pre>
<p>Graph Produced:<br>
<a href="https://imgur.com/a/WKtgzUU" class="onebox" target="_blank" rel="noopener nofollow ugc">https://imgur.com/a/WKtgzUU</a><br>
Weights after training: <code>tensor([[-0.9212, -1.6551,  2.8089]], requires_grad=True)</code></p>
<p>Does anyone know what might be going on here?</p>

AlphaBetaGamma96: <p>Firstly, I’d check you use the same starting weights by printing the weights at the beginning of training and fix the RNG seed via <code>torch.manual_seed</code>. Also, <code>nn.Linear</code> uses kaiming_uniform initialisation where you’re using <code>torch.randn</code> which is a distributed in accordance to a gaussian. So, your initial weights are most likely entirely different.</p>
<p>Secondly, your manual implementation of the Linear layer differs from the <code>nn.Linear</code> (<a href="https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=nn%20linear#torch.nn.Linear" class="inline-onebox" rel="noopener nofollow ugc">Linear — PyTorch 1.10.0 documentation</a>).</p>
<p>Your implementation is y = A@x + b whereas PyTorch’s implemenetation is x@A^T + b (where A and b are the weight and bias respectively). Although the output shape is the same, the derivatives are different w.r.t A, e.g. dy/dA is x is the first case and x^T in the second case.</p>