Which parameters torch.nn forward() method takes in? Is PyTorch forward() equivalent to standard forward pass?

R.Mukhamadiarov: <p>Hello!</p>
<p>I’ve implemented <a href="https://github.com/RusFortunat/java_ML_library/tree/main/src/main/java" rel="noopener nofollow ugc">my own code in Java</a> (and in C++ as well) that constructs and trains the neural networks on MNIST hand-written number dataset, using simple stochastic gradient descent while achieving 89-90% guessing accuracy. My problem is when I try to load PyTorch network parameters that I got from training on the same MNIST dataset and pass them to my own <code>forward()</code> function, PyTorch model always predicts the same number (different model, different number), i.e. 10% accuracy which you get when always predicting the same number.</p>
<p>Am I missing something? Does PyTorch use just network parameters to do regular forward prop: compute a matrix product of activations and weights, add biases, and then apply activation function when needed? Or does it perform some other operations or use other parameters in addition to tensor behind the scenes that I’m not aware of? Provided that the network structure, the training dataset, and the <code>forward()</code> methods are the same, I was expecting at least comparable results when loading PyTorch parameters to my own implementation of supervised learning methods.</p>
<p>The <code>forward()</code> method that I’ve implemented in my own training method is equivalent to the <code>forward()</code> method I used in PyTorch:</p>
<pre><code class="lang-auto">def forward(self, x):
        x = torch.flatten(x, 1) # flatten all dimensions except batch
        x = F.relu(self.fc1(x))
        return self.fc2(x)
</code></pre>
<p>I save the PyTorch trained network parameters with this code:</p>
<pre><code class="lang-auto">torch.set_printoptions(precision=20,threshold=sys.maxsize)
with open(file, 'w') as f:
    # Print model's state_dict
    f.write("Model's state_dict:")
    for param_tensor in fcNet.state_dict():
        output_string = str(fcNet.state_dict()[param_tensor])
        f.write(output_string)
</code></pre>
<p>The PyTorch accuracy that I get from <a href="https://github.com/RusFortunat/alternative-ML-lib-C2plus/blob/main/image_classification/check_data_set.py" rel="noopener nofollow ugc">my PyTorch training code</a> is about 91%.</p>

Eduardo_Lawson: <p>I think you have some problems:</p>
<ul>
<li>First you are saving a tensor as a string</li>
<li>Destroy the structure/shape of the tensor</li>
<li>Makes it impossible to properly reload the weights back into a model</li>
</ul>
<p>I always suggest using <code>torch.save</code>, here is a tutorial: <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/saving_loading_models.html</a></p>

R.Mukhamadiarov: <p>Hi Eduardo,</p>
<p>and thank you for your answer. Perhaps I haven’t made it clear, but I am not loading the model parameters back into PyTorch code or for any code that uses PyTorch, but rather I read these parameters to load them into my own ML code that I’ve written.</p>
<p>Cheers, R</p>

Eduardo_Lawson: <p>Now i see the issue, maybe <a class="mention" href="/u/ptrblck">@ptrblck</a> can help you out</p>