Why can't I reimplement my tensorflow model with pytorch?

username: kkpytorch
cooked: <p>I am developing a model in tensorflow and find that it is good on my specific evaluation method. But when I transfer to pytorch, I can’t achieve the same results. I have checked the model architecture, the weight init method, the lr schedule, the weight decay, momentum and epsilon used in BN layer, the optimizer, and the data preprocessing. All things are the same. But I can’t get the same results as in tensorflow. Anybody have met the same problem?</p>

username: InnovArul
cooked: <p>You may need to give specific details about different components of your project. (data, model,  some code etc).  Without that, it’s difficult to point the problem.</p>

username: KFrank
cooked: <p>Hi kk!</p>
<p>I built a simple (just for practice) network in tensorflow – two<br>
fully-connected layers with biases.  I successfully rewrote it in<br>
pytorch and got <em>statistically</em> the same results.</p>
<p>I could not find matching built-in weight-initialization methods.<br>
(The weights and/or biases differed one way or another.)  So I<br>
initialized my weights and biases by hand.</p>
<p>Tensorflow was choosing my batches (stochastic gradient<br>
descent), while in pytorch I was choosing my own.  I didn’t<br>
bother trying to make the batch selection match exactly.<br>
So specific numbers generated by tensorflow and pytorch<br>
differed randomly, but, as I said above, they were statistically<br>
the same.</p>
<p>(In my case, it took me a little while to realize that the seemingly<br>
same weight-initialization methods has slightly different recipes<br>
for initializing the biases.)</p>
<p>Yours is a much more complicated network than my toy model,<br>
but I do believe that you should be able to get them to give you<br>
the same results.</p>
<p>Good luck.</p>
<p>K. Frank</p>

username: kkpytorch
cooked: <p>Thank you, KFrank.<br>
In my case, I have trained the model in TF repeatedly. I find that even I remove color augmentation, the results remain good. But in pytorch, the results are all bad.<br>
There must exists something I don’t know affecting the results.<br>
Now I plan to try some small network and check the models again.<br>
I am not the only one meeting this problem.</p>

username: kkpytorch
cooked: <p>Here， I have dumped all the tensorflow model weights into pytorch model.<br>
I use a simple input in test and find that the forward pass and the backward pass are all the same.<br>
In my case, I use resnet110 as the backbone. The dataset is cifar10.<br>
The only difference is that I use cifar10 bin version in TF and cifar10 python version in pytorch.<br>
Except this, the two models use the same initial weight, the same optimizer (SGD with momentum 0.9), the same batch normalization layer with momentum=0.99 and epsilon=1e-3, the same lr schedule and weight decay=2e-4.<br>
But the results in pytorch are still worse than that in TF.<br>
I really don’t know why.</p>

username: Oli
cooked: <p>Hi,</p>
<p>I found two differences that might be relevant. The first one is different epsilon for the batch norm layers, mentioned in your first post. The second one is how the padding works. Pytorch add equal padding on all the sides whilst tensorflow’s “same” padding will not do this. This goes into effect when stride=2 for resnet.</p>
<p>If you have compared a specific input in both of the models and seen that they produce the same output I guess that this is not the problem.</p>
<p>Let us know if you figure it out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: kkpytorch
cooked: <p>Thank you for you reply.<br>
You are right. But I have found these points too.<br>
The bn momentum and bn epsilon are different.<br>
I also write my own padding like that in TF before conv layer with stride=2.<br>
I have test the two models with the same weights and the same input tensor.<br>
I follow each model layer by layer in the forward pass. All tensors are the same.<br>
For the backward pass, I only checked one step of update and find that the first conv layer after update are still the same.<br>
Until now, I have also tested to use the pytorch dataset to feed data into tensorflow model (a wired thing). After training, the evaluation result is good. So dataset is not the culprit.<br>
Besides, I use SGD with momentum =0.9 and weight decay factor=2e-4.<br>
I really don’t know what else hyperparameters I can set.</p>

username: Oli
cooked: <p>Are you putting the model in train() and val() mode?</p>
<p>You can check that the outputs are the same for the two models after one weight update.</p>

username: kkpytorch
cooked: <p>Yes, I set model.train() and model.eval().<br>
I also checked the outpus, they are the same on the same input.<br>
I also execute one step of the optimization method.<br>
After update, the first conv layer kernel keep the same.<br>
I did’t execute more steps.</p>

username: kkpytorch
cooked: <p>I have trained the TF model for more than fifty times.<br>
In all cases, the results are good enough.<br>
I also trained the pytorch model for nearly twenty times. All results are not so good.<br>
Until now, the only thing I have not checked is the optimization procedure.<br>
But I have checked the SGD code of pytorch, the weight decay is implemented sightly different with TF.<br>
In TF, the L2 loss is added into the total loss.<br>
In pytorch, the L2 loss is directly implemented in the grad manipulation.<br>
Theoretically, they are the same.<br>
But I don’t know whether these two implementation have difference for the sake of precision or approximation.</p>

username: Oli
cooked: <p>Oh yeah, I remember reading something about that. Couldn’t find a link though <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Perhaps try to train for a bit without weight decay to see if that actually is the error?</p>

username: kkpytorch
cooked: <p>Do you really read something about the difference between these two implementations?<br>
In TF, the operation is 1/2 * sum(weight * weight). The grad is the same.<br>
Since TF use static computation graph, the Jacobian is straight foward.<br>
I will try to train the model without weight decay.<br>
But in my model, the final evaluation result relies on the classification accuracy.<br>
If the weight decay implementation is the curprit, this is a bad thing. It means that I can’t use pytorch to train the model.</p>

username: Oli
cooked: <p>I don’t remember exactly what I read but I believe it stated that the pytorch way to use weight decay was different from the original paper. I don’t think it was in relation to tensorflow.</p>
<p>You probably know much more than me on this subject <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>blog post <a href="https://bbabenko.github.io/weight-decay/" rel="nofollow noopener">L2 != weight decay</a></p>

username: kkpytorch
cooked: <p>Thank you.<br>
I have read the blog.<br>
I also checked the SGD implementation in pytorch. The weight decay part is accumulated in the momentum term. This is the same as the L2 regularization implementation in TF.<br>
So, for momentum optimzation method in pytroch, weight decay is the same as L2 regularization.<br>
Personally, I think the only difference is that,  in pytorch the weight decay term in the grad is x. In TF, I have not find out the source code on l2_loss(), but basically I think according to the static graph, the grad on L2 loss equals to  0.5x + 0.5x,When the weight is small enough, 0.5x causes underflow and becomes to 0 in float32. But such small weight x barely affects the whole grad. So the optimization method may be not the culprit.</p>

username: yw_tt
cooked: <p>Hello, kk, do you find your problem’s resolution? I have met the same problem. Thanks for your reply.</p>