Save and load
Author: LeninGF
Pull request: no

Hello community, I am having the same problem described to save and load a fine tuned model using transformers and tensorflow. I have used save_pretrained, save_weights and model.save with save_format=tf. I have been able to load the model with from_pretrained but it loads no weights and when I perform evaluation performance is too low compared when training while the fine tuned model is in memory. You could check my code at GitHub  in LeninGF/clasificaion_robos_fge in model_train_huggingface.ipynb and evaluate Notebook.ipynb

Comments:
cc @Rocketknight1 @gante

Hi @LeninGF 👋 I had a look into your notebooks but they are very long, which makes it very hard to pin the problem. Would you be able to share a short notebook (as short as possible) where the problem can be reproduced? Thanks :)

Hi Huggingface/Transformers I will do it . The only problem is that I will
use other dataset different from the one I am working because if privacy
policy... Give some hours to upload it

On Mon, Aug 1, 2022, 4:18 PM Joao Gante ***@***.***> wrote:

> Hi @LeninGF <https://github.com/LeninGF> 👋 I had a look into your
> notebooks but they are very long, which makes it very hard to pin the
> problem. Would you be able to share a short notebook (as short as possible)
> where the problem can be reproduced? Thanks :)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/18214#issuecomment-1201732303>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AH7TWKIUSR6BG5KVBRGMBO3VXA5KPANCNFSM54DTBWPA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


Hi Joao Please you can check my training code here:
https://colab.research.google.com/gist/LeninGF/89234ab4ba45147d34b8e8657caff761/model_train_huggingface_gitnew.ipynb

I think that the problem should happen with any dataset used. I am using a
multi labelled dataset. For company reasons I am not yet able to share it.
Please let me know if you would need a sample of it to reproduce the
problem.

The following colab  shows how I am trying to train the model
https://colab.research.google.com/gist/LeninGF/89234ab4ba45147d34b8e8657caff761/model_train_huggingface_gitnew.ipynb

The following colab shows that I am trying to load the weights of the
trained model to test it again with the test set by using a new
colab notebook

https://colab.research.google.com/gist/LeninGF/08b2824b73692134ec27979a7e6011ea/testingsavedfthfmodel.ipynb

you can reach me at ***@***.*** too

The problem is as follows: it does not matter how I train the model. While
the notebook where it was trained is active, you can see that the
model.evaluate(test_dataset) achieves a satisfactory 0.8 accuracy (even
though there is some overfitting) However, once I saved the model and I try
to load it again, it does not work and you can see that repeating the
weights load, model compile and model evaluate gives me an accuracy off 0.08

thanks for your kind help. If It is not to bother you a lot I am trying to
replicate this problem using the tweet emotion dataset that
huggingface has, I can send you the gist if you agree. I have already
trained the model and I am about to test if the downloaded model will be
working



Atentamente,

Lenin Falconí Estrada


El lun, 1 ago 2022 a las 16:18, Joao Gante ***@***.***>)
escribió:

> Hi @LeninGF <https://github.com/LeninGF> 👋 I had a look into your
> notebooks but they are very long, which makes it very hard to pin the
> problem. Would you be able to share a short notebook (as short as possible)
> where the problem can be reproduced? Thanks :)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/18214#issuecomment-1201732303>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AH7TWKIUSR6BG5KVBRGMBO3VXA5KPANCNFSM54DTBWPA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.