backwards compatibility with older versions
Author: MasoudMaani
Pull request: no

I have seen a frequent issue with the newer updates, I have been using llama 3 models for a long time (since they got uploaded first), and with newer updates of transformers, I have seen degradation of performance across the board with llama 3 models.
eventho that shouldn't be the case, but I think updates in functions in other parts of the generation loop has affected the generation loop for llama 3 too, and model now greatly hallucinates with newer versions.
I have been investigating my own code vigorously for the past couple of months, and I didn't open this issue lightly, it's been after through investigation and back and forth between older and newer versions of the transformers library that I am opening this issue.
note that models work great with older versions (specifically up to 4.47.1) but with newer versions (everything after 4.50) there are errors and these errors seem to grow larger with every update, the model is essentially fully hallucination and unusable with version 5.
this is a huge problem with backward compatibility, at this rate, any model that works today will be also unusable a year from now. which is a huge red flag for compatibility and overall research. this means all benchmarks are out of wack since older models are just working with worse code. and many other issues.
I really hope this is just on my side but I have been investigating this for months, I thought the only logical move atp is to open this issue.

Comments:
hey @MasoudMaani ,

It is true that we have been introducing a lot of changes recently which might have affected llama. Though I haven't noticed huge differences with simple generation cases, could you share a reproducer and which versions specifically it worked correctly with? I will check out on my end 

@zucchini-nlp 
absolutely. from version 4.45 to 4.47.1 everything works perfectly smooth, from 4.48 to 4.53 things got degraded by a little, on 4.55 things got really shaky, and from there to the latest version things have been super shaky.
I use my own math which is really sensitive to changes and may break the model if math is corrupted elsewhere, I can confirm that simple generation cases don't show it outright but I can clearly see the generation on my side degrading because I use lots of operations on the math.

again, you might think it's my math that's the issue, and that's exactly why I have been holding back for more than 6 months, rewritten things from scratch a few times, only to confirm it's not my error and corruption comes from upstream processes because when downgrading transformers it works fine. I have also tested bnb and torch versions and for a moment bnb had a problem too (0.48 which I reported on the day and got resolved on 0.48.1).

there are two ways out of this, either group and isolate everything by model and related version which pretty much fixes everything, or rewrite everything for backward compatibility of the latest versions with older models. and this needs to be done at least at F32 to ensure math isn't corrupted, you can't simply see it in generation unless you spend 6 months debugging on the same model like I did.

I am not sure what you mean by "math is corrupted", please share a reproducer so I can investigate. It can be a simple model forward or similar showing that the final logits in older versions are completely different from newer versions

I manipulate generation loop's math heavily and in multiple points of the loop, and that code I can't share. you can instead ensure of the difference by checking the logit values in older and newer versions.
my duty to the community was reporting an error I see everyone else is totally absent minded about, not sharing my work.