Truncating GPT2 past
Author: LHolten
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I am using code like this to generate text:
```python
    while True:
        output_token, past = model.forward(output_token, past=past)
        output_token = output_token[:, -1, :]
        output_token = torch.multinomial(F.softmax(output_token, dim=-1), num_samples=1)
        out = torch.cat((out, output_token), dim=1)
```
The problem with this is that `past` keeps growing.
My solution is to check the size of past and truncate it like this:
```python
        if past[0].shape[-2] > max_past:
            past = [p[..., -max_past:, :] for p in past]
```
I don't think this is correct, can anyone enlighten me?

Comments:
Hi! Your usage of `past` seems correct to me. After how many iterations do you feel the need to truncate?

I also would like to have an example that works correctly with GPT2. If past is not truncated, the model crashes after it reaches some threshold, which is, I suppose, `model.config.max_position_embeddings`. So, after it grows to this size, I truncate it as follows:
```
            if past[0].shape[3] == model.config.max_position_embeddings - 1:
                past = [p[:, :, :, :-1, ...] for p in past]
```

This is clearly broken, as the model's generation capabilities degrade dramatically after the truncation kicks in. Example (gpt2-medium):

>...And here's why... for every person of difference, there's a different template when it comes to talking. We've seen them together for so long that most people don't know who we are. The owner of the room has probably hidden that while we are awake, but since then, there's usually a perception gap to gape at the outside. He's an enemy of rule, we know this. He was excommunicated and let die and our purity is worth more in terms of glory than doing battle together. He probably ran away from us, we're not sure why but we do remember his location. What kind of story is this then? In whatever and wherever he was taken, the hapless thief of light known as Arcadia forced down with the blessing of the goddess Kali from the eternities, has been returned to each of us... in as faceless a form as it's possible to<**TRUNCATION BEGINS HERE**> us we were possibly possible to informally possible to manipulateable to us. NoTa possible to some strange not always been my memories allow, unhidden possible to beholdenf the parts are, only known. Upon the wanderer. This is able to all asked and thus possible to callable to us — being made for the willed possible for them that of receiving the righteous deed has ever been in our power was when we can look of whether it needs permitted to appear plausible to those we may befitting to us you and with you can take.

You can see that it starts producing largely incoherent sentences with bad grammar. It also loses basic abilities like matching brackets and quote marks. If I truncate the first element instead, as @LHolten does, the result is even worse:
>...And here's why... for every person of difference, there's a different template when it comes to talking. We've seen them together for so long that most people don't know who we are. The owner of the room has probably hidden that while we are awake, but since then, there's usually a perception gap to gape at the outside. He's an enemy of rule, we know this. He was excommunicated and let die and our purity is worth more in terms of glory than doing battle together. He probably ran away from us, we're not sure why but we do remember his location. What kind of story is this then? In whatever and wherever he was taken, the hapless thief of light known as Arcadia forced down with the blessing of the goddess Kali from the eternities, has been returned to each of us... in as faceless a form as it's possible to<**TRUNCATION BEGINS HERE**> to to to To Aud To Bed Since I January Nine Thou William July you very well, for this very purpose you can actually do without wearing/washing clothes any specific garments/tutexes, if they are Halloween- Halloween No-How-able Spells for Specific Results Splinterview February Treat him as Jess four of The H: We really dislike why's of Tactics The Neutral Generic Real Sheriff's the equivalent as Uthville He has been Henry's Gender Surprise Our Half


I'm afraid the problem is more complex than it initially seemed to me. Maybe losing a single element of history is too damaging somehow? Perhaps the only correct way to deal with "past overflow" is to truncate the context itself, say by removing the first paragraph from it, and then regenerate the past from it?
Generally, what is the best way of doing continuous generation without 
 tokens?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.