Question: How do LLMs learn to be "Generative", as we often describe them?
Author: metalwhale
Pull request: no

(Please forgive me and let me know if I'm not allowed to ask this kind of question here. I'm so sorry if I'm bothering everyone.)

AFAIK to be called "generative", a model should have the ability to learn the joint probability over the training data. In the case of LLMs, we apply the chain rule of Bayes' formula to achieve this by leveraging the autoregressive method for every token of each input text sequence. For example, with a text sequence of 4 tokens, it can be written as:
```
p(x4,x3,x2,x1) = p(x4|x3,x2,x1) * p(x3|x2,x1) * p(x2|x1) * p(x1)
```
where `x1` denotes the 1st token, `x2` denotes the 2nd token and so on, respectively.

I understand the conditional terms `p(x_n|...)` where we use cross-entropy to calculate their losses. However, I'm unsure about the probability of the very first token `p(x1)`. How is it calculated? Is it in some configurations of the training process, or in the model architecture, or in the loss function?

IMHO, if the model doesn't learn `p(x1)` properly, the entire formula for Bayes' rule cannot be completed, and we can't refer to LLMs as "truly generative". Am I missing something here?

I asked the [same question on `nanoGPT` repo](https://github.com/karpathy/nanoGPT/issues/432) and [on HN](https://news.ycombinator.com/item?id=39249301). I'm also reading Transformer codes from this repo, but I haven't found the answer I'm looking for yet. Could someone please enlighten me? Thank in advance!

Comments:
When you run a forward pass, you are always predicting the distribution of the next token. So quite simply, model(input_1) with let's say [3442] will compute the distribution of `x1`. 

@ArthurZucker 
Thank you for your answer!

But in that case `x1` is no longer the "first token", isn't it? The probability of `x1` now is `p(x1|input_1)`, not `p(x1)` IMHO. Am I correct?

What I want to know is how we compute the probability of the "very first token", a.k.a the token that doesn't have any token before it.

you can't compute the proba of the first token, it's given by the user. Since it s a constant if you apply your rule, `p(input_1) = 1 `

In a way, you can say the `p(x1)` is always `1` since `x1 = the input given by the user` it is certain

@ArthurZucker 
Your answer makes everything very clear to me.

My understanding of a generative model is that it learns the probability of every "piece" of the dataset. In the case of autoregressive progress for LLMs training, they learn about the relation of all the tokens, including the first tokens of each text sequence.

However I guess that's not entirely true: the "generative aspect" of LLMs is true only in particular cases if we provide them with the "initial context" (the first tokens, or the "prompt"). Otherwise they are not generative at all. They are "conditional generative", not generative over the entire dataset. Am I right?

If I understand what you are saying correctly, we can have a "truly generative" LLMs by setting a special token with a probability of 1, that always appears as the first token of every text sequence, something like BOS token. In other words, LLMs trained with the BOS token are "more generative" than ones that don't have it. What do you think?

I’m not sure. But yes, all decoders are trained with the BOS which serves exactly that purpose: the one input that is always of constant proba. I’ll close this issue but feel free to discuss 🙂

@ArthurZucker Thank you so much for your help!