How does chat GPT generate different responses for the same question that we give? Chat GPT generates words one at a time. Let's say that right now It has generated one two three four words and we want to to generate this fifth word We typically see chat GPT only generate one word on the surface But behind the scenes it actually has this little table in its brain Where it's a set of words along with the corresponding probability of occurrences We don't want to just select the top word every time because that's just not human And so we'll sample from these words depending on their probabilities in order to generate this word and this sampling Gives a sense of randomness which allows chat GPT to generate different responses even for the same input question that you might ask to it