Generating accurate output
Generating accurate output Foundation models sometimes generate output that is not factually accurate. If factual accuracy is important for your project, set yourself up for success by learning how and why these models might sometimes get facts wrong and how you can ground generated output in correct facts. Why foundation models get facts wrong Foundation models can get facts wrong for a few reasons: * Pre-training builds word associations, not facts * Pre-training data sets contain out-of-date facts * Pre-training data sets do not contain esoteric or domain-specific facts and jargon * Sampling decoding is more likely to stray from the facts Pre-training builds word associations, not facts During pre-training, a foundation model builds up a vocabulary of words ([tokens](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-tokens.html)) encountered in the pre-training data sets. Also during pre-training, statistical relationships between those words become encoded in the model weights. For example, "Mount Everest" often appears near "tallest mountain in the world" in many articles, books, speeches, and other common pre-training sources. As a result, a pre-trained model will probably correctly complete the prompt "The tallest mountain in the world is " with the output "Mount Everest." These word associations can make it seem that facts have been encoded into these models too. For very common knowledge and immutable facts, you might have good luck generating factually accurate output using pre-trained foundation models with simple prompts like the tallest-mountain example. However, it is a risky strategy to rely on only pre-trained word associations when using foundation models in applications where accuracy matters. Pre-training data sets contain out-of-date facts Collecting pre-training data sets and performing pre-training runs can take a significant amount of time, sometimes months. If a model was pre-trained on a data set from several years ago, the model vocabulary and word associations encoded in the model weights won't reflect current world events or newly popular themes. For this reason, if you submit the prompt "The most recent winner of the world cup of football (soccer) is " to a model pre-trained on information a few years old, the generated output will be out of date. Pre-training data sets do not contain esoteric or domain-specific facts and jargon Common foundation model pre-training data sets, such as [The Pile (Wikipedia)](https://en.wikipedia.org/wiki/The_Pile_%28dataset%29), contain hundreds of millions of documents. Given how famous Mount Everest is, it's reasonable to expect a foundation model to have encoded a relationship between "tallest mountain in the world" and "Mount Everest". However, if a phenomenon, person, or concept is mentioned in only a handful of articles, chances are slim that a foundation model would have any word associations about that topic encoded in its weights. Prompting a pre-trained model about information that was not in its pre-training data sets is unlikely to produce factually accurate generated output. Sampling decoding is more likely to stray from the facts Decoding is the process a model uses to choose the words (tokens) in the generated output: * Greedy decoding always selects the token with the highest probability * Sampling decoding selects tokens pseudo-randomly from a probability distribution Greedy decoding generates output that is more predictable and more repetitive. Sampling decoding is more random, which feels "creative". If, based on pre-training data sets, the most likely words to follow "The tallest mountain is " are "Mount Everest", then greedy decoding could reliably generate that factually correct output, whereas sampling decoding might sometimes generate the name of some other mountain or something that's not even a mountain. How to ground generated output in correct facts Rather than relying on only pre-trained word associations for factual accuracy, provide context in your prompt text. Use context in your prompt text to establish facts When you prompt a foundation model to generate output, the words (tokens) in the generated output are influenced by the words in the model vocabulary and the words in the prompt text. You can use your prompt text to boost factually accurate word associations. Example 1 Here's a prompt to cause a model to complete a sentence declaring your favorite color: My favorite color is Given that only you know what your favorite color is, there's no way the model could reliably generate the correct output. Instead, a color will be selected from colors mentioned in the model's pre-training data: * If greedy decoding is used, whichever color appears most frequently with statements about favorite colors in pre-training content will be selected. * If sampling decoding is used, a color will be selected randomly from colors mentioned most often as favorites in the pre-training content. Example 2 Here's a prompt that includes context to establish the facts: I recently painted my kitchen yellow, which is my favorite color. My favorite color is If you prompt a model with text that includes factually accurate context like this, then the output the model generates