# Foundation model parameters: decoding and stopping criteria #

You can specify parameters to control how the model generates output in response to your prompt\. This topic lists parameters that you can control in the Prompt Lab\.

## Decoding ##

*Decoding* is the process a model uses to choose the tokens in the generated output\.

*Greedy decoding* selects the token with the highest probability at each step of the decoding process\. Greedy decoding produces output that closely matches the most common language in the model's pretraining data and in your prompt text, which is desirable in less creative or fact\-based use cases\. A weakness of greedy decoding is that it can cause repetitive loops in the generated output\.

*Sampling decoding* is more variable, more random than greedy decoding\. Variability and randomness is desirable in creative use cases\. However, with greater variability comes the risk of nonsensical output\. Sampling decoding selects tokens from a probability distribution at each step:

<!-- <ul> -->

 *  *Temperature sampling* refers to selecting a high\- or low\-probability next token\.
 *  *Top\-k sampling* refers to selecting the next token randomly from a specified number, k, of tokens with the highest probabilities\.
 *  *Top\-p sampling* refers to selecting the next token randomly from the smallest set of tokens for which the cumulative probability exceeds a specified value, p\. (Top\-p sampling is also called *nucleus sampling*\.)

<!-- </ul> -->

You can specify values for both Top K and Top P\. When both parameters are used, Top K is applied first\. When Top P is computed, any tokens below the cutoff set by Top K are considered to have a probability of zero\.

<!-- <table> -->

Table 1\. Supported values, defaults, and usage notes for sampling decoding

| Parameter       | Supported values                                                                                | Default | Use                                       |
| --------------- | ----------------------------------------------------------------------------------------------- | ------- | ----------------------------------------- |
| **Temperature** | Floating\-point number in the range 0\.0 (same as greedy decoding) to 2\.0 (maximum creativity) | 0\.7    | Higher values lead to greater variability |
| **Top K**       | Integer in the range 1 to 100                                                                   | 50      | Higher values lead to greater variability |
| **Top P**       | Floating\-point number in the range 0\.0 to 1\.0                                                | 1\.0    | Higher values lead to greater variability |

<!-- </table ""> -->

### Random seed ###

When you submit the same prompt to a model multiple times with sampling decoding, you'll usually get back different generated text each time\. This variability is the result of intentional pseudo\-randomness built into the decoding process\. *Random seed* refers to the number used to generate that pseudo\-random behavior\.

<!-- <ul> -->

 *  **Supported values:** Integer in the range 1 to 4 294 967 295
 *  **Default:** Generated based on the current server system time
 *  **Use:** To produce repeatable results, set the same random seed value every time\.

<!-- </ul> -->

### Repetition penalty ###

If you notice the result generated for your chosen prompt, model, and parameters consistently contains repetitive text, you can try adding a *repetition penalty*\.

<!-- <ul> -->

 *  **Supported values:** Floating\-point number in the range 1\.0 (no penalty) to 2\.0 (maximum penalty)
 *  **Default:** 1\.0
 *  **Use:** The higher the penalty, the less likely it is that the result will include repeated text\.

<!-- </ul> -->

## Stopping criteria ##

You can affect the length of the output generated by the model in two ways: specifying stop sequences and setting Min tokens and Max tokens\. Text generation stops after the model considers the output to be complete, a stop sequence is generated, or the maximum token limit is reached\.

### Stop sequences ###

A *stop sequence* is a string of one or more characters\. If you specify stop sequences, the model will automatically stop generating output after one of the stop sequences that you specify appears in the generated output\. For example, one way to cause a model to stop generating output after just one sentence is to specify a period as a stop sequence\. That way, after the model generates the first sentence and ends it with a period, output generation stops\. Choosing effective stop sequences depends on your use case and the nature of the generated output you expect\.

**Supported values:** 0 to 6 strings, each no longer than 40 tokens

**Default:** No stop sequence

**Use:**

<!-- <ul> -->

 *  Stop sequences are ignored until after the number of tokens that are specified in the Min tokens parameter are generated\.
 *  If your prompt includes examples of input\-output pairs, ensure the sample output in the examples ends with one of the stop sequences\.

<!-- </ul> -->

### Minimum and maximum new tokens ###

If you're finding the output from the model is too short or too long, try adjusting the parameters that control the number of generated tokens:

<!-- <ul> -->

 *  The *Min tokens* parameter controls the minimum number of tokens in the generated output
 *  The *Max tokens* parameter controls the maximum number of tokens in the generated output

<!-- </ul> -->

The maximum number of tokens that are allowed in the output differs by model\. For more information, see the *Maximum tokens* information in [Supported foundation models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html)\.

**Defaults:**

<!-- <ul> -->

 *  Min tokens: 0
 *  Max tokens: 20

<!-- </ul> -->

**Use:**

<!-- <ul> -->

 *  Min tokens must be less than or equal to Max tokens\.
 *  Because the cost of using foundation models in IBM watsonx\.ai is based on use, which is partly related to the number of tokens that are generated, specifying the lowest value for Max tokens that works for your use case is a cost\-saving strategy\.
 *  For Lite plans, output stops being generated after a dynamic, model\-specific, environment\-driven upper limit is reached, even if the value specified with the Max tokens parameter is not reached\. To determine the upper limit, see the *Tokens limits* section for the model in [Supported foundation models](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html) or call the [`get_details`](https://ibm.github.io/watson-machine-learning-sdk/foundation_models.html#ibm_watson_machine_learning.foundation_models.Model.get_details) function of the foundation models Python library\.

<!-- </ul> -->

**Parent topic:**[Prompt Lab](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-prompt-lab.html)

<!-- </article "role="article" "> -->