Repository: huggingface/text-generation-inference

Issue:
TGI server and python client's `generate`/`generate_stream` endpoints/methods should support `presence_penalty`/`frequency_penalty`
### System Info

2024-03-18T19:02:01.628354Z  INFO text_generation_launcher: Runtime environment:
Target: x86_64-unknown-linux-gnu
Cargo version: 1.75.0
Commit sha: 9c1cb81cd8fc01c8f736c554f316ba42b0695717
Docker label: sha-9c1cb81
nvidia-smi:
Mon Mar 18 19:02:01 2024       
   +-----------------------------------------------------------------------------+
   | NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.1     |
   |-------------------------------+----------------------+----------------------+
   | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
   | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
   |                               |                      |               MIG M. |
   |===============================+======================+======================|
   |   0  NVIDIA A100-SXM...  On   | 00000000:8A:00.0 Off |                    0 |
   | N/A   29C    P0    74W / 400W |  76893MiB / 81920MiB |      0%      Default |
   |                               |                      |             Disabled |
   +-------------------------------+----------------------+----------------------+
                                                                                  
   +-----------------------------------------------------------------------------+
   | Processes:                                                                  |
   |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
   |        ID   ID                                                   Usage      |
   |=============================================================================|


TGI version: 1.4.2

### Information

- [ ] Docker
- [ ] The CLI directly

### Tasks

- [X] An officially supported command
- [ ] My own modifications

### Reproduction

The TGI python client's `generate` and `generate_stream` methods don't have support for `presence_penalty` or `frequency_penalty`. The TGI server /generate and /generate_stream endpoints support `frequency_penalty`, but not `presence_penalty`.

```
from text_generation import Client as TGIClient
client = TGIClient(service_url)


client.generate(...)

client.generate_stream(...)
```
IDE shows no support for `presence_penalty` or `frequency_penalty`:
![image](https://github.com/huggingface/text-generation-inference/assets/119249469/fa3b396a-80ea-41cd-b038-32337d42e634)

Running the TGI server and accessing the docs shows the API endpoint only supports `frequency_penalty`, but not `presence_penalty`:
<img width="424" alt="image" src="https://github.com/huggingface/text-generation-inference/assets/119249469/369dd0c8-83a1-40dd-a928-73c66e6ec178">


### Expected behavior

I would expect:

- The python client's `generate` and generate_stream` methods to support passing in `frequency_penalty` and `presence_penalty`
- The TGI server's `/generate` and `/generate_stream`endpoints to support `presence_penalty`


Discussion:

Resolving patch:
diff --git a/clients/python/text_generation/client.py b/clients/python/text_generation/client.py
--- a/clients/python/text_generation/client.py
+++ b/clients/python/text_generation/client.py
@@ -67,6 +67,7 @@ def __init__(
     def chat(
         self,
         messages: List[Message],
+        repetition_penalty: Optional[float] = None,
         frequency_penalty: Optional[float] = None,
         logit_bias: Optional[List[float]] = None,
         logprobs: Optional[bool] = None,
@@ -87,9 +88,13 @@ def chat(
         Args:
             messages (`List[Message]`):
                 List of messages
-            frequency_penalty (`float`):
-                The parameter for frequency penalty. 0.0 means no penalty. See [this
+            repetition_penalty (`float`):
+                The parameter for repetition penalty. 0.0 means no penalty. See [this
                 paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
+            frequency_penalty (`float`):
+                The parameter for frequency penalty. 0.0 means no penalty
+                Penalize new tokens based on their existing frequency in the text so far,
+                decreasing the model's likelihood to repeat the same line verbatim.
             logit_bias (`List[float]`):
                 Adjust the likelihood of specified tokens
             logprobs (`bool`):
@@ -121,6 +126,7 @@ def chat(
         request = ChatRequest(
             model="tgi",
             messages=messages,
+            repetition_penalty=repetition_penalty,
             frequency_penalty=frequency_penalty,
             logit_bias=logit_bias,
             logprobs=logprobs,
@@ -179,6 +185,7 @@ def generate(
         max_new_tokens: int = 20,
         best_of: Optional[int] = None,
         repetition_penalty: Optional[float] = None,
+        frequency_penalty: Optional[float] = None,
         return_full_text: bool = False,
         seed: Optional[int] = None,
         stop_sequences: Optional[List[str]] = None,
@@ -207,6 +214,10 @@ def generate(
             repetition_penalty (`float`):
                 The parameter for repetition penalty. 1.0 means no penalty. See [this
                 paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
+            frequency_penalty (`float`):
+                The parameter for frequency penalty. 1.0 means no penalty
+                Penalize new tokens based on their existing frequency in the text so far,
+                decreasing the model's likelihood to repeat the same line verbatim.
             return_full_text (`bool`):
                 Whether to prepend the prompt to the generated text
             seed (`int`):
@@ -245,6 +256,7 @@ def generate(
             do_sample=do_sample,
             max_new_tokens=max_new_tokens,
             repetition_penalty=repetition_penalty,
+            frequency_penalty=frequency_penalty,
             return_full_text=return_full_text,
             seed=seed,
             stop=stop_sequences if stop_sequences is not None else [],
@@ -278,6 +290,7 @@ def generate_stream(
         do_sample: bool = False,
         max_new_tokens: int = 20,
         repetition_penalty: Optional[float] = None,
+        frequency_penalty: Optional[float] = None,
         return_full_text: bool = False,
         seed: Optional[int] = None,
         stop_sequences: Optional[List[str]] = None,
@@ -303,6 +316,10 @@ def generate_stream(
             repetition_penalty (`float`):
                 The parameter for repetition penalty. 1.0 means no penalty. See [this
                 paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
+            frequency_penalty (`float`):
+                The parameter for frequency penalty. 1.0 means no penalty
+                Penalize new tokens based on their existing frequency in the text so far,
+                decreasing the model's likelihood to repeat the same line verbatim.
             return_full_text (`bool`):
                 Whether to prepend the prompt to the generated text
             seed (`int`):
@@ -340,6 +357,7 @@ def generate_stream(
             do_sample=do_sample,
             max_new_tokens=max_new_tokens,
             repetition_penalty=repetition_penalty,
+            frequency_penalty=frequency_penalty,
             return_full_text=return_full_text,
             seed=seed,
             stop=stop_sequences if stop_sequences is not None else [],
@@ -435,6 +453,7 @@ def __init__(
     async def chat(
         self,
         messages: List[Message],
+        repetition_penalty: Optional[float] = None,
         frequency_penalty: Optional[float] = None,
         logit_bias: Optional[List[float]] = None,
         logprobs: Optional[bool] = None,
@@ -455,9 +474,13 @@ async def chat(
         Args:
             messages (`List[Message]`):
                 List of messages
-            frequency_penalty (`float`):
+            repetition_penalty (`float`):
                 The parameter for frequency penalty. 0.0 means no penalty. See [this
                 paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
+            frequency_penalty (`float`):
+                The parameter for frequency penalty. 0.0 means no penalty
+                Penalize new tokens based on their existing frequency in the text so far,
+                decreasing the model's likelihood to repeat the same line verbatim.
             logit_bias (`List[float]`):
                 Adjust the likelihood of specified tokens
             logprobs (`bool`):
@@ -489,6 +512,7 @@ async def chat(
         request = ChatRequest(
             model="tgi",
             messages=messages,
+            repetition_penalty=repetition_penalty,
             frequency_penalty=frequency_penalty,
             logit_bias=logit_bias,
             logprobs=logprobs,
@@ -546,6 +570,7 @@ async def generate(
         max_new_tokens: int = 20,
         best_of: Optional[int] = None,
         repetition_penalty: Optional[float] = None,
+        frequency_penalty: Optional[float] = None,
         return_full_text: bool = False,
         seed: Optional[int] = None,
         stop_sequences: Optional[List[str]] = None,
@@ -574,6 +599,10 @@ async def generate(
             repetition_penalty (`float`):
                 The parameter for repetition penalty. 1.0 means no penalty. See [this
                 paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
+            frequency_penalty (`float`):
+                The parameter for frequency penalty. 1.0 means no penalty
+                Penalize new tokens based on their existing frequency in the text so far,
+                decreasing the model's likelihood to repeat the same line verbatim.
             return_full_text (`bool`):
                 Whether to prepend the prompt to the generated text
             seed (`int`):
@@ -614,6 +643,7 @@ async def generate(
             do_sample=do_sample,
             max_new_tokens=max_new_tokens,
             repetition_penalty=repetition_penalty,
+            frequency_penalty=frequency_penalty,
             return_full_text=return_full_text,
             seed=seed,
             stop=stop_sequences if stop_sequences is not None else [],
@@ -644,6 +674,7 @@ async def generate_stream(
         do_sample: bool = False,
         max_new_tokens: int = 20,
         repetition_penalty: Optional[float] = None,
+        frequency_penalty: Optional[float] = None,
         return_full_text: bool = False,
         seed: Optional[int] = None,
         stop_sequences: Optional[List[str]] = None,
@@ -669,6 +700,10 @@ async def generate_stream(
             repetition_penalty (`float`):
                 The parameter for repetition penalty. 1.0 means no penalty. See [this
                 paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
+            frequency_penalty (`float`):
+                The parameter for frequency penalty. 1.0 means no penalty
+                Penalize new tokens based on their existing frequency in the text so far,
+                decreasing the model's likelihood to repeat the same line verbatim.
             return_full_text (`bool`):
                 Whether to prepend the prompt to the generated text
             seed (`int`):
@@ -706,6 +741,7 @@ async def generate_stream(
             do_sample=do_sample,
             max_new_tokens=max_new_tokens,
             repetition_penalty=repetition_penalty,
+            frequency_penalty=frequency_penalty,
             return_full_text=return_full_text,
             seed=seed,
             stop=stop_sequences if stop_sequences is not None else [],
diff --git a/clients/python/text_generation/types.py b/clients/python/text_generation/types.py
--- a/clients/python/text_generation/types.py
+++ b/clients/python/text_generation/types.py
@@ -109,7 +109,12 @@ class ChatRequest(BaseModel):
     model: str
     # List of messages in the conversation
     messages: List[Message]
-    # Penalty for frequency of new tokens
+    # The parameter for repetition penalty. 1.0 means no penalty.
+    # See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
+    repetition_penalty: Optional[float] = None
+    # The parameter for frequency penalty. 1.0 means no penalty
+    # Penalize new tokens based on their existing frequency in the text so far,
+    # decreasing the model's likelihood to repeat the same line verbatim.
     frequency_penalty: Optional[float] = None
     # Bias values for token selection
     logit_bias: Optional[List[float]] = None
@@ -145,6 +150,10 @@ class Parameters(BaseModel):
     # The parameter for repetition penalty. 1.0 means no penalty.
     # See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details.
     repetition_penalty: Optional[float] = None
+    # The parameter for frequency penalty. 1.0 means no penalty
+    # Penalize new tokens based on their existing frequency in the text so far,
+    # decreasing the model's likelihood to repeat the same line verbatim.
+    frequency_penalty: Optional[float] = None
     # Whether to prepend the prompt to the generated text
     return_full_text: bool = False
     # Stop generating tokens if a member of `stop_sequences` is generated
@@ -201,6 +210,12 @@ def valid_repetition_penalty(cls, v):
             raise ValidationError("`repetition_penalty` must be strictly positive")
         return v
 
+    @field_validator("frequency_penalty")
+    def valid_frequency_penalty(cls, v):
+        if v is not None and v <= 0:
+            raise ValidationError("`frequency_penalty` must be strictly positive")
+        return v
+
     @field_validator("seed")
     def valid_seed(cls, v):
         if v is not None and v < 0:


Test patch: