Repository: vllm-project/vllm

Issue:
Code llama output gibberish code.
Test on this
[https://huggingface.co/docs/transformers/main/model_doc/code_llama](https://huggingface.co/docs/transformers/main/model_doc/code_llama)

Env:
1. model: `Code Llama 7b Instruct`
2. `vLLM` `main` branch 0.1.4 on [4b6f069](https://github.com/vllm-project/vllm/commit/4b6f069b6fbb4f2ef7d4c6a62140229be61c5dd3).
3. transformers: 4.33.0.dev0 (build from main branch on [960807f](https://github.com/huggingface/transformers/commit/960807f62e53676723ab8281019219864ef3db4d))
4. torch_dtype=torch.float16 (my gpu doesn't support bfloat16)

Generated text from `transformers`:
```Python
def remove_non_ascii(s: str) -> str:
    """ Remove non-ASCII characters from a string.

    Args:
        s (str): The string to remove non-ASCII characters from.

    Returns:
        str: The string with non-ASCII characters removed.
    """
    result = ""
    for c in s:
        if ord(c) < 128:
            result += c
    return result


def remove_non_ascii_and_spaces(s: str) -> str:
    """ Remove non-ASCII and space characters from a string.


    return result
```


Generated text from `vLLM`:
```python
<- Prompt ->:
def remove_non_ascii(s: str) -> str:
    """ <FILL_ME>
    return result

<- Answer ->:

   For comments

    """
        Remove non ascii inplace of special characters from a given string.""""
        non ascii => blank space, punctuation""""""codepointpython| nap string""" in Python"""test_safe
    u""
    
    among'".
   328
    <PDFutable       # tring control chars's to be "-"a"

    value.
// Dasailss
    ASCII characters so:""""
   
    X" 
    :5
	 Non-""
    <
```

Engine Log:
```
Received request cmpl-31a97cd2e50f4c18a9ac9e5fc4df4920: prompt: 'def remove_non_ascii(s: str) -> str:\n    """ <FILL_ME>\n    return result\n', sampling params: SamplingParams(n=1, best_of=1, presence_penalty=0.0, frequency_penalty=0.0, temperature=1.0, top_p=1.0, top_k=-1, use_beam_search=False, stop=[], ignore_eos=False, max_tokens=128, logprobs=None), prompt token ids: [1, 822, 3349, 29918, 5464, 29918, 294, 18869, 29898, 29879, 29901, 851, 29897, 1599, 851, 29901, 13, 1678, 9995, 529, 3738, 2208, 29918, 2303, 29958, 13, 1678, 736, 1121, 13].
```


Discussion:

Resolving patch:
diff --git a/vllm/model_executor/layers/attention.py b/vllm/model_executor/layers/attention.py
--- a/vllm/model_executor/layers/attention.py
+++ b/vllm/model_executor/layers/attention.py
@@ -264,10 +264,10 @@ def __init__(
         self.is_neox_style = is_neox_style
 
         # Create the cos and sin cache.
-        inv_freq = 1.0 / (base**(
-            torch.arange(0, rotary_dim, 2, device="cuda") / rotary_dim))
-        t = torch.arange(max_position, device="cuda").float()
-        freqs = torch.einsum("i,j -> ij", t, inv_freq.float())
+        inv_freq = 1.0 / (base**(torch.arange(
+            0, rotary_dim, 2, dtype=torch.float, device="cuda") / rotary_dim))
+        t = torch.arange(max_position, dtype=torch.float, device="cuda")
+        freqs = torch.einsum("i,j -> ij", t, inv_freq)
         cos = freqs.cos()
         sin = freqs.sin()
         cache = torch.cat((cos, sin), dim=-1)


Test patch:
diff --git a/tests/kernels/test_pos_encoding.py b/tests/kernels/test_pos_encoding.py
--- a/tests/kernels/test_pos_encoding.py
+++ b/tests/kernels/test_pos_encoding.py
@@ -133,9 +133,10 @@ def test_rotary_embedding(
                       device="cuda")
 
     # Create the rotary embedding.
-    inv_freq = 1.0 / (base**(torch.arange(0, rotary_dim, 2) / rotary_dim))
+    inv_freq = 1.0 / (base**(
+        torch.arange(0, rotary_dim, 2, dtype=torch.float) / rotary_dim))
     t = torch.arange(max_position).float()
-    freqs = torch.einsum("i,j -> ij", t, inv_freq.float())
+    freqs = torch.einsum("i,j -> ij", t, inv_freq)
     cos = freqs.cos()
     sin = freqs.sin()
     cos_sin_cache = torch.cat((cos, sin), dim=-1)