Get different inference answers and short time usage when calling LLama the second time
Author: yixliu1
Pull request: no

So when I first time load in and generate result, the load in time and inference time is longer (about 70s), it gives me a much sensible result. But when I reexecute the same code, I get a different result that it's non-sensible at all. (for ex: I ask it to generate 10 Q&A, the first time it will gives me the Q&A pairs but the second time it won't). Also, the inference time is much shorter (about 3s).

My code is as following:
tokenizer = LlamaTokenizer.from_pretrained(model_path, maxlength=maxlength)
model = LlamaForCausalLM.from_pretrained(
    model_path,
    load_in_8bit=False,
    torch_dtype=torch.float16,
    device_map="auto",
)

result = []

input_ids = tokenizer(sent, return_tensors="pt").input_ids.to("cuda")
generated_ids = model.generate(
    input_ids,
    #max_length=maxlength,
    max_new_tokens=maxlength,
    do_sample=False,
    repetition_penalty=10.0,
    temperature=0.8,
    top_p=0.75,
    top_k=40
)
res = tokenizer.decode(generated_ids[0])
result.append({'convo': sent, 'response': res})

Comments:
cc @ArthurZucker @younesbelkada 

Hey @yixliu1, thanks for your report!

Could you share the details of your environment so that we may investigate? 

You can do so by pasting the result of `transformers-cli env` to this report.

You need specific hardware to run 8 bit models, and it's important to know the version of `transformers` you're using and the model checkpoint.

Thanks.

Hi LysandreJik,

Thanks for your quick reply! As I am using sagemaker training job, I can't using `transformer-cli env` to get the details of transformer. However, I look up the wheel building process and find the following message:
```
/opt/conda/bin/python3.8 -m pip install -r requirements.txt
Collecting git+https://github.com/huggingface/transformers (from -r requirements.txt (line 11))
Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-ywnmp5u2
Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers /tmp/pip-req-build-ywnmp5u2
Resolved https://github.com/huggingface/transformers to commit bd6205919aad4d3a2300a39a98a642f1cc3a5348
```
In requirements.txt, I list `git+https://github.com/huggingface/transformers`. The version should be `transformers==4.35.0.dev0`

But I also run it on sagemaker locally, this time the version is like:
```
- `transformers` version: 4.34.0
- Platform: Linux-5.10.186-179.751.amzn2.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

There are more details about the inference result I would like to give.
For the latter time inference instead of the first one, when I change the prompt a little bit, the result will be changed following as well. For ex: (as in the inference result, it typically first duplicate the whole instruction then answer my question) if my prompt is 'Please generate 10 Q&A', the inference will have this sentence; if my prompt is 'Please generate 10 Q&A [with some restrictions]', the inference will also output these restrictions as well. 


Hi there,

I think somehow my model has "memorization" ability? For example, in my assumption, when I regenerate inference using my loaded in model, I assume I should have a fully new result. Like if my instruction is "{some paper} hi there", it should gives me some feedback for "hi there". But in my second time inference, it will gives me something like "hi there, i have written q&a questions based on the information you have provided. please continue providing more information, and i will generate Q&A questions based on the information you provide". The model is not supposed to know I want to generate Q&A as in this time's loop, I havn't give it any information that I want to generate it. 

I hope I make it clearly and hope that can help!!



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.