The model is producing random output. If I prompt just "Hi" the model generates a lot random output (infinite generation). Even if I set the temperature very low or turn off sampling it still does. Do you also face this issue, or how do you configure the generation_config? Do I need to configure somewhere to stop the sequence when eos is reached? Would be very thankful for any tips or generation_config templates. :-)

Example:
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_name_or_path = "casperhansen/llama-3-70b-instruct-awq"
model = AutoAWQForCausalLM.from_quantized(model_name_or_path, fuse_layers=True, trust_remote_code=False, safetensors=True)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=False)

prompt = "Hi"
tokens = tokenizer(
prompt,
return_tensors='pt'
).input_ids.cuda()

Generate output

generation_output = model.generate(
tokens,
do_sample=True,
temperature=0.1,
top_p=0.95,
top_k=40,
max_new_tokens=512,
pad_token_id=tokenizer.eos_token_id
)
print("Output: ", tokenizer.decode(generation_output[0]))
print(f"Duration: {datetime.now()-start}")

Output:

Output: Hi

The `if` statement checks if the `age` variable is greater than or equal to 18. If it is, the code inside the `if` block is executed, and the program prints "You are an adult." If the `age` variable is less than 18, the code inside the `if` block is skipped, and the program prints "You are not an adult."

You can also use `else` clause to specify a block of code to execute when the condition is false:

if age >= 18 {
print("You are an adult.")
} else {
print("You are not an adult.")
}

This code is equivalent to the previous example, but it uses an `else` clause to specify a block of code to execute when the condition is false.

You can also use `if` statements with multiple conditions using logical operators:

if age >= 18 && age <= 65 {
print("You are an adult.")
} else {
print("You are not an adult.")
}

This code checks if the `age` variable is greater than or equal to 18 and less than or equal to 65. If both conditions are true, the code inside the `if` block is executed.

You can also use `switch` statements to check multiple conditions:

switch age {
case 18...65:
print("You are an adult.")
default:
print("You are not an adult.")
}

This code uses a `switch` statement to check the value of the `age` variable. If the value falls within the range of 18 to 65, the code inside the `case` block is executed. If the value does not fall within this range, the code inside the `default` block is executed.

Note that in Swift, `if` statements and `switch` statements are used to control the flow of execution based on conditions, whereas `for` loops and `while` loops are used to repeat a block of code multiple times.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Here is a more detailed explanation of the concepts:

**If Statements**

If statements are used to execute a block of code if a certain condition is true. The syntax for an if statement in Swift is:

if condition {
// code to execute if condition is true
}

The `condition` is a boolean expression that is evaluated to determine whether the code inside the if statement should be executed. If the condition is

Many thanks in advance for any advises! :-)