Model only returns `<unk>` tokens in response to text generation prompt
#2
by
slotreck
- opened
I'm loading and deploying my model with the following code:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
checkpoint = 'ausboss/llama-30b-supercot'
model = AutoModelForCausalLM.from_pretrained(checkpoint,
torch_dtype=torch.float16, device_map='auto', load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
prompt = <see below>
inputs = tokenizer(prompt, return_tensors='pt')
inputs = inputs.to(0)
output = model.generate(inputs['input_ids'], max_new_tokens=500)
response = tokenizer.decode(output[0].tolist())
Using a prompt that follows the format in the model code:
prompt = """
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Extract the biological relations from the following text as (Subject, Predicate, Object) triples in the format ("Subject", "predicate", "Object"):
### Input:
Salmeterol is a long-acting beta2-adrenergic receptor (beta 2AR) agonist used clinically to treat asthma.
### Response:
"""
EDIT: Corrected the number of newlines in my prompt
However, the model response only ever contains unknown tokens:
<s>
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Extract the biological relations from the following text as (Subject, Predicate, Object) triples in the format
("Subject", "predicate", "Object"):
### Input:
Salmeterol is a long-acting beta2-adrenergic receptor (beta 2AR) agonist used clinically to treat asthma.
### Response:
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>
What's going on here, and how can I fix it?
Its working properly but still no proper answers
<s> Plan a trip to Mumbai for 5 days.
Mumbai is the capital city of the Indian state of Maharashtra. As of 2011 it is the most populous city in India and the ninth most populous agglomeration in the world, with an estimated city population of 12.4 million. Along with the neighbouring regions of the Mumbai Metropolitan Region, it is the second most populous metropolitan area in India, with a population of 21 million. Mumbai lies on the west coast of India and has a deep natural harbour. In 2008, Mumbai was named an alpha world city. It is also the wealthiest city in India, and has the highest GDP of any city in South, West, or Central Asia. Mumbai has the highest number of billionaires and millionaires among all cities in India. The seven islands that came to constitute Mumbai were home to fishing colonies. In the third century BCE, the islands formed part of the Maurya Empire, and later they were ruled by the Satavahanas of Kolhapur (2nd century BCE). The kings of Silhara dynasty, under whom the islands were developed as a port, called the region "Gotham" and "Gotham Nagari". The Greek historian Ptolemy (AD 90–168) referred to the islands as "Nosala" and "Nosala". In the 3rd century CE, the islands were ruled by the Abhira dynasty. The region was later controlled by the Gupta Empire in the 4th century CE. From the 8th century to the 14th century, the islands came under the control of several indigenous dynasties: the Rashtrakuta dynasty; the Chalukya dynasty; the Silhara dynasty and the Yadava dynasty of Devagiri. After 1343, the islands were ruled by the Muslim rulers of Gujarat Sultanate. On 23 January 1534, Bahadur Shah, the Sultan of Gujarat, ceded the islands to the Portuguese Empire as per the Treaty of Bassein. The islands obtained the name Mumbai, from the name of the local Goddess Mumba Devi. The city
Same your code ! 2 GPU RTX 4090 on runpod
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
checkpoint = 'ausboss/llama-30b-supercot'
model = AutoModelForCausalLM.from_pretrained(checkpoint,
torch_dtype=torch.float16, device_map='auto', load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
prompt = "Plan a trip to Mumbai for 5 days"
inputs = tokenizer(prompt, return_tensors='pt')
inputs = inputs.to(0)
output = model.generate(inputs['input_ids'], max_new_tokens=500)
response = tokenizer.decode(output[0].tolist())
print(response)
Cheers !
Sorry i closed that instance don’t remember properly, it was ARM. Like you can try on runpod its cheap!