GENIE Model Output Returns Narrative Instead of Structured JSON
I've been trying to use the GENIE (THUMedInfo/GENIE_zh_7b) model to extract structured information from electronic health records (EHRs) as described in the model card. The expected output should be a JSON format with biomedical named entities and their attributes. However, when running the provided code snippet, I'm consistently getting a narrative response rather than the expected structured output.
A narrative response such as:
"Human：慢性乙型肝炎病史10余年，曾有肝功能异常，中医治疗后好转；1年余前查HBsAg转阴，但肝脏病理提示病毒性肝炎伴肝纤维化（G1s3-4)
Assistant：根据您提供的信息，患者是一位有10余年慢性乙型肝炎病史的患者，曾经出现过肝功能异常的情况。经过中医治疗后，患者的HBSAg转阴，但肝脏病理检查提示存在病毒性肝炎伴肝纤维化(G1S3-4)
针对这种情况，建议患者继续接受抗病毒治疗，以控制病毒复制和减轻肝脏炎症。同时，患者需要定期进行肝功能检查和肝脏病理检查，以监测病情的变化。此外，患者还需要注意饮食和生活习惯，避免饮酒和吸烟等不良习惯，以减轻肝脏负担。"
I have used the code provided in the model card, I wonder if there's an issue with the prompt template, sampling parameters, or if additional steps/configurations are needed to ensure the model outputs structured JSON rather than narrative text. Any insights or suggestions would be greatly appreciated!
Hi,
I have tried your example: with
prompt = ['Human:\n慢性乙型肝炎病史10余年，曾有肝功能异常，中医治疗后好转；1年余前查HBsAg转阴，但肝脏病理提示病毒性肝炎伴肝纤维化（G1s3-4)\nAssistant:\n']
the output is [{'术语': '慢性乙型肝炎', '语义类型': '疾病、综合征、病理功能', '叙述状态': '存在', '身体部位': '肝脏', '数值': 'NA', '单位': 'NA', '修饰词': '无'}, {'术语': '肝功能异常', '语义类型': '症状、体征、临床所见', '叙述状态': '存在', '身体部位': '肝脏', '数值': 'NA', '单位': 'NA', '修饰词': '无'}, {'术语': '病毒性肝炎', '语义类型': '疾病、综合征、病理功能', '叙述状态': '存在', '身体部位': '肝脏', '数值': 'NA', '单位': 'NA', '修饰词': '无'}, {'术语': '肝纤维化', '语义类型': '疾病、综合征、病理功能', '叙述状态': '存在', '身体部位': '肝脏', '数值': 'NA', '单位': 'NA', '修饰词': '无'}]
which does not reflect a problem.
Can you provide your environment settings and specific code for the generation, so we may reproduce your problem and debug.
Thank you!
Thank you for your response! Here are my environment details:
- OS: Ubuntu 22.04 - GPU: RTX 3090 - CUDA Version: 12.1 - PyTorch Version: 2.5.1+cu121
And here is the code snippet I am using:
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
MODEL_NAME = "../../data/GENIE_zh_7b"
LOCAL_MODEL_DIR = "./hf_model"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, cache_dir=LOCAL_MODEL_DIR)
model = AutoModelForCausalLM.from_pretrained(
MODEL_NAME,
cache_dir=LOCAL_MODEL_DIR,
torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
low_cpu_mem_usage=True
)
max_new_tokens = 512
PROMPT_TEMPLATE = "Human:\n{query}\n\nAssistant:"
EHR = ['慢性乙型肝炎病史10余年，曾有肝功能异常，中医治疗后好转；1年余前查HBsAg转阴，但肝脏病理提示病毒性肝炎伴肝纤维化（G1S3-4）']
texts = [PROMPT_TEMPLATE.format(query=k) for k in EHR]
for text in texts:
inputs = tokenizer(text, return_tensors="pt").to(device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
pad_token_id=tokenizer.eos_token_id
)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)
Written in complement: A quick fix can be: change prompt template to: PROMPT_TEMPLATE = "Human:\n{query}\n\nAssistant:\n"
Hi, I have reproduced your problems, but we have not figured out the reason. It has been witnessed in various github issues that, "vllm" and "huggingface transformers" performs rather differently when given the same parameters.
We will dig into this,
but for this while, if you want to use GENIE, please refer to the vllm package and example code in the model card. Sorry for the inconvenience.