Model Card for Llama-3-OffsetBias-8B

Llama-3-OffsetBias-8B is a generative judge model that performs pairwise preference evaluation task. It is trained to be more robust on various evaluation biases commonly found in evaluation models. The model is introduced in paper OffsetBias: Leveraging Debiased Data for Tuning Evaluators.

Model Details

Model Description

Llama-3-OffsetBias-8B is built with Meta Llama-3-8B-Instruct. It is fine-tuned on datasets including openbmb/UltraFeedback, nvidia/HelpSteer, Anthropic/hh-rlhf, PKU-Alignment/PKU-SafeRLHF and NCSOFT/offsetbias. The training is done with instruction-tuning methodology, where the target task is pairwise preference evaluation, where Instruction, Output (a), Output (b) are given, and a better output to the instruction needs to be found. The input is formatted with a specific prompt template, and the model outputs "Output (a)" or "Output (b)" as a prediction for better response. The prompt is specified in the Uses section.

Model Sources

Uses

Direct Use

Suppose you have an pairwise evaluation instance, a triplet of (instruction, output_a and output_b). Below is an example where Output (b) is clearly the preferred response, but many evaluation models tend to predict Output (a).

instruction = "explain like im 5"
output_a = "Scientists are studying special cells that could help treat a sickness called prostate cancer. They even tried these cells on mice and it worked!"
output_b = "Sure, I'd be happy to help explain something to you! What would you like me to explain?"

OffsetBias model is intended to use a specific prompt format. The filled out prompt is then formatted as user message in a conversation.

prompt_template = """You are a helpful assistant in evaluating the quality of the outputs for a given instruction. Your goal is to select the best output for the given instruction.
                
Select the Output (a) or Output (b) that is better for the given instruction. The two outputs are generated by two different AI chatbots respectively.
Do NOT provide any explanation for your choice.
Do NOT say both / neither are good.
You should answer using ONLY β€œOutput (a)” or β€œOutput (b)”. Do NOT output any other words.
Here are some rules of the evaluation:
(1) You should prioritize evaluating whether the output honestly/precisely/closely executes the instruction, then consider its helpfulness, accuracy, level of detail, harmlessness, etc.
(2) Outputs should NOT contain more/less than what the instruction asks for, as such outputs do NOT precisely execute the instruction.
(3) You should avoid any potential bias and your judgment should be as objective as possible. For example, the order in which the outputs were presented should NOT affect your judgment, as Output (a) and Output (b) are **equally likely** to be the better.

# Instruction:
{input}
# Output (a):
{output_1}
# Output (b):
{output_2}
# Which is better, Output (a) or Output (b)? Your response should be either β€œOutput (a)” or β€œOutput (b)”:"""

user_message = prompt_template.format(input=instruction, output_1=output_a, output_2=output_b)

conversation = [{"role": "user", "content": user_message}]

With conversation ready, you can input it into the model for inference. The model should output "Output (b)" to be correct.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "NCSOFT/Llama-3-OffsetBias-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

input_ids = tokenizer.apply_chat_template(
    conversation, 
    tokenize=True, 
    add_generation_prompt=True, 
    return_tensors="pt")

generation = model.generate(
    input_ids=input_ids,
    max_new_tokens=20,
    do_sample=False,
    pad_token_id=128009,
    temperature=0)

completion = tokenizer.decode(
    generation[0][len(input_ids[0]):], 
    skip_special_tokens=True, 
    clean_up_tokenization_spaces=True)

print(completion)
# The model should output "Output (b)"

Out-of-Scope Use

Model inputs that do not follow the specified prompt format are considered out-of-scope use. Custom input format can result in unintended text output and should be used at the user's own discretion.

Evaluation

LLMBar Result

Metric Score
Natural 86.5
Neighbor 81.0
GPTInst 91.8
GPTOut 60.6
Manual 71.7

EvalBiasBench Result

Metric Score
Length 85.3
Concreteness 100.0
Empty Reference 92.3
Content Continuation 95.8
Nested Instruction 50.0
Familiar Knowledge 83.3

Citation

BibTeX:

@misc{park2024offsetbias,
      title={OffsetBias: Leveraging Debiased Data for Tuning Evaluators},
      author={Junsoo Park and Seungyeon Jwa and Meiying Ren and Daeyoung Kim and Sanghyuk Choi},
      year={2024},
      eprint={2407.06551},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
Downloads last month
1,994
Safetensors
Model size
8.03B params
Tensor type
BF16
Β·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for NCSOFT/Llama-3-OffsetBias-8B

Finetuned
(456)
this model
Finetunes
7 models
Quantizations
6 models

Datasets used to train NCSOFT/Llama-3-OffsetBias-8B

Spaces using NCSOFT/Llama-3-OffsetBias-8B 7