--- library_name: transformers tags: [] --- # Model Card for Model ID ## Model Details ### Model Description This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated. - **Developed by:** Haoxiang Wang - **Model type:** Decoder-only LLM - **Language(s) (NLP):** English - **License:** Apache-2.0 - **Finetuned from model [optional]:** https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2 ### Model Sources [optional] - **Repository:** https://github.com/RLHFlow/directional-preference-alignment - **Paper [ACL 2024]:** [Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards](https://arxiv.org/abs/2402.18571) ## How to Get Started with the Model Use the code below to get started with the model. + System Prompt: + Template: `"You are a helpful, respectful, and honest assistant who always responds to the user in a harmless way. Your response should maximize weighted rating = helpfulness*{weight_helpfulness} + verbosity*{weight_verbosity}"` + Value Choices: `weight_helpfulness` is an integer from 0 to 100 and `(weight_verbosity/100)**2 + (weight_helpfulness/100)**2 == 1` + The maximum `weight_helpfulness` is 100 the lowest suggested value is 71. + The model will generate a response that implicitly maximizes the weighted rating `helpfulness*weight_helpfulness + verbosity*weight_verbosity`, where `helpfulness` and `verbosity` are two reward objectives that range from 0 to 100. We suggest starting with a ratio of `weight_verbosity/weight_helpfulness` first. For instance, considering `weight_verbosity/weight_helpfulness` is equal to `tan(-15°)` ```python from transformers import AutoTokenizer, AutoModelForCausalLM import torch import numpy as np # Here we show how to use the DPA model to generate a response to a user prompt. device = "cuda" model = AutoModelForCausalLM.from_pretrained("RLHFlow/DPA-v1-Mistral-7B", torch_dtype=torch.bfloat16, device_map=device) tokenizer = AutoTokenizer.from_pretrained("Haoxiang-Wang/DPA-v1-Mistral-7B") degree = -15 # weight_verbosity/weight_helpfulness = tan(-15°) rad = np.radians(degree) # convert from degree to radian weight_helpfulness = np.round((np.cos(rad) * 100)).astype(int) # compute weight_helpfulness, scale it by 100x, and round it to an integer weight_verbosity = np.round((np.sin(rad) * 100)).astype(int) # compute weight_verbosity, scale it by 100x, and round it to an integer ## Now (weight_helpfulness/100)**2 + (weight_verbosity/100)**2 ≈ 1 - it is not an exact equivalence due to the round() operations above sys_prompt = f"You are a helpful, respectful, and honest assistant who always responds to the user in a harmless way. Your response should maximize weighted rating = helpfulness*{weight_helpfulness} + verbosity*{weight_verbosity}" user_prompt = "Write a summary of Romeo and Juliet." messages = [ {"role": "system", "content": sys_prompt}, {"role": "user", "content": user_prompt}, ] input_ids = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(device) output = model.generate(input_ids=input_ids, max_new_tokens=2048,temperature=0.7) prompt_len = input_ids.shape[-1] generated_response = tokenizer.decode(output[0][prompt_len:], skip_special_tokens=True) print(generated_response) # 'Romeo and Juliet is a tragic love story written by William Shakespeare, believed to have been written between 1591 and 1595. The play is based on an Italian tale called "The Tragical History of Romeus and Juliet" by Arthur Brooke, which was published in 1562.\n\nThe story revolves around two young star-crossed lovers, Romeo Montague and Juliet Capulet, from rival families in Verona, Italy. Their love is forbidden by their families, who have a long-standing feud. Despite the obstacles, Romeo and Juliet marry in secret and spend a few blissful days together before fate intervenes.\n\nA series of misunderstandings, miscommunications, and tragic events lead to the deaths of both Romeo and Juliet. Romeo believes that Juliet is dead, and in a fit of despair, he takes his own life. Juliet, who is actually still alive, awakens to find Romeo dead and takes her own life in grief.\n\nThe play explores themes of love, hate, fate, and the consequences of actions. It is known for its iconic characters, including the passionate Romeo, the fiery Juliet, and the noble Friar Lawrence, who tries to help the young lovers.\n\nRomeo and Juliet has been adapted into numerous films, stage productions, and other media over the years, and it remains a beloved and tragic tale of forbidden love.' ``` ## Training ![image/png](https://github.com/RLHFlow/directional-preference-alignment/raw/main/assets/preference-conflict.jpg) ![image/png](https://github.com/RLHFlow/directional-preference-alignment/raw/main/assets/algo-illustration.jpg) ## Evaluation ![image/png](https://cdn-uploads.huggingface.co/production/uploads/638fb8cf2380ffd99caf8c2a/IEO1xFOzopiOEWrxlDCem.png) ## Citation **BibTeX:** If you find this work useful to your research, please consider citing our paper ``` @inproceedings{wang2024arithmetic, title={Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards}, author={Haoxiang Wang and Yong Lin and Wei Xiong and Rui Yang and Shizhe Diao and Shuang Qiu and Han Zhao and Tong Zhang}, year={2024}, booktitle={ACL}, } ``` ## Model Card Authors Haoxiang Wang ## Model Card Contact hwang264@illinois.edu