|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- stanfordnlp/SHP |
|
- Anthropic/hh-rlhf |
|
- OpenAssistant/oasst1 |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
tags: |
|
- human feedback |
|
- rlhf |
|
- preferences |
|
- alignment |
|
- HALO |
|
- halos |
|
- dpo |
|
- rl |
|
--- |
|
|
|
 |
|
|
|
This repo contains the model checkpoints for: |
|
- model family <b>pythia1-4b</b> |
|
- optimized with the loss <b>KTO</b> |
|
- aligned using the SHP, Anthropic HH and Open Assistant datasets. |
|
|
|
To prompt Archangel models, ensure that the format is consistent with that of TuluV2. |
|
For example, a prompt should be formatted as follows, where `<|user|>` corresponds to the human's role and `<|assistant|>` corresponds to the LLM's role. |
|
The human should speak first: |
|
``` |
|
<|user|> |
|
Hi! I'm looking for a cake recipe. |
|
<|assistant|> |
|
What kind of cake? |
|
<|user|> |
|
Chocolate cake. |
|
<|assistant|> |
|
``` |
|
Note that a beginning-of-sequence (BOS) token is automatically added by all Archangel models during tokenization and does not have to be added by you. No end-of-sequence (EOS) token is added to the prompt. |
|
|
|
Please refer to our [code repository](https://github.com/ContextualAI/HALOs) or [blog](https://contextual.ai/better-cheaper-faster-llm-alignment-with-kto/) which contains intructions for training your own HALOs and links to our model cards. |
|
|
|
If you find this repo or the technical paper useful in your research, please feel free to cite [our work](https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf): |
|
``` |
|
@techreport{ethayarajh2023halos, |
|
author = {Ethayarajh, Kawin and Xu, Winnie, and Jurafsky, Dan and Kiela, Douwe}, |
|
title = {Human-Centered Loss Functions (HALOs)}, |
|
institution = {Contextual AI}, |
|
note = {https://github.com/ContextualAI/HALOs/blob/main/assets/report.pdf}, |
|
year = {2023}, |
|
} |
|
``` |
