Update README.md

README.md

@@ -3,30 +3,41 @@ model-index:
 - name: notus-7b-v1-lora-adapter
   results: []
 datasets:
-- argilla/ultrafeedback-binarized-avg-rating-for-dpo
+- argilla/ultrafeedback-binarized-preferences
 language:
 - en
 base_model: alignment-handbook/zephyr-7b-sft-full
-library_name: peft
+library_name: transformers
 pipeline_tag: text-generation
 tags:
 - dpo
 - preference
 - ultrafeedback
-license: apache-2.0
+- lora
+license: mit
 ---
 
-# Model Card for Notus 7B v1 (LoRA Adapters)
-
 <div align="center">
-  <img src="https://cdn-uploads.huggingface.co/production/uploads/60f0608166e5701b80ed3f02/LU-vKiC0R7UxxITrwE1F_.png" alt="Image was artificially generated by Dalle-3 via ChatGPT Pro"/>
+  <img src="https://cdn-uploads.huggingface.co/production/uploads/60f0608166e5701b80ed3f02/x4YL8t0IX52HWzLOzkKR3.png" alt="A banner representing Notus, the wind god of the south, in a mythical and artistic style. The banner features a strong, swirling breeze, embodying the warm, wet character of the southern wind. Gracefully flowing across the scene are several paper planes, caught in the gentle yet powerful gusts of Notus. The background is a blend of warm colors, symbolizing the heat of the south, with hints of blue and green to represent the moisture carried by this wind. The overall atmosphere is one of dynamic movement and warmth."/>
 </div>
 
-Notus is going to be a collection of fine-tuned models using DPO, similarly to Zephyr, but mainly focused
-on the Direct Preference Optimization (DPO) step, aiming to incorporate preference feedback into the LLMs
-when fine-tuning those. Notus models are intended to be used as assistants via chat-like applications, and 
-are evaluated with the MT-Bench, AlpacaEval, and LM Evaluation Harness benchmarks, to be directly compared
-with Zephyr fine-tuned models also using DPO.
+# Model Card for Notus 7B v1 (LoRA Adapters)
+
+Notus is a collection of fine-tuned models using Direct Preference Optimization (DPO) and related RLHF techniques. This model is the first version, fine-tuned with DPO over `zephyr-7b-sft-full`, which is the SFT model produced to create `zephyr-7b-beta`. 
+
+Following a **data-first** approach, the only difference between Notus-7B-v1 and Zephyr-7B-beta is the preference dataset used for dDPO. 
+
+In particular, when we started building [distilabel](https://github.com/argilla-io/distilabel), we invested time understanding and deep-diving into the UltraFeedback dataset. Using [Argilla](https://argilla.io/), we've found data issues in the original UltraFeedback dataset, leading to high-scores for bad responses (more details in the training data section). After curating several hundreds of data points, we decided to binarize the dataset using the preference ratings, instead of the original critique `overall_score`, and verified the new dataset with Argilla.
+
+Using preference ratings, instead of critiques scores, led to a new dataset where the chosen response is different in ~50% of the cases. Using this new dataset with DPO we fine-tuned Notus, a 7B model, that **surpasses Zephyr-7B-beta and Claude 2 on AlpacaEval**.
+
+> **Important note**: While we opted for the average of multi-aspect ratings, while we fix the original dataset, a very interesting open question remains: once critique data is fixed, what works better? using the critique scores or the preference ratings? We're very excited to do this comparison in the coming weeks, stay tuned!
+
+This model **wouldn't have been possible without the amazing [Alignment Handbook](https://github.com/huggingface/alignment-handbook), [OpenBMB](https://www.openbmb.cn/home) for releasing the Ultrafeedback dataset**, and it's based on fruitful discussions with the HuggingFace H4 team. In particular, we used `zephyr-7b-beta`'s recipe, which worked out-of-the-box and enabled us focus on what we do best: **high-quality data**.
+
+Notus models are intended to be used as assistants via chat-like applications, and are evaluated with Chat (MT-Bench, AlpacaEval) and Academic (Open LLM Leaderboard) benchmarks for a direct comparison with the original Zephyr dDPO model and other 7B models.
+
+> **Why Notus?**: Notus name comes from the ancient Greek god Notus, as a wink to Zephyr, which comes from the ancient Greek god Zephyrus; with the difference that Notus is the god of the south wind, and Zephyr the god of the west wind. More information at https://en.wikipedia.org/wiki/Anemoi.
 
 ## Model Details
 
@@ -41,10 +52,46 @@ with Zephyr fine-tuned models also using DPO.
 
 ### Model Sources [optional]
 
-- **Repository:** https://github.com/argilla-io/notus-7b
+- **Repository:** https://github.com/argilla-io/notus
 - **Paper:** N/A
 - **Demo:** https://argilla-notus-chat-ui.hf.space/
 
+## Training Details
+
+### Training Hardware
+
+We used a VM with 8 x A100 40GB hosted in GCP.
+
+### Training Data
+
+We used a a new curated version of [`openbmb/UltraFeedback`](https://huggingface.co/datasets/openbmb/UltraFeedback), named [`argilla/ultrafeedback-binarized-preferences`](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences).
+
+TL;DR
+
+After visually browsing around some examples using the sort and filter feature of Argilla (sort by highest rating for chosen responses), we noticed a strong mismatch between the `overall_score` in the original UF dataset (and the Zephyr train_prefs dataset) and the quality of the chosen response. 
+
+By adding the critique rationale to our Argilla Dataset, we confirmed the critique rationale was highly negative, whereas the rating was very high (the highest in fact: `10`). 
+
+See screenshot below for one example of this issue. 
+
+After some quick investigation, we identified hundreds of examples having the same issue, reported a bug on the UltraFeedback repo, and informed the H4 team.
+
+While we're working on fixing the original dataset (already narrowed down ~2K problematic examples). We decided to leverage the multi-preference ratings, leading to Notus!
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/60420dccc15e823a685f2b03/M9qCKyAB_G1MbVBAPeitd.png)
+
+## Prompt template
+
+We use the same prompt template as [`HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta):
+
+```
+<|system|>
+</s>
+<|user|>
+{prompt}</s>
+<|assistant|>
+```
+
 ## Usage
 
 As the current model only contains the adapters, you will need to use PEFT to merge the adapters into the original model first.