upload model

README.md

@@ -0,0 +1,69 @@
+---
+license: mit
+---
+
+#### Current Training Steps: 45000
+
+
+This repo contains a low-rank adapter (LoRA) for Bloom-7b1 
+fit on the [Stanford-Alpaca-52k](https://github.com/tatsu-lab/stanford_alpaca)
+and [databricks-dolly-15k](https://github.com/databrickslabs/dolly/tree/master/data) data in 52 languages.
+
+### Dataset Creation
+
+1. English Instructions: The English instuctions are obtained from [alpaca-52k](https://github.com/tatsu-lab/stanford_alpaca), and [dolly-15k](https://github.com/databrickslabs/dolly/tree/master/data).
+2. Instruction Translation: The instructions (and inputs) are translated into the target languages using Google Translation API (conducted on April 2023).
+3. Output Generation: We generate output from `gpt-3.5-turbo` for each language (conducted on April 2023).
+
+<h3 align="center">
+<img src="https://raw.githubusercontent.com/fajri91/eval_picts/master/BactrianX_dataset.jpg" width="950" align="center">
+</h3>
+
+### Training Parameters
+
+The code for training the model is provided in our [github](https://github.com/mbzuai-nlp/Bactrian-X), which is adapted from [Alpaca-LoRA](https://github.com/tloen/alpaca-lora).
+This version of the weights was trained with the following hyperparameters:
+
+
+- Epochs: 10
+- Batch size: 128
+- Cutoff length: 512
+- Learning rate: 3e-4
+- Lora _r_: 64
+- Lora target modules: q_proj, k_proj, v_proj, o_proj
+
+
+That is:
+
+```
+python finetune.py \
+    --base_model='bigscience/bloom-7b1' \
+    --num_epochs=10 \
+    --batch_size=128 \
+    --cutoff_len=512 \
+    --group_by_length \
+    --output_dir='./bactrian-x-bloom-7b1-lora' \
+    --lora_target_modules='q_proj,k_proj,v_proj,o_proj' \
+    --lora_r=64 \
+    --micro_batch_size=32
+```
+
+Instructions for running it can be found at https://github.com/MBZUAI-nlp/Bactrian-X.
+
+### Discussion of Biases
+
+(1) Translation bias; (2) Potential English-culture bias in the translated dataset.
+
+
+### Citation Information
+
+```
+@misc{bactrian,
+  author = {Haonan Li and Fajri Koto and Minghao Wu and Alham Fikri Aji and Timothy Baldwin},
+  title = {Bactrian-X: A Multilingual Replicable Instruction-Following Model},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/MBZUAI-nlp/Bactrian-X}},
+}
+```

adapter_config.json

@@ -0,0 +1,16 @@
+{
+  "base_model_name_or_path": "bigscience/bloom-7b1",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "lora_alpha": 16,
+  "lora_dropout": 0.05,
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 64,
+  "target_modules": [
+    "query_key_value"
+  ],
+  "task_type": "CAUSAL_LM"
+}

adapter_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1514278c059c70f3cbfc300a3a9908bb2aa9eafd8b0a8da8b5593e64b90262d2
+size 125851429