README.md

metadata

license: apache-2.0
datasets:
  - databricks/databricks-dolly-15k
  - lucasmccabe-lmi/CodeAlpaca-20k
model-index:
  - name: Instruct_Yi-6B_Dolly_CodeAlpaca
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 53.16
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=HenryJJ/Instruct_Yi-6B_Dolly_CodeAlpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 75.3
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=HenryJJ/Instruct_Yi-6B_Dolly_CodeAlpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 63.06
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=HenryJJ/Instruct_Yi-6B_Dolly_CodeAlpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 41.42
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=HenryJJ/Instruct_Yi-6B_Dolly_CodeAlpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 75.37
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=HenryJJ/Instruct_Yi-6B_Dolly_CodeAlpaca
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 28.35
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=HenryJJ/Instruct_Yi-6B_Dolly_CodeAlpaca
          name: Open LLM Leaderboard

Instruct_Yi-6B_Dolly15K

Fine-tuned from Yi-6B， used Dolly15k for the dataset. 90% for training, 10% validation. Trained for 2.0 epochs using Lora. Trained with 2048 context window. Compared with https://huggingface.co/HenryJJ/Instruct_Yi-6B_Dolly15K, I add additional CodeAlpaca_20K dataset that good at coding.

Model Details

• Trained by: trained by HenryJJ.
• Model type: Instruct_Yi-6B_Dolly15K is an auto-regressive language model based on the Llama 2 transformer architecture.
• Language(s): English
• License for Instruct_Yi-6B_Dolly15K: apache-2.0 license

Prompting

Prompt Template With Context

<|startoftext|>[INST]{instruction} {context}[/INST]{response}<|endoftext|>

<|startoftext|>[INST]
Write a 10-line poem about a given topic
The topic is about racecars
[/INST]

Prompt Template Without Context

<|startoftext|>[INST]
Who was the was the second president of the United States?
[/INST]

Training script:

Fully opensourced at: https://github.com/hengjiUSTC/learn-llm/blob/main/trl_finetune.py. Run on aws g4dn.12xlarge instance for 10 hours.

python3 trl_finetune.py --config configs/yi_6b-large.yml

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	56.11
AI2 Reasoning Challenge (25-Shot)	53.16
HellaSwag (10-Shot)	75.30
MMLU (5-Shot)	63.06
TruthfulQA (0-shot)	41.42
Winogrande (5-shot)	75.37
GSM8k (5-shot)	28.35