Feature Extraction
Transformers
Safetensors
English
bamboo
custom_code
yixinsong commited on
Commit
ed75727
1 Parent(s): 058d800

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -3
README.md CHANGED
@@ -44,7 +44,7 @@ The following table shows the hyper-paramters we used in our training process.
44
  | Batch Size | 4M |
45
  | Weight Decay | 0.1 |
46
 
47
- **Second phase**: We further adjusted the training corpus ratio, incorporating more domain-specific datasets(MathCoding), and continued training for 50B tokens.
48
 
49
  | Hyper-parameters | |
50
  | --------------------- | ----------- |
@@ -59,8 +59,7 @@ The following table shows the hyper-paramters we used in our training process.
59
  Our evaluation is based on the framework lm-evaluation-harness and opencompass. The evaluation details are listed as follows:
60
 
61
  - Huggingface LLM Leaderboard tasks.
62
- - Commonsense: We report the average of PIQA, SIQA, ARC easy and challenge and CommonsenseQA.
63
- - Other Popular Benchmarks: We report the average accuracies on Big Bench Hard (BBH) (3-shot), HumanEval, MBPP, MATH.
64
 
65
  | | MMLU | Winogrande | TruthfulQA | Hellaswag | GSM8K | Arc-C | HumanEval | BBH | Average |
66
  | ------- | ------ | ---------- | ---------- | --------- | ------ | ------ | --------- | ---- | ------- |
 
44
  | Batch Size | 4M |
45
  | Weight Decay | 0.1 |
46
 
47
+ **Second phase**: We further adjusted the training corpus ratio, incorporating more domain-specific datasets(Math, Coding), and continued training for 50B tokens.
48
 
49
  | Hyper-parameters | |
50
  | --------------------- | ----------- |
 
59
  Our evaluation is based on the framework lm-evaluation-harness and opencompass. The evaluation details are listed as follows:
60
 
61
  - Huggingface LLM Leaderboard tasks.
62
+ - Other Popular Benchmarks: We report the average accuracies on Big Bench Hard (BBH) (3-shot), HumanEval.
 
63
 
64
  | | MMLU | Winogrande | TruthfulQA | Hellaswag | GSM8K | Arc-C | HumanEval | BBH | Average |
65
  | ------- | ------ | ---------- | ---------- | --------- | ------ | ------ | --------- | ---- | ------- |