Create README.md

README.md

@@ -0,0 +1,53 @@
+Intro
+Sparse computing is increasingly recognized as an important direction to improve the computational efficiency of large language models (LLM). Among various approaches, a mixture of experts (MoE) methods (exemplified by models such as Mixtral) show particular promise. MoE works by selectively activating different model components (experts), thereby optimizing resource usage.
+Recent studies (Zhang et al., 2021; Liu et al., 2023; Mirzadeh et al., 2023) have shown that LLM inherently exhibits properties that favor sparse computation when using the ReLU activation function. This insight opens new avenues for model efficiency, similar to the selective activation of MoE. By dynamically selecting model parameters for calculations, we can significantly improve efficiency.
+However, the widespread adoption of ReLU-based models in the LLM field is still limited. To inspire more research for inference efficiency, we introduce our  Mistral-level ReLU-based LLM model, XXX.
+Model Architecture
+As the ReGLU-based LLM has limited sparsity, for example, ReLULLaMA has just nearly 70% sparsity. To further push the sparsity, we add a relu component after GLU. So our FFN network works as follows:
+class XXXMLP(nn.Module):                                                                                                                   
+    def __init__(self, config):                                                                                                                
+        super().__init__()                                                                                                                     
+        self.config = config                                                                                                                   
+        self.hidden_size = config.hidden_size                                                                                                  
+        self.intermediate_size = config.intermediate_size                                                                                      
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)                                                       
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)                                                         
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)                                                       
+        self.act_fn = ACT2FN[config.hidden_act]                                                                                                
+                                                                                                                                               
+    def forward(self, x):                                                                                                                      
+        return self.down_proj(self.act_fn(self.gate_proj(x)) * self.act_fn(self.up_proj(x)))
+Training Details
+In this subsection, we will introduce the details of training our model, including types of data used, and hyperparameters.
+We initialized the model weights to Mistral's model weights and modified the FFN structure to the ReGLU+ReLU structure, then continued pre-training for 200B tokens, divided into two phases:
+First phase: For the proportion of training corpus, we followed the data mix ratio and sources of the StableLM-3B model, conducting a further pre-training with 150B tokens.(link)
+The following table shows the hyper-paramters we used in our training process.
+|-------------------------|----------------------|
+| GPUs                    | 64 80G-A100          |
+| Learning Rate Control   | Cosine               |
+| Peak Learning Rate      | 5e-5                 |
+| Batch Size              | 4M                   |
+| Weight Decay            | 0.1                  |
+Second phase: We further adjusted the training corpus ratio, incorporating more domain-specific datasets(Math、Coding), and continued training for 50B tokens.
+|-------------------------|----------------------|
+| GPUs                    | 64 80G-A100          |
+| Learning Rate Control   | Cosine               |
+| Peak Learning Rate      | 5e-6                 |
+| Batch Size              | 4M                   |
+| Weight Decay            | 0.01                  |
+Performance Evaluation Results
+Our evaluation is based on the framework lm-evaluation-harness and opencompass. The evaluation details are listed as follows:
+- Huggingface LLM Leaderboard tasks.
+- Commonsense: We report the average of PIQA, SIQA,  ARC easy and challenge and  CommonsenseQA.
+- Other Popular Benchmarks: We report the average accuracies on Big Bench Hard (BBH) (3-shot), HumanEval, MBPP, MATH.
+
+| MMLU    | Winogrande | TruthfulQA | Hellaswag | GSM8K  | Arc-C  |        |   |   |   |   |   |
+|---------|------------|------------|-----------|--------|--------|--------|---|---|---|---|---|
+| Ours    | 0.6389     | 0.7593     | 0.4406    | 0.8217 | 0.5315 | 0.6195 |   |   |   |   |   |
+| Mistral | 0.6265     | 0.7924     | 0.4262    | 0.8332 | 0.4018 | 0.6143 |   |   |   |   |   |
+
+
+
+
+Speed Evaluation Results
+We utilize PowerInfer, a state-of-the-art acceleration framework leveraging activation sparsity. Here we show the inference speed compared with llama.cpp/transformers.