README.md

metadata

pipeline_tag: text-generation
inference: false
license: apache-2.0
model-index:
  - name: ibm/PowerMoE-3b
    results:
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: ARC
        metrics:
          - name: accuracy-norm
            type: accuracy-norm
            value: 58.1
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: BoolQ
        metrics:
          - name: accuracy
            type: accuracy
            value: 65
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: Hellaswag
        metrics:
          - name: accuracy-norm
            type: accuracy-norm
            value: 71.5
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: OpenBookQA
        metrics:
          - name: accuracy-norm
            type: accuracy-norm
            value: 41
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: PIQA
        metrics:
          - name: accuracy-norm
            type: accuracy-norm
            value: 79.1
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: Winogrande
        metrics:
          - name: accuracy-norm
            type: accuracy-norm
            value: 65
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: MMLU (5 shot)
        metrics:
          - name: accuracy
            type: accuracy
            value: 42.8
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: GSM8k (5 shot)
        metrics:
          - name: accuracy
            type: accuracy
            value: 25.9
            verified: false
      - task:
          type: text-generation
        dataset:
          type: lm-eval-harness
          name: math (4 shot)
        metrics:
          - name: accuracy
            type: accuracy
            value: 14.8
            verified: false
      - task:
          type: text-generation
        dataset:
          type: bigcode-eval
          name: humaneval
        metrics:
          - name: pass@1
            type: pass@1
            value: 20.1
            verified: false
      - task:
          type: text-generation
        dataset:
          type: bigcode-eval
          name: MBPP
        metrics:
          - name: pass@1
            type: pass@1
            value: 32.4
            verified: false
base_model:
  - ibm/PowerMoE-3b

Model Summary

PowerMoE-3B is a 3B sparse Mixture-of-Experts (sMoE) language model trained with the Power learning rate scheduler. It sparsely activates 800M parameters for each token. It is trained on a mix of open-source and proprietary datasets. PowerMoE-3B has shown promising results compared to other dense models with 2x activate parameters across various benchmarks, including natural language multi-choices, code generation, and math reasoning. Paper: https://arxiv.org/abs/2408.13359

This is a GGUF quantized version.

Usage

Requires latest llama.cpp to run.

Generation

This is a simple example of how to use the PowerMoe GGUF:

./llama-cli -m PowerMoE4x800M_q3km.gguf -p "How about a snack?"