# ci_thresholding

Weight-sparse transformer trained with the procedure from Gao et al. (2025).

## Model Details

- **Layers**: 2
- **Model Dimension**: 3072
- **Context Length**: 512
- **Head Dimension**: 16
- **Vocabulary Size**: 4096

## Sparsity

- **Weight Sparsity**: True
- **Target L0 Fraction**: 0.05
- **Activation Sparsity**: True

## Training

- **Dataset**: jacobcd52/simplestories-tokenized
- **Tokenizer**: SimpleStories/SimpleStories-1.25M
- **Total Tokens**: 1,000,000,000

## Training Run

- **W&B Run**: [https://wandb.ai/asher577/nodesparse_adv_pretraining/runs/g7kp6fcv](https://wandb.ai/asher577/nodesparse_adv_pretraining/runs/g7kp6fcv)

## Usage

```python
import torch
from huggingface_hub import hf_hub_download

# Download model
model_path = hf_hub_download(repo_id="asher577/ci_thresholding", filename="pytorch_model.bin")
config_path = hf_hub_download(repo_id="asher577/ci_thresholding", filename="config.json")

# Load (requires the SparseGPT model class from this repo)
state_dict = torch.load(model_path)
```