Pruned MoEs (Mixtral-8x7B-Instruct-v0.1)

Na0s 's Collections

Depth pruned and fine tuned Llama-3.1-8B

updated Nov 18

Pruned experts from Mixtral-8x7B-Instruct-v0.1 with respect to the paper "A Provably Effective Method for Pruning Experts in Fine-tuned Sparse MoEs"

Upvote

mistralai/Mixtral-8x7B-Instruct-v0.1

Text Generation • Updated Aug 19 • 3.16M • • 4.24k
Na0s/Mixtral-8x7B-Instruct-v0.1-LoRA-on-Gates

Text Generation • Updated Sep 26 • 32
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA

Text Generation • Updated Nov 12 • 26
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-1-experts

Text Generation • Updated Oct 6 • 23
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-2-experts

Text Generation • Updated Oct 6 • 23
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-3-experts

Text Generation • Updated Oct 6 • 28
Na0s/Mixtral-8x7B-v0.1-instruct-pruned-random-4-experts

Text Generation • Updated Oct 6 • 23
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-1-expert

Text Generation • Updated Oct 6 • 28
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-2-experts

Text Generation • Updated Oct 5 • 17
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-3-experts

Text Generation • Updated Oct 6 • 29
Na0s/Mixtral-8x7B-v0.1-instruct-l2-norm-post-Gates-LoRA-SFT-pruned-4-experts

Text Generation • Updated Oct 6 • 25
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-1-expert

Text Generation • Updated Nov 18 • 12
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-2-experts

Text Generation • Updated Nov 18 • 10
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-3-experts

Text Generation • Updated Nov 18 • 11
Na0s/Mixtral-8x7B-Instruct-v0.1-exhaustive-LoRA-SFT-pruned-4-experts

Text Generation • Updated Nov 18 • 11

Upvote