Update README.md

cdc9e50 verified about 1 month ago

No virus

5.9 kB

	---
	license: apache-2.0
	tags:
	- Solar Moe
	- Solar
	- Lumosia
	pipeline_tag: text-generation
	model-index:
	- name: Lumosia-v2-MoE-4x10.7
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 70.39
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Steelskull/Lumosia-v2-MoE-4x10.7
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 87.87
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Steelskull/Lumosia-v2-MoE-4x10.7
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 66.45
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Steelskull/Lumosia-v2-MoE-4x10.7
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 68.48
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Steelskull/Lumosia-v2-MoE-4x10.7
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 84.21
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Steelskull/Lumosia-v2-MoE-4x10.7
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 65.13
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Steelskull/Lumosia-v2-MoE-4x10.7
	name: Open LLM Leaderboard
	---
	# Lumosia-v2-MoE-4x10.7

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/64545af5ec40bbbd01242ca6/fKdOLTQNerr2fYYnWOiQD.png)

	The Lumosia Series upgraded with Lumosia V2.

	# What's New in Lumosia V2?

	Lumosia V2 takes the original vision of being an "all-rounder" and refines it with more nuanced capabilities.

	Topic/Prompt Based Approach:

	Diverging from the keyword-based approach of its counterpart, Umbra.

	Context and Coherence:

	With a base context of 8k scrolling window and the ability to maintain coherence up to 16k.

	Balanced and Versatile:

	The core ethos of Lumosia V2 is balance. It's designed to be your go-to assistant.

	Experimentation and User-Centric Development:

	Lumosia V2 remains an experimental model, a mosaic of the best-performing Solar models, (selected based on user experience).
	This version is a testament to the idea that innovation is a journey, not a destination.


	Template:
	```
	### System:

	### USER:{prompt}

	### Assistant:
	```


	Settings:
	```
	Temp: 1.0
	min-p: 0.02-0.1
	```

	## Evals:

	* Avg:
	* ARC:
	* HellaSwag:
	* MMLU:
	* T-QA:
	* Winogrande:
	* GSM8K:

	## Examples:
	```
	Example 1:

	User:

	Lumosia:

	```
	```
	Example 2:

	User:

	Lumosia:

	```

	## 🧩 Configuration

	```
	yaml
	base_model: DopeorNope/SOLARC-M-10.7B
	gate_mode: hidden
	dtype: bfloat16

	experts:
	- source_model: DopeorNope/SOLARC-M-10.7B
	positive_prompts:

	negative_prompts:

	- source_model: Sao10K/Fimbulvetr-10.7B-v1 [Updated]
	positive_prompts:

	negative_prompts:

	- source_model: jeonsworld/CarbonVillain-en-10.7B-v4 [Updated]
	positive_prompts:

	negative_prompts:

	- source_model: kyujinpy/Sakura-SOLAR-Instruct
	positive_prompts:

	negative_prompts:
	```

	## 💻 Usage

	```
	python
	!pip install -qU transformers bitsandbytes accelerate

	from transformers import AutoTokenizer
	import transformers
	import torch

	model = "Steelskull/Lumosia-v2-MoE-4x10.7"

	tokenizer = AutoTokenizer.from_pretrained(model)
	pipeline = transformers.pipeline(
	"text-generation",
	model=model,
	model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
	)

	messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
	prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
	print(outputs[0]["generated_text"])
	```
	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Steelskull__Lumosia-v2-MoE-4x10.7)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|73.75\|
	\|AI2 Reasoning Challenge (25-Shot)\|70.39\|
	\|HellaSwag (10-Shot) \|87.87\|
	\|MMLU (5-Shot) \|66.45\|
	\|TruthfulQA (0-shot) \|68.48\|
	\|Winogrande (5-shot) \|84.21\|
	\|GSM8k (5-shot) \|65.13\|