Adding Evaluation Results (#1)

ac5f76a verified 3 months ago

No virus

4.98 kB

	---
	license: apache-2.0
	tags:
	- finetuned
	pipeline_tag: text-generation
	model-index:
	- name: deepseek-coder-1.3b-chat
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 25.85
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=AIGym/deepseek-coder-1.3b-chat
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 39.59
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=AIGym/deepseek-coder-1.3b-chat
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 26.36
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=AIGym/deepseek-coder-1.3b-chat
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 43.92
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=AIGym/deepseek-coder-1.3b-chat
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 51.7
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=AIGym/deepseek-coder-1.3b-chat
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 3.03
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=AIGym/deepseek-coder-1.3b-chat
	name: Open LLM Leaderboard
	---

	# deepseek-coder-1.3b-chat
	It was created by starting with the deepseek-coder-1.3b and training it on the open assistant dataset. We have attached the wandb report in pdf form to view the training run at a glance.

	# Reson
	This model was fine tned to allow it to follow direction and is a steeping stone to further training, but still would be good for asking qestions about code.

	# How to use
	You will need the transformers>=4.31
	```python
	from transformers import AutoTokenizer
	import transformers
	import torch
	model = "AIGym/deepseek-coder-1.3b-chat"
	tokenizer = AutoTokenizer.from_pretrained(model)
	pipeline = transformers.pipeline(
	"text-generation",
	model=model,
	torch_dtype=torch.float16,
	device_map="auto",
	)

	prompt = "What are the values in open source projects?"
	formatted_prompt = (
	f"### Human: {prompt}### Assistant:"
	)


	sequences = pipeline(
	formatted_prompt,
	do_sample=True,
	top_k=50,
	top_p = 0.7,
	num_return_sequences=1,
	repetition_penalty=1.1,
	max_new_tokens=500,
	)
	for seq in sequences:
	print(f"Result: {seq['generated_text']}")
	```

	# Referrals
	Run Pod - This is who I use to train th emodels on huggingface. If you use it we both get free crdits. - <a href="https://runpod.io?ref=kilq83n1" target="_blank" style="color: #3498db; text-decoration: none; font-weight: bold;">Visit Runpod's Website!</a>

	Paypal - If you want to leave a tip, it is appecaheted. - <a href="https://paypal.me/OpenSourceTraining" target="_blank" style="color: #3498db; text-decoration: none; font-weight: bold;">Visit My Paypal!</a>
	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_AIGym__deepseek-coder-1.3b-chat)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|31.74\|
	\|AI2 Reasoning Challenge (25-Shot)\|25.85\|
	\|HellaSwag (10-Shot) \|39.59\|
	\|MMLU (5-Shot) \|26.36\|
	\|TruthfulQA (0-shot) \|43.92\|
	\|Winogrande (5-shot) \|51.70\|
	\|GSM8k (5-shot) \| 3.03\|