Upload folder using huggingface_hub

a21bdf4 verified 3 months ago

No virus

6.14 kB

	---
	language:
	- en
	license: apache-2.0
	tags:
	- smol_llama
	- llama2
	datasets:
	- JeanKaddour/minipile
	- pszemraj/simple_wikipedia_LM
	- mattymchen/refinedweb-3m
	- BEE-spoke-data/knowledge-inoc-concat-v1
	inference:
	parameters:
	max_new_tokens: 64
	do_sample: true
	temperature: 0.8
	repetition_penalty: 1.05
	no_repeat_ngram_size: 4
	eta_cutoff: 0.0006
	renormalize_logits: true
	widget:
	- text: My name is El Microondas the Wise, and
	example_title: El Microondas
	- text: Kennesaw State University is a public
	example_title: Kennesaw State University
	- text: Bungie Studios is an American video game developer. They are most famous for
	developing the award winning Halo series of video games. They also made Destiny.
	The studio was founded
	example_title: Bungie
	- text: The Mona Lisa is a world-renowned painting created by
	example_title: Mona Lisa
	- text: The Harry Potter series, written by J.K. Rowling, begins with the book titled
	example_title: Harry Potter Series
	- text: 'Question: I have cities, but no houses. I have mountains, but no trees. I
	have water, but no fish. What am I?

	Answer:'
	example_title: Riddle
	- text: The process of photosynthesis involves the conversion of
	example_title: Photosynthesis
	- text: Jane went to the store to buy some groceries. She picked up apples, oranges,
	and a loaf of bread. When she got home, she realized she forgot
	example_title: Story Continuation
	- text: 'Problem 2: If a train leaves Station A at 9:00 AM and travels at 60 mph,
	and another train leaves Station B at 10:00 AM and travels at 80 mph, when will
	they meet if the distance between the stations is 300 miles?

	To determine'
	example_title: Math Problem
	- text: In the context of computer programming, an algorithm is
	example_title: Algorithm Definition
	pipeline_tag: text-generation
	model-index:
	- name: smol_llama-220M-GQA
	results:
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: AI2 Reasoning Challenge (25-Shot)
	type: ai2_arc
	config: ARC-Challenge
	split: test
	args:
	num_few_shot: 25
	metrics:
	- type: acc_norm
	value: 24.83
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: HellaSwag (10-Shot)
	type: hellaswag
	split: validation
	args:
	num_few_shot: 10
	metrics:
	- type: acc_norm
	value: 29.76
	name: normalized accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: MMLU (5-Shot)
	type: cais/mmlu
	config: all
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 25.85
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: TruthfulQA (0-shot)
	type: truthful_qa
	config: multiple_choice
	split: validation
	args:
	num_few_shot: 0
	metrics:
	- type: mc2
	value: 44.55
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: Winogrande (5-shot)
	type: winogrande
	config: winogrande_xl
	split: validation
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 50.99
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA
	name: Open LLM Leaderboard
	- task:
	type: text-generation
	name: Text Generation
	dataset:
	name: GSM8k (5-shot)
	type: gsm8k
	config: main
	split: test
	args:
	num_few_shot: 5
	metrics:
	- type: acc
	value: 0.68
	name: accuracy
	source:
	url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=BEE-spoke-data/smol_llama-220M-GQA
	name: Open LLM Leaderboard
	---


	# smol_llama: 220M GQA

	> model card WIP, more details to come


	A small 220M param (total) decoder model. This is the first version of the model.

	- 1024 hidden size, 10 layers
	- GQA (32 heads, 8 key-value), context length 2048
	- train-from-scratch on one GPU :)

	## Links

	[Here](https://huggingface.co/collections/BEE-spoke-data/finetuned-smol-220m-65998b080ae723e79c830f83) are some fine-tunes we did, but there are many more possibilities out there!

	- instruct
	- openhermes - [link](https://huggingface.co/BEE-spoke-data/smol_llama-220M-openhermes)
	- open-instruct - [link](https://huggingface.co/BEE-spoke-data/smol_llama-220M-open_instruct)
	- code
	- python (pypi) - [link](https://huggingface.co/BEE-spoke-data/beecoder-220M-python)
	- zephyr DPO tune
	- SFT - [link](https://huggingface.co/BEE-spoke-data/zephyr-220m-sft-full)
	- full DPO - [link](https://huggingface.co/BEE-spoke-data/zephyr-220m-dpo-full)

	---

	# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
	Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_BEE-spoke-data__smol_llama-220M-GQA)

	\| Metric \|Value\|
	\|---------------------------------\|----:\|
	\|Avg. \|29.44\|
	\|AI2 Reasoning Challenge (25-Shot)\|24.83\|
	\|HellaSwag (10-Shot) \|29.76\|
	\|MMLU (5-Shot) \|25.85\|
	\|TruthfulQA (0-shot) \|44.55\|
	\|Winogrande (5-shot) \|50.99\|
	\|GSM8k (5-shot) \| 0.68\|