santacoderpack / README.md

Update README.md

fb30a58 10 months ago

No virus

5.81 kB

	---
	pipeline_tag: text-generation
	inference: true
	widget:
	- text: '<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n for idx, elem in enumerate(numbers):\n for idx2, elem2 in enumerate(numbers):\n if idx != idx2:\n distance = elem - elem2\n if distance < threshold:\n return True\n\n return False<commit_message>Fix bugs in has_close_elements.<commit_after>'
	example_title: Fix has_close_elements
	group: Python
	license: bigcode-openrail-m
	datasets:
	- bigcode/commitpack-subset-cf
	metrics:
	- code_eval
	library_name: transformers
	tags:
	- code
	model-index:
	- name: SantaCoderPack
	results:
	- task:
	type: text-generation
	dataset:
	type: bigcode/humanevalpack
	name: HumanEvalFix Python
	metrics:
	- name: pass@1
	type: pass@1
	value: 3.2
	verified: false
	- task:
	type: text-generation
	dataset:
	type: bigcode/humanevalpack
	name: HumanEvalFix JavaScript
	metrics:
	- name: pass@1
	type: pass@1
	value: 4.9
	verified: false
	- task:
	type: text-generation
	dataset:
	type: bigcode/humanevalpack
	name: HumanEvalFix Java
	metrics:
	- name: pass@1
	type: pass@1
	value: 1.8
	verified: false
	- task:
	type: text-generation
	dataset:
	type: bigcode/humanevalpack
	name: HumanEvalFix Go
	metrics:
	- name: pass@1
	type: pass@1
	value: 3.6
	verified: false
	- task:
	type: text-generation
	dataset:
	type: bigcode/humanevalpack
	name: HumanEvalFix C++
	metrics:
	- name: pass@1
	type: pass@1
	value: 4.2
	verified: false
	- task:
	type: text-generation
	dataset:
	type: bigcode/humanevalpack
	name: HumanEvalFix Rust
	metrics:
	- name: pass@1
	type: pass@1
	value: 1.7
	verified: false
	- task:
	type: text-generation
	dataset:
	type: bigcode/humanevalpack
	name: HumanEvalFix Average
	metrics:
	- name: pass@1
	type: pass@1
	value: 3.3
	verified: false
	---
	![Octopack](https://github.com/bigcode-project/octopack/blob/31f3320f098703c7910e43492c39366eeea68d83/banner.png?raw=true)

	# Table of Contents

	1. [Model Summary](#model-summary)
	2. [Use](#use)
	3. [Training](#training)
	4. [Citation](#citation)

	# Model Summary

	SantaCoderPack is an pre-trained model with the same architecture of SantaCoder on <th><a href=https://huggingface.co/datasets/bigcode/commitpack>CommitPack</a> using this format:

	```
	<commit_before>code_before<commit_msg>message<commit_after>code_after
	```

	- Repository: [bigcode/octopack](https://github.com/bigcode-project/octopack)
	- Paper: [OctoPack: Instruction Tuning Code Large Language Models](https://arxiv.org/abs/2308.07124)
	- Languages: Python, JavaScript, Java, C++, Go, Rust
	- SantaCoderPack:
	<table>
	<tr>
	<th>Data</t>
	<th><a href=https://huggingface.co/datasets/bigcode/commitpack>CommitPack</a></th>
	<td>4TB of GitHub commits across 350 programming languages</td>
	</tr>
	<tr>
	<th>Model</t>
	<th><a href=https://huggingface.co/bigcode/octocoder>SantaCoderPack</a></th>
	<td>SantaCoderPack (1.1B parameters) pre-trained on CommitPack</td>
	</tr>
	<tr>
	<th>Evaluation  </t>
	<th><a href=https://huggingface.co/datasets/bigcode/humanevalpack>HumanEvalPack/HumanEvalFix</a></th>
	<td>Extension of OpenAI's HumanEval to HumanEvalFix</td>
	</tr>
	</table>


	# Use

	## Intended use

	The model follows instructions provided in the input. We recommend prefacing your input with "<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n for idx, elem in enumerate(numbers):\n for idx2, elem2 in enumerate(numbers):\n if idx != idx2:\n distance = elem - elem2\n if distance < threshold:\n return True\n\n return False<commit_message>Fix bugs in has_close_elements.<commit_after>"

	Feel free to share your generations in the Community tab!

	## Generation

	```python
	# pip install -q transformers
	from transformers import AutoModelForCausalLM, AutoTokenizer
	checkpoint = "bigcode/santacoderpack"
	device = "cuda" # for GPU usage or "cpu" for CPU usage
	tokenizer = AutoTokenizer.from_pretrained(checkpoint)
	model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
	inputs = tokenizer.encode("Q<commit_before>def has_close_elements(numbers: List[float], threshold: float) -> bool:\n for idx, elem in enumerate(numbers):\n for idx2, elem2 in enumerate(numbers):\n if idx != idx2:\n distance = elem - elem2\n if distance < threshold:\n return True\n\n return False<commit_message>Fix bugs in has_close_elements.<commit_after>", return_tensors="pt").to(device)
	outputs = model.generate(inputs)
	print(tokenizer.decode(outputs[0]))
	```

	# Training

	## Model

	- Architecture: GPT-2 model with multi-query attention
	- Steps: 250k pretraining
	- Pretraining tokens: 131B
	- Precision: bfloat16

	## Hardware

	- Pretraining:
	- GPUs: 32 Tesla A100
	- Training time: 15 days

	## Software

	- Orchestration: [Megatron-LM/Transformers](https://github.com/bigcode-project/santacoderpack#training)
	- Neural networks: [PyTorch](https://github.com/pytorch/pytorch)

	# Citation

	```bibtex
	@article{muennighoff2023octopack,
	title={OctoPack: Instruction Tuning Code Large Language Models},
	author={Niklas Muennighoff and Qian Liu and Armel Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro von Werra and Shayne Longpre},
	journal={arXiv preprint arXiv:2308.07124},
	year={2023}
	}
	```