Model Card

README.md

@@ -0,0 +1,166 @@
+---
+pipeline_tag: text-generation
+license: apache-2.0
+tags:
+- text generation
+- Deci AI
+- DeciCoder
+programming_language:
+  - Java
+  - JavaScript
+  - Python
+  - Rust
+  - Go
+  - C++
+  - C
+  - C#
+metrics:
+- code_eval
+inference: true
+widget:
+- text: 'def print_hello_world():'
+  example_title: Hello world
+  group: Python
+model-index:
+- name: DeciCoder-6b
+  results:
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-HumanEval (Python)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.34
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-HumanEval (JavaScript)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.29
+      verified: false
+  - task:
+      type: text-generation
+    dataset:
+      type: nuprl/MultiPL-E
+      name: MultiPL-HumanEval (Java)
+    metrics:
+    - name: pass@1
+      type: pass@1
+      value: 0.30
+      verified: false
+datasets:
+- bigcode/starcoderdata
+---
+
+# Model Card for DeciCoder 6B
+
+DeciCoder 6B is a 6 billion parameter decoder-only code completion model
+trained on the Python, Java, Javascript, Go, Rust, C++, C, and C# subset of [Starcoder Training Dataset](https://huggingface.co/datasets/bigcode/starcoderdata)..
+The model uses variable Grouped Query Attention and has a context window of 4096
+tokens. It was trained using a Fill-in-the-Middle training objective. The model's
+architecture was generated by Deci's proprietary Neural Architecture
+Search-based technology, AutoNAC.
+
+## Model Details
+
+- **Developed by:** Deci 
+- **Model type:** DeciCoder is an auto-regressive language model based on the transformer decoder architecture, using variable Grouped Query Attention.
+- **Language(s):** Python, Java, JavaScript, Go, Rust, C++, C, C#
+- **License:** Model checkpoints are licensed under the [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+## Model Architecture
+
+| Parameters | Layers | Heads  | Sequence Length  | GQA num_key_value_heads  | Hidden Size  |
+|:----------|:----------|:----------|:----------|:----------|:----------|
+| 6B    | 32    | 32    | 4096   | Variable  | 4096 |  |
+
+
+- **Decoder layer:** Variable Grouped Query Attention. Grouped Query Attention was introduced in [Ainslie et al., 2023](https://arxiv.org/abs/2305.13245)
+- **Position Embeddings:** Rotary Position Embeddings [Su et al., 2021](https://arxiv.org/abs/2104.09864)
+
+## Uses
+
+The model is intended to do single/multiline code completion from a
+context window of up to 4096k tokens. It is *not* an instruction model
+and commands like \"Write a function that computes the absolute value of
+an integer,\" won't yield the desired results. A more effective approach
+is to frame instructions in the style of source code comments (e.g. \#
+this function calculates the absolute value of an integer) or to present
+a function signature and docstring, enabling the model to complete the
+function's body.
+
+### How to Use
+
+```bibtex
+# pip install -q transformers
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+checkpoint = "Deci/DeciCoder-6b"
+device = "cuda" # for GPU usage or "cpu" for CPU usage
+
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.bfloat16, trust_remote_code=True).to(device)
+
+inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
+outputs = model.generate(inputs, max_new_tokens=100)
+print(tokenizer.decode(outputs[0]))
+
+### Attribution
+
+DeciCoder was trained on StarCoder Training Dataset, filtered for
+Python, Java, JavaScript, Rust, Go, C++, C, and C#. For additional information, please
+refer to [https://huggingface.co/datasets/bigcode/starcoderdata](https://huggingface.co/datasets/bigcode/starcoderdata).
+
+```
+
+### Limitations
+
+The model has undergone training with source code from Python, Java,
+JavaScript, Go, Rust, C++, C, and C#. While the primary language in the source is English, it does
+contain other languages. Therefore, the model can produce code snippets
+given some context. However, there\'s no assurance that the resulting
+code will function as expected. It might be suboptimal, contain bugs, or
+even exploits.
+
+## Evaluation
+
+Below are DeciCoder's pass@1 on MultiPL HumanEval scores
+
+| Python | JavaScript | Java  | C++  | C#  | Rust  | Go  | C  | 
+|:----------|:----------|:----------|:----------|:----------|:----------|:----------|:----------|
+| 33.5%    | 29.3%    | 30.3%    |29.93%    |20.31%    |20.5%    |77.47%    |xx%    |
+
+
+### Runtime Benchmarks
+
+|Inference Tool/Hardware | Qualcomm AI 100 (tokens/sec) |
+|:----------|:----------|
+| Infery LLM | xxx   | 
+
+- Throughput (tokens/sec) - Measured with an optimal batch size of 96
+
+## Documentation
+
+- [Notebook](https://colab.research.google.com/drive/1JCxvBsWCZKHfIcHSMVf7GZCs3ClMQPjs) CHANGE
+- Blog post: [Introducing DeciCoder: The New Gold Standard in Efficient and Accurate Code Generation](https://deci.ai/blog/decicoder-efficient-and-accurate-code-generation-llm/)CHANGE
+- Questions:Feel free to contact us via our [Discord Community!](https://discord.com/invite/p9ecgRhDR8/)CHANGE
+
+## How to Cite
+
+Please cite this model using this format.
+
+```bibtex
+@misc{DeciFoundationModels,
+title = {DeciCoder},
+author = {DeciAI Research Team},
+year = {2023}
+url={[https://huggingface.co/deci/decicoder-6b](https://huggingface.co/deci/decicoder-6b)},
+}
+```