|
--- |
|
license: other |
|
base_model: deepseek-ai/deepseek-coder-1.3b-base |
|
tags: |
|
- axolotl |
|
- generated_from_trainer |
|
model-index: |
|
- name: deepseek-coder-1.3b-typescript |
|
results: [] |
|
datasets: |
|
- bigcode/the-stack-dedup |
|
widget: |
|
- text: "class Person {\n constructor(public name:" |
|
example_title: "class" |
|
- text: "function quickSort" |
|
example_title: "function" |
|
--- |
|
|
|
<p align="center"> |
|
<img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="codegpt-deepseek-typescript.png?raw=true"> |
|
</p> |
|
<p align="center"><a href="https://codegpt.co/">[CodeGPT.co]</a> | <a href="https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript">[🦙 Ollama]</a> | <a href="https://discord.gg/fKyyJX5pne">[Discord]</a> | <a href="https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.dscodegpt">[VSCode Extension]</a> </p> |
|
<hr> |
|
|
|
[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl) |
|
<details><summary>See axolotl config</summary> |
|
|
|
axolotl version: `0.3.0` |
|
```yaml |
|
base_model: deepseek-ai/deepseek-coder-1.3b-base |
|
model_type: AutoModelForCausalLM |
|
trust_remote_code: true |
|
load_in_8bit: false |
|
load_in_4bit: false |
|
strict: false |
|
|
|
|
|
datasets: |
|
- path: CodeGPTPlus/typescript-0-500000-seq1024 |
|
type: completion |
|
field: text |
|
|
|
|
|
val_set_size: 0.001 |
|
output_dir: ./fft-out |
|
|
|
sequence_len: 1024 |
|
|
|
adapter: |
|
lora_model_dir: |
|
lora_r: |
|
lora_alpha: |
|
lora_dropout: |
|
lora_target_linear: |
|
lora_fan_in_fan_out: |
|
lora_modules_to_save: |
|
|
|
wandb_project: deepseek_1.3_fft |
|
wandb_entity: |
|
wandb_watch: |
|
wandb_name: aws_a10g |
|
wandb_log_model: end |
|
|
|
|
|
gradient_accumulation_steps: 2 |
|
micro_batch_size: 20 |
|
num_epochs: 1 |
|
optimizer: adamw_bnb_8bit |
|
adam_beta1: 0.9 |
|
adam_beta2: 0.999 |
|
adam_epsilon: 0.000001 |
|
max_grad_norm: 1.0 |
|
weight_decay: 0.1 |
|
lr_scheduler: cosine |
|
learning_rate: 0.00002 |
|
train_on_inputs: false |
|
group_by_length: false |
|
bf16: true |
|
fp16: false |
|
tf32: false |
|
gradient_checkpointing: true |
|
early_stopping_patience: |
|
resume_from_checkpoint: |
|
local_rank: |
|
logging_steps: 1 |
|
xformers_attention: |
|
flash_attention: true |
|
|
|
loss_watchdog_threshold: 5.0 |
|
loss_watchdog_patience: 3 |
|
|
|
hub_model_id: CodeGPTPlus/deepseek_coder_1.3b_typescript |
|
hub_strategy: every_save |
|
warmup_ratio: 0.01 |
|
evals_per_epoch: 20 |
|
saves_per_epoch: 3 |
|
debug: |
|
deepspeed: |
|
|
|
fsdp: |
|
fsdp_config: |
|
special_tokens: |
|
bos_token: "<｜begin▁of▁sentence｜>" |
|
eos_token: "<｜end▁of▁sentence｜>" |
|
pad_token: "<｜end▁of▁sentence｜>" |
|
``` |
|
|
|
</details><br> |
|
|
|
# deepseek-coder-1.3b-typescript |
|
|
|
CodeGPTPlus/deepseek-coder-1.3b-typescript, emerges as a fine-tuned iteration of [deepseek-ai/deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base), meticulously crafted by the CodeGPT team to excel in generating expert code in TypeScript. With specific fine-tuning for TypeScript and a dataset of 0.5B tokens, this model excels in producing precise and efficient solutions in this programming language. |
|
|
|
The 16K window size and an additional fill-in-the-middle task are employed to deliver project-level code completion. |
|
|
|
This new model stands as the ideal choice for those seeking a specialized code generator for TypeScript, backed by the expertise of the CodeGPT team. |
|
|
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.7681 |
|
|
|
**Model Developers** CodeGPT Team |
|
|
|
**Variations** 1.3B |
|
|
|
**Input** Models input text only. |
|
|
|
**Output** Models generate text only. |
|
|
|
## How to Use |
|
This model is for completion purposes only. Here give some examples of how to use the model. |
|
|
|
#### Running the model on a GPU |
|
```python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
tokenizer = AutoTokenizer.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript", |
|
trust_remote_code=True) |
|
model = AutoModelForCausalLM.from_pretrained("CodeGPTPlus/deepseek-coder-1.3b-typescript", |
|
trust_remote_code=True).cuda() |
|
|
|
input_text = """<｜fim▁begin｜>function quickSort(arr: number[]): number[] { |
|
if (arr.length <= 1) { |
|
return arr; |
|
} |
|
const pivot = arr[0]; |
|
const left = []; |
|
const right = []; |
|
<｜fim▁hole｜> |
|
return [...quickSort(left), pivot, ...quickSort(right)]; |
|
}<｜fim▁end｜>""" |
|
|
|
inputs = tokenizer(input_text, return_tensors="pt").to(model.device) |
|
outputs = model.generate(**inputs, max_length=256) |
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
``` |
|
|
|
### Running with Ollama |
|
**Model:** https://ollama.ai/codegpt/deepseek-coder-1.3b-typescript |
|
|
|
```ollama run codegpt/deepseek-coder-1.3b-typescript``` |
|
|
|
### Running with Ollama and CodeGPT Autocomplete in VSCode |
|
|
|
**Documentation:** https://docs.codegpt.co/docs/tutorial-features/code_autocompletion |
|
|
|
Select "Ollama - codegpt/deepseek-coder-1.3b-typescript" in the autocomplete model selector. |
|
|
|
Then, write any code or comment in the vscode text editor, and the model will provide you with code suggestions through the CodeGPT code autocomplete. |
|
|
|
<img width="1000px" alt="CodeGPT: DeepSeek Coder - Typescript" src="ollama_autocomplete_codegpt.gif"> |
|
|
|
### Fill In the Middle (FIM) |
|
```python |
|
<｜fim▁begin｜>function quickSort(arr: number[]): number[] { |
|
if (arr.length <= 1) { |
|
return arr; |
|
} |
|
const pivot = arr[0]; |
|
const left = []; |
|
const right = []; |
|
<｜fim▁hole｜> |
|
return [...quickSort(left), pivot, ...quickSort(right)]; |
|
}<｜fim▁end｜> |
|
``` |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 2e-05 |
|
- train_batch_size: 20 |
|
- eval_batch_size: 20 |
|
- seed: 42 |
|
- gradient_accumulation_steps: 2 |
|
- total_train_batch_size: 40 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-06 |
|
- lr_scheduler_type: cosine |
|
- lr_scheduler_warmup_steps: 261 |
|
- num_epochs: 1 |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | |
|
|:-------------:|:-----:|:-----:|:---------------:| |
|
| 1.0745 | 0.0 | 1 | 0.8681 | |
|
| 1.2267 | 0.05 | 1308 | 0.8130 | |
|
| 1.1594 | 0.1 | 2616 | 0.8018 | |
|
| 0.7674 | 0.15 | 3924 | 0.7942 | |
|
| 0.6443 | 0.2 | 5232 | 0.7889 | |
|
| 0.9155 | 0.25 | 6540 | 0.7847 | |
|
| 0.7501 | 0.3 | 7848 | 0.7819 | |
|
| 0.8835 | 0.35 | 9156 | 0.7792 | |
|
| 0.7261 | 0.4 | 10464 | 0.7769 | |
|
| 0.9746 | 0.45 | 11772 | 0.7748 | |
|
| 0.6884 | 0.5 | 13080 | 0.7734 | |
|
| 0.6104 | 0.55 | 14388 | 0.7722 | |
|
| 0.8876 | 0.6 | 15696 | 0.7710 | |
|
| 0.9567 | 0.65 | 17004 | 0.7703 | |
|
| 0.6915 | 0.7 | 18312 | 0.7696 | |
|
| 0.8874 | 0.75 | 19620 | 0.7691 | |
|
| 0.6124 | 0.8 | 20928 | 0.7686 | |
|
| 0.8147 | 0.85 | 22236 | 0.7684 | |
|
| 0.8021 | 0.9 | 23544 | 0.7683 | |
|
| 0.8665 | 0.95 | 24852 | 0.7681 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.37.0.dev0 |
|
- Pytorch 2.0.1+cu118 |
|
- Datasets 2.16.1 |
|
- Tokenizers 0.15.0 |
