---
language:
- en
license: wtfpl
library_name: transformers
tags:
- code
- text-generation-inference
datasets:
- flytech/python-codes-25k
- espejelomar/code_search_net_python_10000_examples
metrics:
- accuracy
pipeline_tag: text-generation
---
# Model Card for GPT_2_CODE
-Goal is to create a small GPT2 python coder
#  Table of Contents
- [Model Card for GPT_2_CODE](#model-card-for--model_id-)
- [Table of Contents](#table-of-contents)
- [Table of Contents](#table-of-contents-1)
- [Model Details](#model-details)
  - [Model Description](#model-description)
- [Uses](#uses)
  - [Direct Use](#direct-use)
  - [Downstream Use [Optional]](#downstream-use-optional)
  - [Out-of-Scope Use](#out-of-scope-use)
- [Bias, Risks, and Limitations](#bias-risks-and-limitations)
  - [Recommendations](#recommendations)
- [Training Details](#training-details)
  - [Training Data](#training-data)
  - [Training Procedure](#training-procedure)
    - [Preprocessing](#preprocessing)
    - [Speeds, Sizes, Times](#speeds-sizes-times)
- [Evaluation](#evaluation)
  - [Testing Data, Factors & Metrics](#testing-data-factors--metrics)
    - [Testing Data](#testing-data)
    - [Factors](#factors)
    - [Metrics](#metrics)
  - [Results](#results)
- [Model Examination](#model-examination)
- [Environmental Impact](#environmental-impact)
- [Technical Specifications [optional]](#technical-specifications-optional)
  - [Model Architecture and Objective](#model-architecture-and-objective)
  - [Compute Infrastructure](#compute-infrastructure)
    - [Hardware](#hardware)
    - [Software](#software)
- [Citation](#citation)
- [Glossary [optional]](#glossary-optional)
- [More Information [optional]](#more-information-optional)
- [Model Card Authors [optional]](#model-card-authors-optional)
- [Model Card Contact](#model-card-contact)
- [How to Get Started with the Model](#how-to-get-started-with-the-model)
# Model Details
## Model Description
WIP,Goal is to create a small GPT2 python coder
- **Developed by:** C, o, d, e, M, o, n, k, e, y
- **Shared by [Optional]:** More information needed
- **Model type:** Language model
- **Language(s) (NLP):** eng
- **License:** wtfpl
- **Parent Model:** More information needed
- **Resources for more information:** More information needed
    - [GitHub Repo](None)
    - [Associated Paper](None)
# Uses
coding assistant
## Direct Use
generate python code snippets
## Downstream Use [Optional]
semi auto coder
## Out-of-Scope Use
describe code
Keep Finetuning on question/python datasets
# Training Details

## Training Data
flytech/python-codes-25k
espejelomar/code_search_net_python_10000_examples
## Training Procedure
Train/Val/Scheduler
### Preprocessing
More information needed
### Speeds, Sizes, Times
Epochs 3
# "flytech/python-codes-25k"
Training Loss: 0.4007
Validation Loss: 0.5526
Epochs 3

# "espejelomar/code_search_net_python_10000_examples"
--Starting Loss: 2.0862
-Epoch 1/4 | Training Loss: 1.5355 | Validation Loss: 1.1723
-Epoch 2/4 | Training Loss: 1.0501 | Validation Loss: 1.0702
-Epoch 3/4 | Training Loss: 0.9804 | Validation Loss: 1.0798
-Epoch 4/4 | Training Loss: 0.9073 | Validation Loss: 1.0772

# Evaluation
Manual comparison with base model
### Testing Data
flytech/python-codes-25k
espejelomar/code_search_net_python_10000_examples
### Factors
80/20 train/val
### Metrics
train/validate 
lr scheduling
## Results 
Better in python code generation as base gpt2-medium model
# Model Examination
More information needed
# Environmental Impact
- **Hardware Type:** CPU and Colab T4
- **Hours used:** 4
- **Cloud Provider:** Google Colab
- **Compute Region:** NL
## Model Architecture and Objective
gpt2
## Compute Infrastructure
More information needed
### Hardware
CPU and Colab T4
### Software
pytorch, custom python
# More Information [optional]
Experimental
# Model Card Authors [optional]
CodeMonkeyXL
# Model Card Contact
K00B404 huggingface
# How to Get Started with the Model
Use the code below to get started with the model.