Update README.md

README.md

@@ -1,58 +1,73 @@
 ---
+language: 
+  - en
+  - code
+  
+license: "mit"
+
 tags:
-- generated_from_trainer
-datasets:
-- final_dataset
-model-index:
-- name: diff_6b_full
-  results: []
+- Diff Model
+- pytorch
+- causal-lm
+- code-generation
+- The Pile
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+# Diff-Codegen-6B v2 Model Card
+
+## Model Description
+
+diff-codegen-6b-v2 is a diff model for code generation, released by [CarperAI](http://carper.ai/). A diff model is an autoregressive language model trained on edits to a piece of text, formatted in [Unified Diff Format](https://en.wikipedia.org/wiki/Diff#Unified_format). These diff models can suggest, given a section of text and a description of the desired change, an intelligent change to the text that fits the description, marking the lines added, changed, and deleted in diff format.
+
+In comparison to few-shot prompting of normal code generation models, diff models are specialized for suggesting intelligent changes to existing code, particularly longer pieces of code and where a change is required to follow some natural language text description (provided in the form of a commit message).
+
+This model is a fine-tune of [codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) by Salesforce, trained on a large dataset of commits scraped from GitHub.
+
+diff-codegen-6b-v2 is an experimental research artifact and should be treated as such. We are releasing these results and this model in the hopes that it may be useful to the greater research community, especially those interested in LMs for code.
+
+## Training Data
+
+This model is a fine-tune of [codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) by Salesforce. This language model was first pre-trained on The Pile, an 800Gb dataset composed of varied web corpora. The datasheet and paper for the Pile can be found [here](https://arxiv.org/abs/2201.07311) and [here](https://arxiv.org/abs/2101.00027) respectively. The model was then fine-tuned on a large corpus of code data in multiple languages, before finally being fine-tuned on a Python code dataset. The Codegen paper with full details of these datasets can be found [here](https://arxiv.org/abs/2203.13474).
+
+Our dataset for this fine-tune consists of commits from GitHub, obtained using the [Google BigQuery Public Dataset](https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code), a public up to date snapshot of a huge number of open-source GitHub repositories. We took this dataset and filtered using [BigQuery](https://console.cloud.google.com/marketplace/details/github/github-repos) on the number of stars in the repository to exclude repos with less than 100 stars, and further restricted the query to only repositories with open-source non-copyleft licenses (e.g. MIT, Apache, etc) and commits with more than 10 characters in the commit message. We also restricted ourselves to a list of 22 popular programming, scripting, and markup languages, including Python, HTML, Bash scripts, SQL, C++, etc. This resulted in a dataset of 19 million commits after filtering.
+
+Our diff model was trained on a dataset of commits from BigQuery, a large-scale dataset of many programming languages from GitHub repositories. We filtered the dataset by the number of stars in the repository (>100 stars), license (only open-source non-copyleft licensed code included), and length of file (files greater than 2048 tokens in length were excluded).
+
+The model was trained using the Huggingface Codegen tokenizer.
 
-# diff_6b_full
+## Training Details
 
-This model is a fine-tuned version of [codegen-6b](https://huggingface.co/codegen-6b) on the final_dataset dataset.
+The model was trained on 1.08 billion tokens for 1 epoch on 64 A100 GPUs, provided by [Stability AI](https://stability.ai/).
 
-## Model description
+Each file was formatted as follows for input to the language model:
 
-More information needed
+```
+<NME> {FILE_NAME}
+<BEF> {INPUT_FILE}
+<MSG> {COMMIT_MESSAGE}
+<DFF> {FILE_DIFF}
+```
 
-## Intended uses & limitations
+## Intended Uses and Limitations
 
-More information needed
+Due to the model’s small size and restriction to code, one should not expect the model to generalize to domains beyond code and perform (successful) reasoning over large chunks of code. This model is intended to be used in prototyping code generation systems, and for solely experimental purposes. This model is provided without warranty and should not be used in commercial settings—even though the license permits.
 
-## Training and evaluation data
+## Limitations and Biases
 
-More information needed
+Due to the short context length restriction and due to the fact that all repositories with under 100 stars were excluded, we expect our diff model to underperform on underrepresented languages, for instance Lean or Coq.
 
-## Training procedure
+The output of this model should not be trusted as correct and secure code. This model should not be used in any mission critical setting where security is of importance. When running the output of this model, it should be done as much as possible in a sandbox, such as [gVisor](https://gvisor.dev), since it is very possible for the model to produce code which may delete files, send HTTP requests, or otherwise contain critical security vulnerabilities.
 
-### Training hyperparameters
+As with other language models, diff-codegen is prone to hallucination and biased, stereotyped, or toxic output. There are no guarantees of truthful output when generating from the model.
 
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 2
-- eval_batch_size: 1
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 64
-- gradient_accumulation_steps: 2
-- total_train_batch_size: 256
-- total_eval_batch_size: 64
-- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 895
-- num_epochs: 1.0
+## Evaluation Results
 
-### Training results
+See our blog post for full evaluation results.
 
+## Licensing
 
+This model is licensed as MIT.
 
-### Framework versions
+## Acknowledgements
 
-- Transformers 4.22.0.dev0
-- Pytorch 1.13.0
-- Datasets 2.7.1
-- Tokenizers 0.12.1
+We’d like to thank Honglu Fan, Harry Saini, Herbie Bradley, Reshinth Adithyan, and Joel Lehman for their efforts! Thanks to Nitarshan Rajkumar for feedback on this model card.