Text2Text Generation
Transformers
PyTorch
t5
codet5
text-generation-inference
nielsr HF staff commited on
Commit
6f0f019
1 Parent(s): 3c3c9f4

First draft of model card

Browse files
Files changed (1) hide show
  1. README.md +114 -0
README.md ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ datasets:
5
+ - code_search_net
6
+ ---
7
+
8
+ # CodeT5 (base-sized model)
9
+
10
+ Pre-trained CodeT5 model. It was introduced in the paper [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models
11
+ for Code Understanding and Generation](https://arxiv.org/abs/2109.00859) by Yue Wang, Weishi Wang, Shafiq Joty, Steven C.H. Hoi and first released in [this repository](https://github.com/salesforce/CodeT5).
12
+
13
+ Disclaimer: The team releasing CodeT5 did not write a model card for this model so this model card has been written by the Hugging Face team (more specifically, [nielsr](https://huggingface.co/nielsr)).
14
+
15
+ ## Model description
16
+
17
+ From the abstract:
18
+
19
+ "We present CodeT5, a unified pre-trained encoder-decoder Transformer model that better leverages the code semantics conveyed from the developer-assigned identifiers. Our model employs a unified framework to seamlessly support both code understanding and generation tasks and allows for multi-task learning. Besides, we propose a novel identifier-aware pre-training task that enables the model to distinguish which code tokens are identifiers and to recover them when they are masked. Furthermore, we propose to exploit the user-written code comments with a bimodal dual generation task for better NL-PL alignment. Comprehensive experiments show that CodeT5 significantly outperforms prior methods on understanding tasks such as code defect detection and clone detection, and generation tasks across various directions including PL-NL, NL-PL, and PL-PL. Further analysis reveals that our model can better capture semantic information from code."
20
+
21
+ ## Intended uses & limitations
22
+
23
+ You can use the model to fine-tune it on code understanding tasks, such as . See the [model hub](https://huggingface.co/models?search=google/vit) to look for
24
+ fine-tuned versions on a task that interests you.
25
+
26
+ ### How to use
27
+
28
+ Here is how to use this model:
29
+
30
+ ```python
31
+ from transformers import RobertaTokenizer, T5ForConditionalGeneration
32
+
33
+ tokenizer = RobertaTokenizer.from_pretrained('Salesforce/codet5-base')
34
+ model = T5ForConditionalGeneration.from_pretrained('Salesforce/codet5-base')
35
+
36
+ text = "def greet(user): print(f'hello <extra_id_0>!') </s>"
37
+ inputs = tokenizer(text, return_tensors="pt").input_ids
38
+
39
+ # simply generate a single sequence
40
+ generated_ids = model.generate(input_ids, max_length=8)
41
+ print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
42
+ # this prints {user.name}
43
+
44
+ # or, generating 20 sequences with maximum length set to 10
45
+ outputs = model.generate(input_ids=input_ids,
46
+ num_beams=200, num_return_sequences=20,
47
+ max_length=10)
48
+
49
+ _0_index = text.index('<extra_id_0>')
50
+ _result_prefix = text[:_0_index]
51
+ _result_suffix = text[_0_index+12:] # 12 is the length of <extra_id_0>
52
+
53
+ def _filter(output, end_token='<extra_id_1>'):
54
+ # The first token is <pad> (indexed at 0), the second token is <s> (indexed at 1)
55
+ # and the third token is <extra_id_0> (indexed at 32099)
56
+ # So we only decode from the fourth generated id
57
+ _txt = tokenizer.decode(output[3:], skip_special_tokens=False, clean_up_tokenization_spaces=False)
58
+ if end_token in _txt:
59
+ _end_token_index = _txt.index(end_token)
60
+ return _result_prefix + _txt[:_end_token_index] + _result_suffix
61
+ else:
62
+ return _result_prefix + _txt + _result_suffix
63
+
64
+ results = list(map(_filter, outputs))
65
+ print(results)
66
+ # this prints:
67
+ #["def greet(user): print(f'hello {user.name} {user!') </s>",
68
+ # "def greet(user): print(f'hello {user.username} {user!') </s>",
69
+ # "def greet(user): print(f'hello {user.name}: {user!') </s>",
70
+ # "def greet(user): print(f'hello {user}') print(f!') </s>",
71
+ # "def greet(user): print(f'hello {user.name} �!') </s>",
72
+ # "def greet(user): print(f'hello {user}') print ( f!') </s>",
73
+ # "def greet(user): print(f'hello {user.username}: {user!') </s>",
74
+ # "def greet(user): print(f'hello {user}' ) print(f!') </s>",
75
+ # "def greet(user): print(f'hello {user.username} �!') </s>",
76
+ # "def greet(user): print(f'hello {user.name}, {user!') </s>",
77
+ # "def greet(user): print(f'hello {user.login} {user!') </s>",
78
+ # "def greet(user): print(f'hello {user} →!') </s>",
79
+ # "def greet(user): print(f'hello {user}!') print(!') </s>",
80
+ # "def greet(user): print(f'hello {user.name} ({user!') </s>",
81
+ # "def greet(user): print(f'hello {user.email} {user!') </s>",
82
+ # "def greet(user): print(f'hello {user}!') print (!') </s>",
83
+ # "def greet(user): print(f'hello {user.username}, {user!') </s>",
84
+ # "def greet(user): print(f'hello {user}' ) print ( f!') </s>",
85
+ # "def greet(user): print(f'hello {user.nickname} {!') </s>",
86
+ # "def greet(user): print(f'hello {user} {user.name!') </s>"]
87
+ ```
88
+
89
+ ## Training data
90
+
91
+ The CodeT5 model was pretrained on CodeSearchNet [Husain et al., 2019](https://arxiv.org/abs/1909.09436). Additionally, the authors collected two datasets of C/CSharp from [BigQuery1](https://console.cloud.google.com/marketplace/details/github/github-repos) to ensure that all downstream tasks have overlapped programming languages with the pre-training data. In total, around 8.35 million instances are used for pretraining.
92
+
93
+ ## Training procedure
94
+
95
+ ### Preprocessing
96
+
97
+ This model uses a code-specific BPE (Byte-Pair Encoding) tokenizer. One can prepare text (or code) for the model using RobertaTokenizer, with the files from this repository.
98
+
99
+ ## Evaluation results
100
+
101
+ For evaluation results on several downstream benchmarks, we refer to the paper.
102
+
103
+ ### BibTeX entry and citation info
104
+
105
+ ```bibtex
106
+ @misc{wang2021codet5,
107
+ title={CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation},
108
+ author={Yue Wang and Weishi Wang and Shafiq Joty and Steven C. H. Hoi},
109
+ year={2021},
110
+ eprint={2109.00859},
111
+ archivePrefix={arXiv},
112
+ primaryClass={cs.CL}
113
+ }
114
+ ```