Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
## A Lossless Syntax Tree Generator with Zero-shot Error Correction
|
2 |
+
|
3 |
+
- We follow [jam](https://huggingface.co/apcl/jam)'s pretraining procedure and use the same data to pretrain except we also use srcml to pretrain the models.
|
4 |
+
- In the finetuning stage, we finetune our models for 3 epochs.
|
5 |
+
- Our [GitHub repo](https://github.com/apcl-research/autorepair) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/autorepair).
|
6 |
+
|
7 |
+
|
8 |
+
## Pretrained model parameters
|
9 |
+
| Hyperparameter | Description | Value |
|
10 |
+
| ----------- | ----------- |------------|
|
11 |
+
|e | embedding dimensions | 1024 |
|
12 |
+
|L | number of layers | 24 |
|
13 |
+
|h | attention heads | 16 |
|
14 |
+
|c | block size / context length | 256 |
|
15 |
+
|b | batch size | 4 |
|
16 |
+
|a | accumulation steps | 32 |
|
17 |
+
|r | learning rate | 3e-5 |
|
18 |
+
|y | weight decay | 1e-5 |
|
19 |
+
|iter | iterations | 570000 |
|
20 |
+
|
21 |
+
- Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128.
|
22 |
+
- If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps.
|
23 |
+
|