## A Lossless Syntax Tree Generator with Zero-shot Error Correction

- We follow [jam](https://huggingface.co/apcl/jam)'s pretraining procedure and use the same data to pretrain except we also use srcml to pretrain the models.
- In the finetuning stage, we finetune our models for 3 epochs.
- Our [GitHub repo](https://github.com/apcl-research/autorepair) contains the code for reproduction using the same [data](https://huggingface.co/datasets/apcl/autorepair).


## Pretrained model parameters
| Hyperparameter | Description | Value |
| ----------- | ----------- |------------|
|e | embedding dimensions              | 1024 |		 
|L | number of layers 			  		| 24 | 		 
|h | attention heads             		| 16 |		 
|c | block size / context length       | 256 |  		 
|b | batch size                        | 4  | 		 
|a | accumulation steps				| 32 |		 	 
|r | learning rate                     | 3e-5 |		 
|y | weight decay						| 1e-5 |
|iter | iterations						| 570000  |

## Model files

| Filename | Description |
| ------- | ------- |
|ckpt.pt|A model file for finetuning|
|ckpt_base.pt | A model file for generating syntax tree with the error correction in zero-shot setting|
|ckpt_finetune.pt | A model finetuned with the syntatic error dataset |

- Note that you can adjust the batch size and accumulation steps based on your GPU memory. But, the batch size * accumulation steps should be 128.
- If you finetune your models with multiple GPUs, you can turn down accumulation steps. For example, if you finetune with 2 GPUs, you will need to half the accumulation steps.