README.md

---
license: mit
language:
- en
library_name: transformers
---
# BERT Model for Software Engineering 

This repository was created within the scope of computer engineering undergraduate graduation project.
This research aims to perform an exploratory case study to determine the functional dimensions of user requirements or use cases for software projects.
In order to perform this task we created two models, SE-BERT and [SE-BERTurk](https://huggingface.co/burakkececi/bert-turkish-software-engineering).

# SE-BERT

SE-BERT is a BERT model trained for domain adaptation in a software engineering context.

We applied Masked Language Modeling (MLM), an unsupervised learning technique, for domain adaptation. MLM enhances the model understanding of domain-specific language by masking portions of the input text and training the model to predict the masked words based on the surrounding context.

## Stats
Created a bilingual [SE corpus](https://drive.google.com/file/d/1IgnJTaR2-pe889TdQZtYF8SKOH92mi1l/view?usp=drive_link) (166Mb) ➡️ [Descriptive stats of the corpus](https://docs.google.com/spreadsheets/d/1Xnn_xfu4tdCtWg-nQ8ce_LHe9F-g0BSmUxzTdi5g1r4/edit?usp=sharing) 
 * 166K entry = 886K sentence = 10M words
 * 156K training entry + 10K test entry
 * Each entry has a maximum length of 512 tokens

The final training corpus has a size of 166MB and 10.554.750 words.

## MLM Training (Domain Adaptation)
Used ``AdamW`` optimizer and set ``num_epochs = 1``, ``lr = 2e-5``, ``eps = 1e-8``
  * For T4 GPU ➡️ Set ``batch_size = 6`` (13.5Gb memory) 
  * For A100 GPU ➡️ Set ``batch_size = 50`` (37Gb memory) and ``fp16 = True``

**Perplexity**
 * ``6,673`` PPL for SE-BERT

### Evaluation Steps:
1) Calculate ``PPL`` (perplexity) on the test corpus (10K context with a maximum length of 512 tokens)
2) Calculate ``PPL`` (perplexity) on the requirement datasets
3) Evaluate performance on downstream tasks:
  * For size measurement ➡️ ``MAE``, ``MSE``, ``MMRE``, ``PRED(30)``, ``ACC``

## Usage

With Transformers >= 2.11 our SE-BERT uncased model can be loaded like:

```python
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("burakkececi/bert-software-engineering/model")
model = AutoModel.from_pretrained("burakkececi/bert-software-engineering/tokenizer")
```

# Huggingface model hub

All models are available on the [Huggingface model hub](https://huggingface.co/burakkececi).