Upload folder using huggingface_hub

README.md

@@ -0,0 +1,119 @@
+---
+license: mit
+arxiv: 2205.12424
+datasets:
+- code_x_glue_cc_defect_detection
+metrics:
+- accuracy
+- precision
+- recall
+- f1
+- roc_auc
+model-index:
+  - name: VulBERTa MLP
+    results:
+      - task:
+          type: defect-detection
+        dataset:
+          name: codexglue-devign
+          type: codexglue-devign
+        metrics:
+          - name: Accuracy
+            type: Accuracy
+            value: 64.71
+          - name: Precision
+            type: Precision
+            value: 64.80
+          - name: Recall
+            type: Recall
+            value: 50.76
+          - name: F1
+            type: F1
+            value: 56.93
+          - name: ROC-AUC
+            type: ROC-AUC
+            value: 71.02
+pipeline_tag: text-classification
+tags:
+- devign
+- defect detection
+- code
+---
+
+# VulBERTa MLP Devign
+## VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection
+
+![VulBERTa architecture](https://raw.githubusercontent.com/ICL-ml4csec/VulBERTa/main/VB.png)
+
+## Overview
+This model is the unofficial HuggingFace version of "[VulBERTa](https://github.com/ICL-ml4csec/VulBERTa/tree/main)" with an MLP classification head, trained on CodeXGlue Devign (C code), by Hazim Hanif & Sergio Maffeis (Imperial College London). I simplified the tokenization process by adding the cleaning (comment removal) step to the tokenizer and added the simplified tokenizer to this model repo as an AutoClass.
+
+> This paper presents presents VulBERTa, a deep learning approach to detect security vulnerabilities in source code. Our approach pre-trains a RoBERTa model with a custom tokenisation pipeline on real-world code from open-source C/C++ projects. The model learns a deep knowledge representation of the code syntax and semantics, which we leverage to train vulnerability detection classifiers. We evaluate our approach on binary and multi-class vulnerability detection tasks across several datasets (Vuldeepecker, Draper, REVEAL and muVuldeepecker) and benchmarks (CodeXGLUE and D2A). The evaluation results show that VulBERTa achieves state-of-the-art performance and outperforms existing approaches across different datasets, despite its conceptual simplicity, and limited cost in terms of size of training data and number of model parameters.
+
+## Usage
+**You must install libclang for tokenization.**
+
+```bash
+pip install libclang
+```
+
+Note that due to the custom tokenizer, you must pass `trust_remote_code=True` when instantiating the model.
+Example:
+```
+from transformers import pipeline
+pipe = pipeline("text-classification", model="claudios/VulBERTa-MLP-Devign", trust_remote_code=True, return_all_scores=True)
+pipe("static void filter_mirror_setup(NetFilterState *nf, Error **errp)\n{\n    MirrorState *s = FILTER_MIRROR(nf);\n    Chardev *chr;\n    chr = qemu_chr_find(s->outdev);\n    if (chr == NULL) {\n        error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,\n                  \"Device '%s' not found\", s->outdev);\n    qemu_chr_fe_init(&s->chr_out, chr, errp);")
+>> [[{'label': 'LABEL_0', 'score': 0.014685827307403088},
+  {'label': 'LABEL_1', 'score': 0.985314130783081}]]
+```
+
+***
+
+## Data
+We provide all data required by VulBERTa.  
+This includes:
+ - Tokenizer training data
+ - Pre-training data
+ - Fine-tuning data
+
+Please refer to the [data](https://github.com/ICL-ml4csec/VulBERTa/tree/main/data "data") directory for further instructions and details.
+
+## Models
+We provide all models pre-trained and fine-tuned by VulBERTa.  
+This includes:
+ - Trained tokenisers
+ - Pre-trained VulBERTa model (core representation knowledge)
+ - Fine-tuned VulBERTa-MLP and VulBERTa-CNN models
+
+Please refer to the [models](https://github.com/ICL-ml4csec/VulBERTa/tree/main/models "models") directory for further instructions and details.
+
+## How to use
+
+In our project, we uses Jupyterlab notebook to run experiments.  
+Therefore, we separate each task into different notebook:
+
+ - [Pretraining_VulBERTa.ipynb](https://github.com/ICL-ml4csec/VulBERTa/blob/main/Pretraining_VulBERTa.ipynb "Pretraining_VulBERTa.ipynb") - Pre-trains the core VulBERTa knowledge representation model using DrapGH dataset.
+ - [Finetuning_VulBERTa-MLP.ipynb](https://github.com/ICL-ml4csec/VulBERTa/blob/main/Finetuning_VulBERTa-MLP.ipynb "Finetuning_VulBERTa-MLP.ipynb") - Fine-tunes the VulBERTa-MLP model on a specific vulnerability detection dataset.
+ - [Evaluation_VulBERTa-MLP.ipynb](https://github.com/ICL-ml4csec/VulBERTa/blob/main/Evaluation_VulBERTa-MLP.ipynb "Evaluation_VulBERTa-MLP.ipynb") - Evaluates the fine-tuned VulBERTa-MLP models on testing set of a specific vulnerability detection dataset.
+ - [Finetuning+evaluation_VulBERTa-CNN](https://github.com/ICL-ml4csec/VulBERTa/blob/main/Finetuning%2Bevaluation_VulBERTa-CNN.ipynb "Finetuning+evaluation_VulBERTa-CNN.ipynb") - Fine-tunes VulBERTa-CNN models and evaluates it on a testing set of a specific vulnerability detection dataset.
+
+
+## Citation
+
+Accepted as conference paper (oral presentation) at the International Joint Conference on Neural Networks (IJCNN) 2022.  
+Link to paper: https://ieeexplore.ieee.org/document/9892280  
+
+
+```bibtex
+@INPROCEEDINGS{hanif2022vulberta,
+  author={Hanif, Hazim and Maffeis, Sergio},
+  booktitle={2022 International Joint Conference on Neural Networks (IJCNN)}, 
+  title={VulBERTa: Simplified Source Code Pre-Training for Vulnerability Detection}, 
+  year={2022},
+  volume={},
+  number={},
+  pages={1-8},
+  doi={10.1109/IJCNN55064.2022.9892280}
+  
+}
+```

config.json

@@ -0,0 +1,28 @@
+{
+  "_name_or_path": "VulBERTa-MLP-ReVeal",
+  "architectures": [
+    "RobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 1026,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.37.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 50000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ee6b84af692fad65a53e660f0e63b5c6ccf39beaf35488a297f2c5722cdecdfc
+size 499371608

special_tokens_map.json

@@ -0,0 +1,3 @@
+{
+  "pad_token": "<pad>"
+}

tokenization_vulberta.py

@@ -0,0 +1,51 @@
+from typing import List
+
+from tokenizers import NormalizedString, PreTokenizedString
+from tokenizers.pre_tokenizers import PreTokenizer
+from transformers import PreTrainedTokenizerFast
+
+try:
+    from clang import cindex
+except ModuleNotFoundError as e:
+    raise ModuleNotFoundError(
+        "VulBERTa Clang tokenizer requires `libclang`. Please install it via `pip install libclang`.",
+    ) from e
+
+
+class ClangPreTokenizer:
+    cidx = cindex.Index.create()
+
+    def clang_split(
+        self,
+        i: int,
+        normalized_string: NormalizedString,
+    ) -> List[NormalizedString]:
+        tok = []
+        tu = self.cidx.parse(
+            "tmp.c",
+            args=[""],
+            unsaved_files=[("tmp.c", str(normalized_string.original))],
+            options=0,
+        )
+        for t in tu.get_tokens(extent=tu.cursor.extent):
+            spelling = t.spelling.strip()
+            if spelling == "":
+                continue
+            tok.append(NormalizedString(spelling))
+        return tok
+
+    def pre_tokenize(self, pretok: PreTokenizedString):
+        pretok.split(self.clang_split)
+
+
+class VulBERTaTokenizer(PreTrainedTokenizerFast):
+    def __init__(
+        self,
+        *args,
+        **kwargs,
+    ):
+        super().__init__(
+            *args,
+            **kwargs,
+        )
+        self._tokenizer.pre_tokenizer = PreTokenizer.custom(ClangPreTokenizer())

tokenizer_config.json

@@ -0,0 +1,26 @@
+{
+  "added_tokens_decoder": {
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "max_length": 1024,
+  "model_max_length": 1024,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "stride": 0,
+  "tokenizer_class": "VulBERTaTokenizer",
+  "auto_map": {
+    "AutoTokenizer": ["tokenization_vulberta.VulBERTaTokenizer", null]
+  },
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first"
+}