DOI: 10.5281/zenodo.14911062
title: Full Experimental Results for liBERTa: Browser-based Privacy Policy Classification

description: Introduction

This record provides a comprehensive data set and supporting materials related to a browser extension for real-time privacy policy classification. The contents include:



Privacy Policy Classification Performance: Detailed results from training and evaluating various Transformer-based models (e.g., TinyBERT, BERT, ELECTRA, DeBERTa, ALBERT) on the OPP-115 Corpus. The data set features training and validation loss histories, performance metrics (accuracy, precision, recall, and F1 scores), per-class results, and confusion matrices.

Web Browser Extension Performance: Benchmark data measuring key operational stages of the extension, such as URL retrieval, text segmentation, tokenization, and model inference. Performance is reported across multiple trials to assess real-world latency and efficiency.

Bayesian Search Results: A summary of the optimal hyperparameter configurations obtained through Bayesian optimization, detailing the settings that maximize macro-F1 scores for each model.

Data Splits: An explanation of the contiguous data splitting strategy used to partition the OPP-115 corpus into training, validation, and test sets, along with the mapping between global and local sample indices.


Privacy Policy Classification Performance

The classification results for each model are provided in the file cls_results.json. These results correspond to the best configuration identified through a Bayesian optimization search (see below), which was conducted to maximize the macro-F1 score on the validation set. The hyperparameter tuning process ensured that the reported models represent the optimal balance between performance and complexity. The JSON file contains a list of objects, where each represents the outcome of training and evaluating a specific model configuration. The structure of each object is described below:



model: The name of the model (e.g., TinyBERT, BERT, ELECTRA).

size: The size of the model (e.g., small, base, large).

target: A list of classification categories the model was trained to predict. This field remains consistent across all objects and is provided for completeness.

class_weights: The weight assigned to each class during training to counter class imbalance. The \(i\)-th weight corresponds to the \(i\)-th class in the target list and is used in the loss computation. This field remains consistent across all objects, as all models were trained on the same data, and is provided for completeness.

train_losses: A list representing the loss values computed on the training set after each epoch, capturing the model's learning progress.

val_losses: A list representing the validation loss values computed after each training epoch.

val_metrics: Contains validation metrics recorded after each epoch. Metrics follow the naming convention: <avergaing-method>-<metric-name>, where:



<averaging-method> can assume micro, macro, weighted values.

<metric-name> can be accuracy, precision, recall, f1.



best_epoch: Denotes the epoch at which the model achieved the highest macro-F1 score on the validation set. The first epoch is indexed as 1. The final model corresponds to this epoch.

test_metrics: This object summarizes the model's final performance on the test set, evaluated at the best_epoch. It contains:



Aggregated metrics: similar to val_metrics, it includes micro, macro and weighted averages for accuracy, precision, recall, and F1-score.

Non-aggregated metrics: provides per-class values for accuracy, precision, recall, and f1. The ordering of these metrics corresponds to the order in the target list, i.e., the \(i\)-th element in any of these lists corresponds to the \(i\)-th category.

confusion_matrix: the multi-label confusion matrix for the test set. It is represented \(i\)-th element is a \(2\times 2\) binary confusion matrix, corresponding to the \(i\)-th class in the target list, of the form: $$ [[C_{0,0}, C_{0,1}],$$ $$ [C_{1,0}, C_{1,1}]] $$ where: 



\(C_{0,0}\) are the True Negatives.

\(C_{0,1}\) are the False Positives.

\(C_{1,0}\) are the False Negatives.

\(C_{1,1}\) are the True Positives.






Web Browser Extension Performance

The Web Browser Extension Performance for privacy policy classification are stored in ext_results.json. The structure of each object is described below:



model: The name of the Transformer model used for classification (e.g., TinyBERT, BERT, ELECTRA).

size: The size of the corresponding model (e.g., small, base, large).

urls: The list of tested domains. The list contains 5 lists, each corresponding to an attempt. Thus, \(\text{url}_{i,j}\) corresponds to the \(j\)-th tested domain at the \(i\)-th trial, with \(i=1..5\). This field remains consistent across all objects and is provided for completeness.

segments: The number of text segments extracted from the privacy policy page during each trial. \(s_{i,j}\) corresponds to the number of segments extracted from the \(j\)-th domain in the \(i\)-th trial, with \(i=1..5\). Each element directly corresponds to the same index in the urls field.

url_extraction: The time required (in milliseconds) to extract the privacy policy URL from the active browser tab. \(t_{i,j}\) represents the extraction time for the \(j\)-th domain in the \(i\)-th trial. Each element directly corresponds to the same index in the urls field.

segmentation: 

tokenization: The time required (in milliseconds) to tokenize the extracted privacy policy text for classification. \(\tau_{i,j}\) represents the tokenization time for the \(j\)-th domain in the \(i\)-th trial. Each element directly corresponds to the same index in the urls field.

inference: The time required (in milliseconds) to perform classification on the tokenized privacy policy segments using the specified model. \(\iota_{i,j}\) represents the inference time for the \(j\)-th domain in the \(i\)-th trial. Each element directly corresponds to the same index in the urls field.


📅 Test Dates



Start Date: February 20, 2025.

End Date: February 22, 2025.

Testing Environment: Google Chrome 131.0.6778.69 on Windows 11, NVIDIA RTX 3060 Gaming Edition GPU (12 GB VRAM), AMD Ryzen 7 5800X CPU (8 physical cores, 16 threads) @ 3.8 GHz, 64 GB DDR4 RAM, 2TB NVMe M.2 SSD (PCIe Gen 3 x4).

Test Conditions: All performance tests were conducted under identical network conditions and hardware configurations.
description: Bayesian Search Results

This section details the outcomes of Bayesian optimization used for hyperparameter tuning. It includes the best hyperparameter configurations.

The hyperparameters are defined as follows:



\(b\): Batch size used during training, $b \sim \mathcal{U}\{1, 8, 16, 32 \}$.

\(\eta_1\): Learning rate for the Transformer-based model, $\eta_1 \sim \log\mathcal{U}\left( 1 \times 10^{-5}, 5\times 10^{-4} \right)$.

\(\lambda_1\): Weight decay for the Transformer-based model, $\lambda_1 \sim \log\mathcal{U}\left( 1 \times 10^{-5}, 5\times 10^{-4} \right)$.

\(\eta_2\): Learning rate for the classification head, $\eta_2 \sim \log\mathcal{U}\left( 1 \times 10^{-3}, 1 \times 10^{-2} \right)$.

\(\lambda_2\): Weight decay for the classification head, $\lambda_2 \sim \log\mathcal{U}\left( 1 \times 10^{-5}, 5\times 10^{-4} \right)$.

\(d\): Dropout probability applied during training, $d \sim \mathcal{U} \{ 0.0, 0.1, 0.2, 0.3, 0.4, 0.5 \}$.


where $\mathcal{U}$ denotes a uniform distribution and $\log\mathcal{U}$ denotes a logarithmically uniform distribution.




Model
\(\boldsymbol{b}\)
\(\boldsymbol{\eta_1}\)
\(\boldsymbol{\lambda_1}\)
\(\boldsymbol{\eta_2}\)
\(\boldsymbol{\lambda_2}\)
\(\boldsymbol{d}\)


TinyBERT base
16
1.058e-04
1.027e-04
6.721e-03
5.901e-05
0.20


TinyBERT large
16
8.644e-05
1.136e-04
5.815e-03
4.208e-05
0.20


BERT base
8
3.876e-05
7.357e-05
1.685e-03
8.772e-05
0.00


BERT large
32
3.350e-05
1.371e-05
1.155e-03
4.057e-04
0.10


ELECTRA small
8
5.310e-05
1.085e-04
5.489e-03
4.083e-05
0.30


ELECTRA base
1
1.002e-05
2.624e-05
1.541e-03
1.429e-04
0.20


ELECTRA large
16
1.523e-05
8.097e-05
2.436e-03
4.902e-04
0.30


DeBERTa xsmall
8
3.646e-05
3.889e-05
2.337e-03
1.107e-05
0.30


DeBERTa small
32
3.043e-05
3.326e-04
3.948e-03
1.367e-04
0.00


DeBERTa base
8
3.876e-05
7.357e-05
1.685e-03
8.772e-05
0.00


ALBERT base
8
3.876e-05
7.357e-05
1.685e-03
8.772e-05
0.00


ALBERT large
8
1.001e-05
4.295e-04
9.753e-03
1.098e-04
0.20




 
description: Data Splits

This section outlines the data splitting strategy adopted for training, validation, and testing. The data_splits.json file includes two main keys: splits and domains:



domains: This list specifies the ordered sequence of files from the OPP-115 corpus as they were processed. The ordering in domains directly aligns with the indexing in splits, ensuring that each index accurately references its corresponding file in the data set.

splits: This object comprises three lists:



train: Indices of samples allocated to the training set.

val: Indices of samples allocated to the validation set.

test: Indices of samples allocated to the test set.




The indices are intended to be interpreted contiguously, ensuring a coherent and sequential mapping across the entire data set. Formally, the \(\boldsymbol{i}\)-th sample corresponds to:

$$x_i =D_j^{(k)}$$

where:



\(x_i\) is the \(i\)-th sample in the overall data set.

\(D_j\) is the \(j\)-th domain (file) from the domains list.

\(k\) is the local index of the sample within the domain \(D_j\).


The relationship between \(i\), \(j\) and \(k\) can be expressed as:

$$i = \sum_{m=0}^{d-1} |X_m| + k$$

where:



\(X_m\) denotes the array of segments for the \(d\)-th domain;

\(|X_m|\) represents the number of segments in the \(m\)-th domain.

\(k\) is the local index of the segment wihin \(X_d\).




For example:

- \( x_0 \) is the first segment for the privacy policy of "sci-news.com".

- \( x_{12} \) is the last segment for the same privacy policy (in OPP-115, it has a length of 13 segments).

- \( x_{16} \) corresponds to the 4-th sample (\( k=3 \)) of the domain "redorbit.com", as the total number of segments in the previous domain(s) sums up to 13, etc.