TOSBert

TOSBert is a fine-tuned BERT model for sequence classification tasks. It is trained on a custom dataset for multi-label classification.

Model Details

• Model Name: TOSBert
• Model Architecture: BERT
• Framework: Hugging Face Transformers
• Model Type: Sequence Classification (Multi-label Classification)

Dataset

The model is trained on the online_terms_of_service dataset hosted on Hugging Face. This dataset consists of text sequences extracted from various online terms of service documents. Each sequence is labeled with multiple categories related to legal and privacy terms.

Training

The model was fine-tuned using the following parameters:

• Number of Epochs: 3
• Batch Size: 16 (both for training and evaluation)
• Warmup Steps: 500
• Weight Decay: 0.01
• Learning Rate: Automatically adjusted

Usage

Installation

To use this model, you need to install the transformers library from Hugging Face:

pip install transformers

Loading the Model

You can load the model using the following code:

from transformers import BertForSequenceClassification, BertTokenizer

model_name = "CodeHima/TOSBert"
model = BertForSequenceClassification.from_pretrained(model_name)
tokenizer = BertTokenizer.from_pretrained(model_name)

Inference

Here is an example of how to use the model for inference:

from transformers import pipeline

classifier = pipeline("text-classification", model=model, tokenizer=tokenizer, return_all_scores=True)

text = "Your input text here"
predictions = classifier(text)

print(predictions)

Training Script

Below is an example script used for training the model:

from transformers import Trainer, TrainingArguments, BertForSequenceClassification, BertTokenizer
import torch
from sklearn.metrics import accuracy_score, precision_recall_fscore_support

# Define the model
model_name = "bert-base-uncased"
model = BertForSequenceClassification.from_pretrained(model_name, num_labels=3)

# Define the tokenizer
tokenizer = BertTokenizer.from_pretrained(model_name)

# Load your dataset
# train_dataset and eval_dataset should be instances of torch.utils.data.Dataset
# Example: train_dataset = YourDataset(train_data)

# Define training arguments
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=10,
    eval_strategy="epoch"
)

# Custom data collator to convert labels to floats
def data_collator(features):
    batch = {}
    first = features[0]
    if 'label' in first and first['label'] is not None:
        dtype = torch.float32
        batch['labels'] = torch.tensor([f['label'] for f in features], dtype=dtype)
    for k, v in first.items():
        if k != 'label' and v is not None and not isinstance(v, str):
            batch[k] = torch.stack([f[k] for f in features])
    return batch

# Define the compute metrics function
def compute_metrics(pred):
    labels = pred.label_ids
    preds = (pred.predictions > 0.5).astype(int)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='micro')
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

# Initialize the Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    compute_metrics=compute_metrics,
    data_collator=data_collator
)

# Train the model
trainer.train()

Evaluation

To evaluate the model on the validation set, you can use the following code:

results = trainer.evaluate()
print(results)

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Citation

If you use this model in your research, please cite it as follows:

@misc{TOSBert,
  author = {Himanshu Mohanty},
  title = {TOSBert: Fine-tuned BERT model for multi-label classification},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face Model Hub},
  howpublished = {\url{https://huggingface.co/CodeHima/TOSBert}}
}

Acknowledgements

This project uses the Hugging Face Transformers library. Special thanks to the developers and contributors of this library.