Model Card: POLLCHECK/BERT-classifier

Model Details

Model Name: POLLCHECK/BERT-classifier

Model Description: This BERT-based model is fine-tuned for the task of classifying text as either "biased/ fake" or "unbiased/real". It can be used to identify potential bias in text, which is useful for applications in media analysis, content moderation, and research on bias in written communication.

Base Model: BERT (Bidirectional Encoder Representations from Transformers)

Fine-tuned Dataset: The model was fine-tuned on a custom dataset annotated for bias detection. Details of the dataset and the fine-tuning process are available upon request.

Labels: -0 or biased (fake news)

• 1 or unbiased (real news)

Intended Use

This model is intended for use in identifying bias in text. Users can input a piece of text and receive a prediction indicating whether the text is biased or unbiased.

Class-wise Performance Metrics

Class	Prec	Recall	F1
Biased	0.92	0.95	0.94
Unbiased	0.95	0.92	0.93
Overall	0.93	0.93	0.93

How to Use

To use this model for inference, follow the steps below:

Inference Code

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Load the fine-tuned model and tokenizer
model_name = "POLLCHECK/BERT-classifier"  # Change this to the path of your fine-tuned model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)

# Define the labels
labels = ["biased", "unbiased"]

# Sample input texts
texts = [
    "Religious Extremists Threaten Our Way of Life.",
    "Public Health Officials are working.",
    "The new healthcare policy aims to provide affordable healthcare to all citizens, with a focus on preventive care.",
    "Environmental activists argue that the government's refusal to sign the climate agreement is a clear indication of its disregard for the environment."
]

# Process each text
for text in texts:
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    with torch.no_grad():
        outputs = model(**inputs)
        logits = outputs.logits
        probabilities = torch.softmax(logits, dim=-1)  # Convert logits to probabilities
        biased_prob = probabilities[0][0].item()  # Probability for the "biased" class (index 0)
        unbiased_prob = probabilities[0][1].item()  # Probability for the "unbiased" class (index 1)
        
    # Apply threshold
    predicted_label = "biased" if biased_prob > 0.5 else "unbiased"
    print(f"Text: {text}\nPredicted label: {predicted_label} (Biased Probability: {biased_prob:.2f}, Unbiased Probability: {unbiased_prob:.2f})\n")

Example Output

For the provided sample texts, the model might output:

Text: Religious Extremists Threaten Our Way of Life.
Predicted label: biased (Biased Probability: 0.95, Unbiased Probability: 0.05)

Text: Public Health Officials are working.
Predicted label: unbiased (Biased Probability: 0.10, Unbiased Probability: 0.90)

Text: The new healthcare policy aims to provide affordable healthcare to all citizens, with a focus on preventive care.
Predicted label: unbiased (Biased Probability: 0.20, Unbiased Probability: 0.80)

Text: Environmental activists argue that the government's refusal to sign the climate agreement is a clear indication of its disregard for the environment.
Predicted label: biased (Biased Probability: 0.70, Unbiased Probability: 0.30)

Check inference with these paths:

• Sample Data: News_Bias_Samples.csv
• Inference Script: inference-bert.py

Limitations and Bias

• Dataset Bias: The model's performance is highly dependent on the quality and diversity of the fine-tuning dataset. Biases present in the dataset will affect the model's predictions.
• Context: The model may not perform well on texts that are out of the distribution of the training data or on texts that require nuanced understanding of context.
• Threshold: The choice of threshold (0.5) for classifying a text as "biased" or "unbiased" is arbitrary and may need to be adjusted based on specific use cases and requirements.

Ethical Considerations

• Fairness: Ensure that the model is used in a fair and unbiased manner. Regularly evaluate the model's performance and address any biases that may arise.
• Transparency: Be transparent about the model's limitations and the potential for false positives and false negatives.
• Accountability: Users are responsible for the decisions made based on the model's predictions and should consider multiple sources of information when making important decisions.

Contact Information

For questions, comments, or suggestions, please contact Shaina Raza at shaina.raza@vectorinstitute.ai.