Model Card: POLLCHECK/Pollcheck-llama3-news-classifier

Model Details

Model Name: POLLCHECK/Pollcheck-llama3-news-classifier

Model Description: This is a fine-tuned llama3 model for news classification e.g. "biased" or "unbiased". In this particular task, the term 'biased' represents disinformation, propaganda, loaded language, negative associations, generalization, harm, hatred, satire whereas 'unbiased' represents real news without the spread of misinformation, disinformation, and propaganda. The model can be used to identify potential bias in text, which is useful for applications in media analysis, content moderation, and research on bias in written communication.

Base Model: "meta-llama/Meta-Llama-3-8B-Instruct"

Fine-tuned Dataset: The model was fine-tuned on a custom dataset annotated for bias detection, particularly, news articles related to politics. Details of the dataset and the fine-tuning process are available upon request.

Labels: -0 or biased (fake news)

• 1 or unbiased (real news)

Intended Use

This model is intended for use in identifying biased in text. Users can input a piece of text and receive a prediction indicating whether the text is biased or unbiased.

How to Use

To use this model for inference, follow the steps below:

Inference Code

import torch
from trl import setup_chat_format
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load the fine-tuned model and tokenizer
model_name = "POLLCHECK/Pollcheck-llama3-news-classifier"  # Change this to the path of your fine-tuned model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    return_dict=True,
    low_cpu_mem_usage=True,
    #torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)


# make the model and tokenizer to chat formet
model, tokenizer = setup_chat_format(model, tokenizer)

###

instruction=f"""You are a news classifier AI assistant. You are given with the headline and news article body.
Your task is to read the headline and news articles, and classify the articles as biased on unbiased. Also provide the confidence score for your labels.In this particular task, the term 'biased' represents disinformation, propaganda, loaded language, negative associations, generalization, harm, hatred, satire
                whereas 'unbiased' represents real news without spread of misinformation, disinformation, and propaganda."""
headline="<Headline of the new article>"
article="<Article text body>"                
messages = [
  {"role": "user", "content": f"""{instruction}\nHeadline: {headline}\narticle: {article}\
  Return you answers in the following format.
  1. Labels: [biased/unbiased]
  2. Confidence: """}]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = tokenizer(prompt, return_tensors='pt', padding=True, truncation=True).to("cuda")
terminators = [tokenizer.eos_token_id,tokenizer.convert_tokens_to_ids("<|eot_id|>")]

outputs = model.generate(**inputs, max_new_tokens=30, eos_token_id=terminators, do_sample=True, temperature=0.7, top_p=0.9,)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Limitations and Bias

• Dataset Bias: The model's performance is highly dependent on the quality and diversity of the fine-tuning dataset. Biases present in the dataset will affect the model's predictions.
• Context: The model may not perform well on texts that are out of the distribution of the training data or on texts that require nuanced understanding of context.

Ethical Considerations

• Fairness: Ensure that the model is used in a fair and unbiased manner. Regularly evaluate the model's performance and address any biases that may arise.
• Transparency: Be transparent about the model's limitations and the potential for false positives and false negatives.
• Accountability: Users are responsible for the decisions made based on the model's predictions and should consider multiple sources of information when making important decisions.

Contact Information

For questions, comments, or suggestions, please contact Emrul Hasan at emrul.hasan@vectorinstitute.ai.