Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,225 @@
|
|
1 |
-
---
|
2 |
-
license: cc-by-nc-sa-4.0
|
3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
license: cc-by-nc-sa-4.0
|
3 |
+
datasets:
|
4 |
+
- QCRI/LlamaLens-English
|
5 |
+
- QCRI/LlamaLens-Arabic
|
6 |
+
- QCRI/LlamaLens-Hindi
|
7 |
+
language:
|
8 |
+
- ar
|
9 |
+
- en
|
10 |
+
- hi
|
11 |
+
base_model:
|
12 |
+
- meta-llama/Llama-3.1-8B-Instruct
|
13 |
+
pipeline_tag: text-generation
|
14 |
+
tags:
|
15 |
+
- Social-Media
|
16 |
+
- Hate-Speech
|
17 |
+
- Summarization
|
18 |
+
- offensive-language
|
19 |
+
- News-Genre
|
20 |
+
metrics:
|
21 |
+
- accuracy
|
22 |
+
- f1
|
23 |
+
- rouge
|
24 |
+
---
|
25 |
+
# LlamaLens: Specialized Multilingual LLM forAnalyzing News and Social Media Content
|
26 |
+
|
27 |
+
## Overview
|
28 |
+
LlamaLens is a specialized multilingual LLM designed for analyzing news and social media content. It focuses on 18 NLP tasks, leveraging 52 datasets across Arabic, English, and Hindi.
|
29 |
+
|
30 |
+
<p align="center">
|
31 |
+
<picture>
|
32 |
+
<img width="352" alt="capablities_tasks_datasets" src="./llamalens-avatar.png">
|
33 |
+
</picture>
|
34 |
+
</p>
|
35 |
+
|
36 |
+
## Dataset
|
37 |
+
The model was trained on the [LlamaLens dataset](https://huggingface.co/collections/QCRI/llamalens-672f7e0604a0498c6a2f0fe9).
|
38 |
+
|
39 |
+
## To Replicate the Experiments
|
40 |
+
The code to replicate the experiments is available on [GitHub](https://github.com/firojalam/LlamaLens).
|
41 |
+
|
42 |
+
|
43 |
+
## Model Inference
|
44 |
+
|
45 |
+
To utilize the LlamaLens model for inference, follow these steps:
|
46 |
+
|
47 |
+
1. **Install the Required Libraries**:
|
48 |
+
|
49 |
+
Ensure you have the necessary libraries installed. You can do this using pip:
|
50 |
+
|
51 |
+
```bash
|
52 |
+
pip install transformers torch
|
53 |
+
```
|
54 |
+
2. **Load the Model and Tokenizer:**:
|
55 |
+
Use the transformers library to load the LlamaLens model and its tokenizer:
|
56 |
+
|
57 |
+
```python
|
58 |
+
from transformers import AutoModelForCausalLM, AutoTokenizer
|
59 |
+
|
60 |
+
# Define model path
|
61 |
+
MODEL_PATH = "QCRI/LlamaLens"
|
62 |
+
|
63 |
+
# Load model and tokenizer
|
64 |
+
device_map = "auto"
|
65 |
+
model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, device_map=device_map)
|
66 |
+
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, trust_remote_code=True)
|
67 |
+
tokenizer.pad_token = tokenizer.eos_token
|
68 |
+
|
69 |
+
```
|
70 |
+
3. **Prepare the Input:**:
|
71 |
+
Tokenize your input text:
|
72 |
+
```python
|
73 |
+
# Define task and input text
|
74 |
+
task = "classification" # Change to "summarization" for summarization tasks
|
75 |
+
instruction = (
|
76 |
+
"Analyze the text and indicate if it shows an emotion, then label it as joy, love, fear,"
|
77 |
+
" anger, sadness, or surprise. Return only the label without any explanation, justification, or additional text."
|
78 |
+
)
|
79 |
+
input_text = "I am not creating anything I feel satisfied with."
|
80 |
+
output_prefix = "Summary: " if task == "summarization" else "Label: "
|
81 |
+
|
82 |
+
# Define messages for chat-based prompt format
|
83 |
+
messages = [
|
84 |
+
{"role": "system", "content": "You are a social media expert providing accurate analysis and insights."},
|
85 |
+
{"role": "user", "content": f"{instruction}\nInput: {input_text}"},
|
86 |
+
{"role": "assistant", "content": output_prefix}
|
87 |
+
]
|
88 |
+
|
89 |
+
# Tokenize input
|
90 |
+
input_ids = tokenizer.apply_chat_template(
|
91 |
+
messages,
|
92 |
+
add_generation_prompt=False,
|
93 |
+
continue_final_message=True,
|
94 |
+
tokenize=True,
|
95 |
+
padding=True,
|
96 |
+
return_tensors="pt"
|
97 |
+
).to(model.device)
|
98 |
+
|
99 |
+
|
100 |
+
|
101 |
+
```
|
102 |
+
4. **Generate the Output:**:
|
103 |
+
Generate a response using the model:
|
104 |
+
```python
|
105 |
+
# Generate response
|
106 |
+
outputs = model.generate(
|
107 |
+
input_ids,
|
108 |
+
max_new_tokens=128,
|
109 |
+
do_sample=False,
|
110 |
+
eos_token_id=tokenizer.eos_token_id,
|
111 |
+
pad_token_id=tokenizer.eos_token_id,
|
112 |
+
temperature=0.001
|
113 |
+
)
|
114 |
+
|
115 |
+
# Decode and print response
|
116 |
+
response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
|
117 |
+
print(response)
|
118 |
+
```
|
119 |
+
|
120 |
+
## Results
|
121 |
+
|
122 |
+
Below, we present the performance of **L-Lens: LlamaLens** , where *"Eng"* refers to the English-instructed model and *"Native"* refers to the model trained with native language instructions. The results are compared against the SOTA (where available) and the Base: **Llama-Instruct 3.1 baseline**. The **Δ** (Delta) column indicates the difference between LlamaLens and the SOTA performance, calculated as (LlamaLens – SOTA).
|
123 |
+
|
124 |
+
---
|
125 |
+
|
126 |
+
## Arabic
|
127 |
+
|
128 |
+
| **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** |
|
129 |
+
|:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:|
|
130 |
+
| Attentionworthiness Detection | CT22Attentionworthy | W-F1 | 0.412 | 0.158 | 0.425 | 0.454 | 0.013 |
|
131 |
+
| Checkworthiness Detection | CT24_checkworthy | F1_Pos | 0.569 | 0.610 | 0.502 | 0.509 | -0.067 |
|
132 |
+
| Claim Detection | CT22Claim | Acc | 0.703 | 0.581 | 0.734 | 0.756 | 0.031 |
|
133 |
+
| Cyberbullying Detection | ArCyc_CB | Acc | 0.863 | 0.766 | 0.870 | 0.833 | 0.007 |
|
134 |
+
| Emotion Detection | Emotional-Tone | W-F1 | 0.658 | 0.358 | 0.705 | 0.736 | 0.047 |
|
135 |
+
| Emotion Detection | NewsHeadline | Acc | 1.000 | 0.406 | 0.480 | 0.458 | -0.520 |
|
136 |
+
| Factuality | Arafacts | Mi-F1 | 0.850 | 0.210 | 0.771 | 0.738 | -0.079 |
|
137 |
+
| Factuality | COVID19Factuality | W-F1 | 0.831 | 0.492 | 0.800 | 0.840 | -0.031 |
|
138 |
+
| Harmfulness Detection | CT22Harmful | F1_Pos | 0.557 | 0.507 | 0.523 | 0.535 | -0.034 |
|
139 |
+
| Hate Speech Detection | annotated-hatetweets-4-classes | W-F1 | 0.630 | 0.257 | 0.526 | 0.517 | -0.104 |
|
140 |
+
| Hate Speech Detection | OSACT4SubtaskB | Mi-F1 | 0.950 | 0.819 | 0.955 | 0.955 | 0.005 |
|
141 |
+
| News Categorization | ASND | Ma-F1 | 0.770 | 0.587 | 0.919 | 0.929 | 0.149 |
|
142 |
+
| News Categorization | SANADAkhbarona-news-categorization | Acc | 0.940 | 0.784 | 0.954 | 0.953 | 0.014 |
|
143 |
+
| News Categorization | SANADAlArabiya-news-categorization | Acc | 0.974 | 0.893 | 0.987 | 0.985 | 0.013 |
|
144 |
+
| News Categorization | SANADAlkhaleej-news-categorization | Acc | 0.986 | 0.865 | 0.984 | 0.982 | -0.002 |
|
145 |
+
| News Categorization | UltimateDataset | Ma-F1 | 0.970 | 0.376 | 0.865 | 0.880 | -0.105 |
|
146 |
+
| News Credibility | NewsCredibilityDataset | Acc | 0.899 | 0.455 | 0.935 | 0.933 | 0.036 |
|
147 |
+
| News Summarization | xlsum | R-2 | 0.137 | 0.034 | 0.129 | 0.130 | -0.009 |
|
148 |
+
| Offensive Language Detection | ArCyc_OFF | Ma-F1 | 0.878 | 0.489 | 0.877 | 0.879 | -0.001 |
|
149 |
+
| Offensive Language Detection | OSACT4SubtaskA | Ma-F1 | 0.905 | 0.782 | 0.896 | 0.882 | -0.009 |
|
150 |
+
| Propaganda Detection | ArPro | Mi-F1 | 0.767 | 0.597 | 0.747 | 0.731 | -0.020 |
|
151 |
+
| Sarcasm Detection | ArSarcasm-v2 | F1_Pos | 0.584 | 0.477 | 0.520 | 0.542 | -0.064 |
|
152 |
+
| Sentiment Classification | ar_reviews_100k | F1_Pos | -- | 0.681 | 0.785 | 0.779 | -- |
|
153 |
+
| Sentiment Classification | ArSAS | Acc | 0.920 | 0.603 | 0.800 | 0.804 | -0.120 |
|
154 |
+
| Stance Detection | stance | Ma-F1 | 0.767 | 0.608 | 0.926 | 0.881 | 0.159 |
|
155 |
+
| Stance Detection | Mawqif-Arabic-Stance-main | Ma-F1 | 0.789 | 0.764 | 0.853 | 0.826 | 0.065 |
|
156 |
+
| Subjectivity Detection | ThatiAR | f1_pos | 0.800 | 0.562 | 0.441 | 0.383 | -0.359 |
|
157 |
+
|
158 |
+
---
|
159 |
+
|
160 |
+
## English
|
161 |
+
|
162 |
+
| **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** |
|
163 |
+
|:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:|
|
164 |
+
| Checkworthiness Detection | CT24_checkworthy | f1_pos | 0.753 | 0.404 | 0.942 | 0.942 | 0.189 |
|
165 |
+
| Claim Detection | claim-detection | Mi-F1 | -- | 0.545 | 0.864 | 0.889 | -- |
|
166 |
+
| Cyberbullying Detection | Cyberbullying | Acc | 0.907 | 0.175 | 0.836 | 0.855 | -0.071 |
|
167 |
+
| Emotion Detection | emotion | Ma-F1 | 0.790 | 0.353 | 0.803 | 0.808 | 0.013 |
|
168 |
+
| Factuality | News_dataset | Acc | 0.920 | 0.654 | 1.000 | 1.000 | 0.080 |
|
169 |
+
| Factuality | Politifact | W-F1 | 0.490 | 0.121 | 0.287 | 0.311 | -0.203 |
|
170 |
+
| News Categorization | CNN_News_Articles_2011-2022 | Acc | 0.940 | 0.644 | 0.970 | 0.970 | 0.030 |
|
171 |
+
| News Categorization | News_Category_Dataset | Ma-F1 | 0.769 | 0.970 | 0.824 | 0.520 | 0.055 |
|
172 |
+
| News Genre Categorisation | SemEval23T3-subtask1 | Mi-F1 | 0.815 | 0.687 | 0.241 | 0.253 | -0.574 |
|
173 |
+
| News Summarization | xlsum | R-2 | 0.152 | 0.074 | 0.182 | 0.181 | 0.030 |
|
174 |
+
| Offensive Language Detection | Offensive_Hateful_Dataset_New | Mi-F1 | -- | 0.692 | 0.814 | 0.813 | -- |
|
175 |
+
| Offensive Language Detection | offensive_language_dataset | Mi-F1 | 0.994 | 0.646 | 0.899 | 0.893 | -0.095 |
|
176 |
+
| Offensive Language and Hate Speech | hate-offensive-speech | Acc | 0.945 | 0.602 | 0.931 | 0.935 | -0.014 |
|
177 |
+
| Propaganda Detection | QProp | Ma-F1 | 0.667 | 0.759 | 0.963 | 0.973 | 0.296 |
|
178 |
+
| Sarcasm Detection | News-Headlines-Dataset-For-Sarcasm-Detection | Acc | 0.897 | 0.668 | 0.936 | 0.947 | 0.039 |
|
179 |
+
| Sentiment Classification | NewsMTSC-dataset | Ma-F1 | 0.817 | 0.628 | 0.751 | 0.748 | -0.066 |
|
180 |
+
| Subjectivity Detection | clef2024-checkthat-lab | Ma-F1 | 0.744 | 0.535 | 0.642 | 0.628 | -0.102 |
|
181 |
+
|
|
182 |
+
|
183 |
+
---
|
184 |
+
|
185 |
+
## Hindi
|
186 |
+
|
187 |
+
| **Task** | **Dataset** | **Metric** | **SOTA** | **Base** | **L-Lens-Eng** | **L-Lens-Native** | **Δ (L-Lens (Eng) - SOTA)** |
|
188 |
+
|:----------------------------------:|:--------------------------------------------:|:----------:|:--------:|:---------------------:|:---------------------:|:--------------------:|:------------------------:|
|
189 |
+
| Factuality | fake-news | Mi-F1 | -- | 0.759 | 0.994 | 0.993 | -- |
|
190 |
+
| Hate Speech Detection | hate-speech-detection | Mi-F1 | 0.639 | 0.750 | 0.963 | 0.963 | 0.324 |
|
191 |
+
| Hate Speech Detection | Hindi-Hostility-Detection-CONSTRAINT-2021 | W-F1 | 0.841 | 0.469 | 0.753 | 0.753 | -0.088 |
|
192 |
+
| Natural Language Inference | Natural Language Inference | W-F1 | 0.646 | 0.633 | 0.568 | 0.679 | -0.078 |
|
193 |
+
| News Summarization | xlsum | R-2 | 0.136 | 0.078 | 0.171 | 0.170 | 0.035 |
|
194 |
+
| Offensive Language Detection | Offensive Speech Detection | Mi-F1 | 0.723 | 0.621 | 0.862 | 0.865 | 0.139 |
|
195 |
+
| Cyberbullying Detection | MC_Hinglish1 | Acc | 0.609 | 0.233 | 0.625 | 0.627 | 0.016 |
|
196 |
+
| Sentiment Classification | Sentiment Analysis | Acc | 0.697 | 0.552 | 0.647 | 0.654 | -0.050
|
197 |
+
|
198 |
+
## Paper
|
199 |
+
For an in-depth understanding, refer to our paper: [**LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content**](https://arxiv.org/pdf/2410.15308).
|
200 |
+
|
201 |
+
|
202 |
+
|
203 |
+
|
204 |
+
# License
|
205 |
+
This model is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0).
|
206 |
+
|
207 |
+
|
208 |
+
# Citation
|
209 |
+
Please cite [our paper](https://arxiv.org/pdf/2410.15308) when using this model:
|
210 |
+
|
211 |
+
```
|
212 |
+
@article{kmainasi2024llamalensspecializedmultilingualllm,
|
213 |
+
title={LlamaLens: Specialized Multilingual LLM for Analyzing News and Social Media Content},
|
214 |
+
author={Mohamed Bayan Kmainasi and Ali Ezzat Shahroor and Maram Hasanain and Sahinur Rahman Laskar and Naeemul Hassan and Firoj Alam},
|
215 |
+
year={2024},
|
216 |
+
journal={arXiv preprint arXiv:2410.15308},
|
217 |
+
volume={},
|
218 |
+
number={},
|
219 |
+
pages={},
|
220 |
+
url={https://arxiv.org/abs/2410.15308},
|
221 |
+
eprint={2410.15308},
|
222 |
+
archivePrefix={arXiv},
|
223 |
+
primaryClass={cs.CL}
|
224 |
+
}
|
225 |
+
```
|