Arabic Mini-ClimateGPT : A Climate Change and Sustainability Tailored Arabic LLM
Authors: Sahal Shaji Mullappilly
Abdelrahman M Shaker
Omkar Chakradhar Thawakar
Hisham Cholakkal
Rao Muhammad Anwer
Salman Khan
Fahad Khan
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Model
Climate change
Sustainability
Arabic NLP
TL;DR: An Open-source Climate change and Sustainability Tailored Arabic LLM

Abstract: Climate change is one of the most significant challenges we face together as a society. Creating awareness and educating policy makers the wide-ranging impact of climate change is an essential step towards a sustainable future. Recently, Large Language Models (LLMs) like ChatGPT and Bard have shown impressive conversational abilities and excel in a wide variety of NLP tasks. While these models are close-source, recently alternative open-source LLMs such as Stanford Alpaca and Vicuna have shown promising results. However, these open-source models are not specifically tailored for climate related domain specific information and also struggle to generate meaningful responses in other languages such as, Arabic. To this end, we propose a light-weight Arabic Mini-ClimateGPT that is built on an open-source LLM and is specifically fine-tuned on a conversational-style instruction tuning curated Arabic dataset Clima500-Instruct with over 500k instructions about climate change and sustainability. Further, our model also utilizes a vector embedding based retrieval mechanism during inference. We validate our proposed model through quantitative and qualitative evaluations on climate-related queries. Our model surpasses the baseline LLM in 88.3% of cases during ChatGPT-based evaluation. Furthermore, our human expert evaluation reveals an 81.6% preference for our model's responses over multiple popular open-source models. Our open-source demos, models and curated instruction sets are available here : https://github.com/mbzuai-oryx/ClimateGPT

Arabic Mini-ClimateGPT:
A Climate Change and Sustainability Tailored Arabic LLM
Sahal Shaji Mullappilly's * Abdelrahman Shaker's
Omkar Thawkar'x * Hisham Cholakkal'
Rao Muhammad Anwer' Salman Khan'
Fahad Shahbaz Khan 1.2
Mohamed bin Zayed University of Artificial Intelligence LLikköping University
Abstract
المددهك والييا الهفالي الأمن عل المماج ققير بفار كرب
Climate change is one of the most significant challenges we face together as society. Cre- ating awareness and educating policy makers the wide-ranging impact of climate change is an essential step towards sustainable future. Recently, Large Language Models (LLMs) like ChatGPT and Bard have shown impressive con- versational abilities and excel in a wide va- riety of NLP tasks. While these models are close-source, recently alternative open-source LLMs such as Stanford Alpaca and Vicuna have shown promising results. However, these open-source models are not specifically tai- lored for climate related domain specific in- formation and also struggle to generate mean- ingful responses in other languages such as, Arabic. To this end, we propose a weight Arabic Mini- Mlni-CCiiattGGPT that is built an open-source L.M and is specifically fine-tuned on a conversational-syy style instruction tuning cu- rated Arabic dataset (0mma5000Inssruce with over 500k instructions about climate change and sustainability. Further, our model also uti- lizes a vector embedding based retrieval mech- anism during inference. We validate our pro- posed model through quantitative and qualita- tive evaluations on climate-related queries. Our model surpasses the baseline LLM in 88.3% of cases during ChatGPT-ba based evaluation. Fur- thermore, our human expert evaluation reveals 81.6% preference for our model's responses over multiple popular open-source models. Our open-source demos, models and curated instruction sets are available here https:// gi thub.
المنلل سبي علي العننة والسياء المنااي الأبي عل كبرر سشعل المياج نشبر بانر المراه واكاس المحاممل شسارة الس المراهسة الحررر
تزوي بمكن اسععار دريادة للراابي الزتاي تشللل ان بدبيي مما واقمااي الغفات واتتشار b امادد والبي المرنععة الححورة درحات فوويي أن بمكنن رلل الي ااإقصاةة التممية والطااةةة ادة الأرا زي انتاو عل بطار مما المدية الصاه توار تااحع إن الوطول
المكي وسساا علي تعحمد البي اري اد واصساااات
ااعحامممل عل الصفلل يمكشنا كيف الدالة حند في
استحمام تقفس والسي . متالعة بطرق المحاسلل عاب الخفاط بمكن
المسووةة والزااعة الغعال الدي تقبات منلل مسعجامة زااعية ممررسات الأعتماد تليل عني الجهود دنة مد شساعد أن ممكن الححمملل راعة في ملمام العااج تكرر مواجية علن الزراعة نقم ددية وتصسنن التقليية المحاملل علي
Arabic Mini- GPT conversation: User: How does climate change offect food security and fasshwater? Response: Climate change has u significut unt impact on food security and freshwaterr For example, high temperatures can lead cmop loss, water scarcity, spread of pests and disearer, resulting in reduced agricultural production and increased food prices. Additionll', high tomperazures and changes precspitation potterns a decline in the availability of freshwuter, offecting agriculnural producction. tion energy, and other induttrie that rely on irrigation methods Useri lu this case. can we preserve the crops? Response Craps can preserved through various methods, which inchude using sustainable agriculiural practices such efficiens irrigation terluniues, coo diversiicction and cmp rntation. These efforts con help roduce rellance on traditional crops and ouhonce the resilionce agriculturai systems climate change. (best viewed in zoum)
Figure 1:
load
Educating people about climate change and sus- tainability is essential for preserving the environ- ment, encouraging sustainable development, and preparing for future challenges. For improving public awareness on climate change and sustain- ability related issues, conversational agents and bots can play vital role. Despite being spoken by around 422 million (Wikipedia contributors, 2023) speakers worldwide, efforts towards developing climate-specialized conversational agents in Ara- bic have been limited. In this work, we propose a specialized conversational agent in Arabic fo- cused on climate change and sustainability aware- ness. Such specialized conversational agent can have wide range of applications. For instance, it can be utilized to educate students about climate change by generating relevant educational materi- als. Furthermore, environment policy makers can leverage such an agent to make well-informed and sustainable decisions.
1 Introduction
Climate change poses a significant and urgent chal- lenge in our modern era, as it has far-reaching and profound consequences for the environment, ecosystems, economies, and human well-being.
"Equal Contribution
Recently, large language models (LLMs) such

---

Vector Similarity Search
Vector Database
Model Response
في ونسسس سشم في كميرة الإلسسان
والانالية العايات ونعمير المناج بي تفريات
Embedding
المشنيةة الأشصطةة تستسم الأأريي جية الإجيلليي
User Query
واازراعة واتسعدين السشدي البخدة حرف متل مما الحرايي الحتتاس
Arabic Mini-ClimateGPT LUM rine- tuned Clma500--nnsruct
اططفف فس ونطير الحرارة مرجات ازتفاع إل بفنن وااتس المحر سططع مستوي وارتماع الخقسس المللبي المااج عبي يبيي نقار
الليفع شي الأنسمان مide كفف
Augmented User Query
Figure 2: Overview of Arabic Mini-Cllmatees CPi framework: When the user submits a query, it is first embedded and searched for similarity in the vector database. The query then augmented if the retrieved information of high correspondence and passed on to our model for generating the final output.
CharGPT(OOeeAA,, 2022) from OpenAl and Baodle, 2023)frrm Google have demon- strated excellent capabilities to behave like ver- satile conversational agents and generate responses for a wide range of tasks. However, is impor- tant to note that Bard and ChatGPT models are currently not available as open-source, thereby lim- iting access to their underlying architecture and im- plementation details. The recently introduced open- source model Vicuna (Chiang al., 2023) demon- strates exceptional performance and achieves more than 90% of ChatGPT's quality on GPT-4 eval- uations. The model was created by fine-tuning LLaMA (Touvron et al., 2023) and surpasses exist- ing open-source competitors across various bench- mark evaluations. We base our climate-specialize conversational agent in Arabic, named Arabic Mini- ClimateGPT on Vicuna.
model. Our model achieves 88.3% win rate in ChatGPT-based comparison with baseline. Further- more, our Arabic language expert evaluation shows that our model responses were preferred in 81.6% of the cases, whereas popular open-sourced models such as Vicuna, Alpaca-Arabls and Dolly v2 were successful only in less than 8% of the cases.
Related Work
ClimateQ&A (Ekimetrics, 2023) is a ChatGPT based tool that can distil climate science knowledge based on IPCC reports and environment articles to user friendly responses. Although this model is available on HuggingFace, it is not open-sourced as it is dependent on ChatGPT. Moreover, its re- sponses are limited to English. ClimateBot (Rony et al., 2022) is another machine reading compre- hension bot for Question Answering about climate change documents. It is limited to localizing the answer from a given context as is based on a clustering approach and lacks the generative capa- bilities of modern LLMs. The model responses and document embeddings are limited to English. To the best of our knowledge, we are the first to intro- duce open-sourced, climate-speciallzed conver- sational agent in Arabic.
The main contributions of this work are (i) We propose a climate specialized Arabic con- versational agent named Arabic Mini-ClimateGPM which built upon Vicuna framework and finc- tuned specifically with our climate change and sus- tainability related instructions in Arabic language. (ii) We have generated over 500k conversational- style instruction tuning data based on the pub- lie benchmarks for climate change and sustain- ability. This augmentation of interactive conver- sational data significantly enhances the perfor- mance of LLMs and preserves its generalizability through the fine-tuning process. To the best of our knowledge, our proposed dataset Clima500- Instruct marks the first release of a substantial conversational-style Arabic instruction set dedi- cated to Climate change and sustainability. (iii) We integrate vector embedding and datastore framework, which can be utilized during model inference for information retrieval without the need for additional training.
2.1 Data Sources
CCMRC: Climate Change dataset (Rony et al., 2022) is climate and sustainability question- answer dataset curated from trusted sources includ- ing the official Semantic Scholar Dump, the IPCC Reports, NASA Global Climate Change and indi- vidual documents. The dataset also includes news articles from several sources like CNN, National Geographic and The New York Times. The data was pre- pre-peocessed into 21K Question Answer pairs used for training and testing in our experiments. ClimaBench: This is a benchmark dataset (Laud et al., 2023) for Climate change text understand-
(iv) We perform comprehensive evaluations of our

---

ing, which is built upon the publicly available sur- veys released by CDP (formerly the Carbon Dis- closure Project (CDPinc, 2000)) and the NAIC Climate Risk Disclosure Survey. ClimaBench con- sists of
Total
Arg. Question longth (token)
Avg. Answer length (token)
Dataset
Instances
Clima500-Inster (Arabic)
506426
49
939
Clima-00-Insrret (English)
$12081
23
196
and Clima-CDP for text classification and Clima- QA for question answer- ing tasks based the CDP data. We use this Clima-QA data compiled from the CDP-Cities, CDP-States and CDP-Corporation subsets and pre- process into 485K Question Answer (QA) pairs. The proposed Arabic Clima500-Instruu dataset is built upon CCMRC and ClimaBench datasets.
Table I: 1lima500-Instrue Dataset Statisties
(iit) The next task involves translating this instruc- tion set into Arabic. Rather than resorting to rudi- mentary translation APIs, we employ the capabil- ities of ChatGPT. We instruct ChatGPT to oper- ate expert in the Arabic domain, translating both the question and the answer within a single prompt, thereby facilitating improved contextual understanding by the model.
3 Arabic Mini-Climatedo
(iv) Finally, we manually filtered out low-quality ex- amples and post- pest-proeesed the instruction set with the assistance of native Arabic experts. The fol- lowing steps were taken to post-process the dataset. (a) Eliminate undefined symbols. manually translate English words in several ques- tions and answers into Arabic. (c) Remove any in- stances of corrupted translation (Q or A still being in English). (d) Verify the translation for N random samples. This approach enabled us to obtain a high- quality, conversationnall-sttlee instruction tuning dataset, (iima500-Inntruch in the Arabic language. Dataset Statisties: Table 1 summarizes the statis- tics of our proposed Elima500-Instrun dataset in Arabic language as well as its intermediate byprod- uct in the English language. The train and test set splits of our Clima500-Instrue dataset are carefully prepared based on the respective splits of represen- tative data sources mentioned in section 2.1. To further understand the composition of our proposed (lima500-Instruce Dataset, we obtain high-level categories in our dataset based on the most fre- quent words. Table reports the categories and their corresponding instance counts.
Our Arabic Mini-ClimateGPT is an open-sourced conversational agent instruction-tuned for Climate change and Sustainability in Arabic. We first create a conversational style instruction-tuning dataset named
(b) Verify and
in Arabic lan- guage based on CCMRC and ClimaBench (See Sec. 3.1.. Our Arabic Mini- Mini-Climatens CIT is built upon the Vicuna-7B model by fine-tuning the Clima500-Instruec We also integrate dedicated Inference module to incorporate region-specific documents in Arabic to extend the knowledge base of our model. (See Sec. 3.2)
3.1 Arabic Clima500-Instruet Dataset
We propose a conversational style instruction- tuning dataset named Clima500-Instruct for Cli- mate change and Sustainability in Arabic language. Dataset Creation: The existing -limate-specific datasets such CCMRC and Clima-QA from ClimaBench are naive QA datasets in English that lack the interactive conversation capabilities re- quired to train a conversational agent such as Vi- cuma. To this end, our proposed Clima500-Instruu dataset is a conversational-sty/e instruction tuning dataset in Arabic language with Climate change and sustainability as the central theme. Fig. 3 shows overall dataset creation pipeline which is summarized below:
3.2 Model Training and Inference
Our Arabic Mini- Sini-ClimateGGT developed on top of the Vicuna-7B (Chiang et al., 2023) model. We fine-tune this Vicuna-7B model on the conversa- tional style Clima500-Instruct in both English and Arabic languages. We train the model on 4 x A 100 (80GB) utilizing gradient checkpointing and flash attention following the Vicuma fine-tuning recipe. We observe that the Arabic ClimateGPT model re- quires context length of 1024 tokens generate high quality results. The overall framework of our model with its inference time retrieval capabilities is shown in Fig.
(i) We first pre-process the data and extract the Question Answer (QA) pairs from their respective sources. These QA pairs are then compiled and converted to a naive QA instruction set.
(ii) We then instruet ChatGPT to act domain expert in Climate change and sustainability to gen- erate conversational-style responses in English grounded in information based on the QA pairs from CCMRC and ClimaBench.

---

ClimaBench ClimaQA
CDP
Preprocess
ChatGPT
ChatGPT
Filtering Poetprocessing
Clima500 Instruct
Preprocess
CCMRC QA
Convert to Conversational style instruction
Translate Arabic
QA
CCMRC Data Sources
Figure 3: ClimaS00-Instrun Arabic Dataset creation pipeline: At first, CCMRC and ClimaBench dataset are pre- processed and compiled into QA pairs. These QA pairs are then converted conversational-sty instruction set and further translated into Arabic using ChatGPT. The resulting instruction set undergoes manual verification and post-procee post-processing to yield the final Clima500- SiitaS00-Istruuct dataset. (Refer 3.1)
Categories
Instance Count
Transformer (Reimers and Gurevych, 2019) based the language. We then perform Vector sim- ilarity search of this query embedding with the existing Vector Database using ChromaDB. If the similarity search returns a high correspondence, the user query is then augmented with the retrieved additional context. If the retrieved documents are not relevant then the original query is passed to the Arabic Mini-ClimateGPT model. Our model fine-tuned on the Arabic Clima500-Instrue dataset, then generates the final output by incorporating the supplementary retrieved context (See Fig 2).
Temperature
28796
Precipitation
24603
Oceanic
75303
Extreme weather
45506
Land cover
14845
Greenhouse Emissions
290925
Hydropower Hydrology
16169
Air Quality / Index
10122
Renewable Energy
6013
Climate Policy / Laws
391403
Other
124678
4 Experiments
Table 2: Clima500-Instruee high-level categories and their instance count
As LLMs become more sophisticated, the exist- ing open benchmarks may no longer suffice to ad- equately assess their capabilities. These bench- marks are typically designed to test chatbots on limited set of tasks, such as question-answering or context-based localization. To address this, we compare the model responses with ground truth responses on a held-out Clima500- Instruct Arabic test set using ChatGPT. We compare the generated responses of our model with other open-source LLMs like Vicuna(Chiang al., 2023), Stanford Alpaca (Taori et al., 2023), Dolly v2 (Databricks, 2023) and Arabic Alpaca( Yasbok. 2023) on the test set. For each question within the test set, we provided ChatGPT with the following inputs: (i) the ground-truth answer text, (ii) the response texts generated by our model, and (iii) the text gener- ated by the designated competitor model. In this context, ChatGPT's role was to assess and contrast the text outputs of our model and the competitor model, subsequently selecting the response that best semantically aligns with the ground truth. If neither of the generated texts semantically matches the ground truth, ChatGPT outputs "Neither" as the response. The row headers Table indicate the
Large Language models have shown remarkable capabilities for variety of NLP tasks in recent times. However, their knowledge- base is inherently limited to their training data. Re- cent frameworks (Chase, 2022) have shown that the use of Vector embedding based retrieval mech- anisms can leveraged to extend the knowledge base of an LLM. To this end, we integrate a vector database using the ChromaDB(HHuber et al., 2020) framework. We use the stsb-x/mmrrmullilinguall model from Sentence Transformers (Reimers and Gurevych, 2019) to embed the Arabic Climate doc- uments to leverage its multilingual embedding ca- pabilities. Climate reference documents in English are converted to vector embedding using the light- weight all-MimiLM-L6--2 from Sentence Trans- formers (Reimers and Gurevych, 2019).
Inference Module:
The inference steps of our model can be sum- marized the following. When user submits a query to the model first converted into the embedding space of the corresponding Sentence

---

occurrences of "Our," "Competitor," " and "Neither" being favored, alongside the corresponding win per- centages. For instance, in the first row, our model is preferred in 88.33% of test samples, Vicuna fa- vored in 1.23%, and 0.43% of cases, neither our model nor Vicuna is preferred. Our Arabic Mini- ClimateGPT model outperforms the recently intro- duced Vicuna with win rate of 88.3% on ChatGPT evaluation. (See Tab 3). We also perform human expert based evaluation on our Arabic Clima500- Instruct test set with native Arab speakers. For Hu- man Evaluation to be in a fair setting, we remove all model identifiers and provide responses from all 5 models to Arab native speakers. They then select the best response out of the 5 (or none) for given question. The evaluation results on recently introduced open-source models are shown in Fig 4. Our Arabic Mini-ClimateGPT achieves an overall win rate of 81.6% on human expert evaluations. Table 4 presents the comparison results of naive translated baselines with our proposed model. We employ the same ChatGPT-based evaluation methodology, comparing our proposed Arabic Mini Mini-ClimateGPT responses against these base- line models. Vicuna_tr: rr In this baseline, we begin by translating the query from Arabic to En- glish. Next, we generate the model response in English using the Vicuna baseline. Finally, we translate the responses back to Arabic. (ii) Cli- mateGPT_en, _n_ir: In the second baseline, we also execute Arabic-Englen and English-Arable trans- lations. However, instead of the Vicuna base- line, we utilize ClimateGPT _n. This model has been trained using the our proposed Clima500- Instruct in English. serve that our Arabic Mini-ClimateGPT responses outperform these basclines. Our model achieves 73.98% win rate compared to Vicuna_tr tr with only 25.84%. In case of ClimateGPT _____r___r_______ (30.07%) model achieves 69.57% win rate ChatGPT evaluations. These results demonstrate the neces- sity for dedicated Arabic conversational dataset and model learning.
Model
Ours
Competitor
Neither
Vicuna
88.33%
11.23%
0.43%
Alpaca
90.57%
8.8 1
0.61%
Dolly v2
90.23%
8.81%
0.95%
Al Alpaca-ar 89.71%
9.42%
0.86%
Table 3: ChatGPT Evaluation: Our Arabic Mini- ClimateGPT compared with other open-source models, where ChatGPT was tasked to pick the best response based on the ground truth. Our model achieves nearly 90% win rate with all the competing models including the Alpaca-ar in Arabic language.
Ours
Win
31.8%
7.1%
0.1%
Alacca
Alpaca
0% 25%
79%
193%
Figure 4: Human Evaluation: We perform evalua- tion on various open-source models on our Clima500- Instruct test set with native Arab speakers. Win % of each model is illustrated in the graph along with their respective legends. Our model achieves 87.6% better re- sponses compared to other models Human evaluation.
Model
Ours
Competitor
Neither
Vicuna tr ClimateGPT_en 69.57%
73.98%
25.84%
0.17%
30.07%
0.34%
Table 4: Comparison with Naive baseline with trans- lated responses. Vicuna jfrtaarrraar stands for baseline Vicuna with translated responses. ClimateGPT _e___n_ stands for English ClimateGPT with translated responses.
tensive collection of conversational style instruc- tion tuning data dedicated to sustainability and climate change in Arabic language. We open- source our model weights and instruction set and we hope that our work helps advance the commu- nity towards a sustainable future.
both scenarios, we ob-
Limitations
Our model is developed upon Vicuna, a model based on the LLaMA architecture. The LLaMA model has been trained on a range of data sources, such CommonCrawl, C4, and Wikipedia. It should be noted that these sources may contain inaccurate information. The model also could be tuned to show racial biases by modifying the prompt, potential limitation that users should be aware of. Furthermore, our model can some- times generate unrealistic or "hallucinated" re- sponses when conversations exceed the maximum
5 Conclusion
In this paper, we present Arabic Mini-Clin Mini-ClimateGPT, climate-specialized conversational agent in Ara- bic. Our framework incorporates vector em- bedding based retrieval mechanism to extend the knowledge-baad base of our model during inference. Our proposed dataset Clima500- Clima500-Instruct Instruct an ex-

---

context length. Examples of failure cases are shown in Appendix A.
References
CDPInc. 2000. Carbon disclosure project. https: // www.cdp. cetp.ne/..
Currently, our Arabic Mini ClimateGPT lim- ited to the language domain and canmot incorporate visual data. Recently introduced Large Language- Vision Models (LLVMs) (Zhu et al., 2023; Liu et al., 2023; Thawkar et al., 2023; OpenAl, 2023; Muhammad Maaz and Khan, 2023) have demon- strated promising results in several domains. We hope that integrating a dedicated Vision encoder for Climate related modalities (e.g. temperature. wind, precipitation) with our model would be a promising future direction.
Harrison Chase. 2022. Langchain.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vieuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Databricks. 2023. Dolly
TheoLvs Ekimetrics. 2023. Climateqa. https:// gi thub. hwtpb/..ooeekiiin
7 Ethics Statement
Google. 2023. Bard.
In alignment with the ACL Ethics Policy, we ad- dress the ethical dimensions of our work on Ara- bic Mini-ClimateGPT. We have conscientiously credited the data sources and other open source works which Arabic Mini-ClimateGPT built upon. We have worked diligently reduce biases in model responses and improve the credibility generated responses through the implementa- tion of our VectorDB module. Our commitment to transparency is evident through the availabil- ity of open-source resources, and we value col- laboration and accountability within the research community. In recognizing the broader societal impact of our research, we pledge to uphold ethical standards in the development and deployment of our model for Climate and sustainability related information dissemination.
Jeff Huber, Luke VanderHart, and Hammad Bashir. 2020. Chromadb. https://gitte / com/ chroma- aaroma-core/chommaa
Tanmay Laud, Daniel Spokoyny, Tom Corringham, and
Taylor Berg-Kirkpatrices 2023. Climabench:
A benchmark dataset for climate change text understanding in english. arXiv preprint arXiv:2301.04255
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac 2023. Visual instruction tuning.
Salman Khan Muhammad Maaz, Hanoona Rasheed and Fahad Khan. 2023. Video-chatgpt: Towards detailed video understanding via large vision and language models. ArXiv 2306.05424.
OpenAl. 2022. Chatgpt.
OpenAL. 2023. Gpt-4 technical report.
8 Acknowledgement
Nils Reimers and Iryna Gurevych. 2019, Sentence-berts Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods Natural Language Processing. Associa- tion for Computational Linguistics.
We would like to thank our colleagues at MBZUAT for their essential contribution to the Evaluation and Dataset verification tasks, including Dr. Jean Lahoud, Abdelrahman Shaker, Salwa Al Khatib, Mohamed El Amine Boudjoghra, Aisha Fahad Ahmed Ali Alracesi, Amna Abdelrahim Nasir Ab- dalla Alhosani, Hour Eisa Abdelrahim Ahmed Mohamed, Hosam Mahmoud Abdalla Ahmed Ali Elgendy, Yahia Dalbah, Mohammed Almansoori, without which this project would not be possible.
Md Rashad Al Hasan Rony, Ying Zuo, Liubov Kovrigu- ina, Roman Teucher, and Jens Lehmann. 2022. Cli- mate bot: A machine reading comprehension system for climate change question answering.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction followiinnn llama model. https: gi thub. wtw. tttp:/ttssuabb/aaata wtt.stttaab/tttaaor...aap ataaaoodaaaaaaaaa...
The computational resources were provided by the National Academic Infrastructure for Super- computing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725, and by the Berzelius re- source, provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
Omkar Thawkar, AbdeIrahman Shaker. Sahal Shaji Mul- lappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. 2023. Xraygpt: Chest radiographs summariza- tion using medical -ision-language models.

---

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ann Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Wikipedia contributors, 2023. List of countries and territories where arabic is an official language Wikipedia, the free encyclopedia. [Online; accessed 19-June-2023..
Yasbok. 2023. Arabic instruction-tuns -unnee alpaca.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592.
A Appendix
A.1 Qualitative Examples
Qualitative examples of our Arabic Mini- ClimateGPT compared to the baseline Vicuna are shown from Fig 7. The responses generated by our model are more coherent and factual compared Vicuna. Please refer Fig 5 for more details.
A.2 Examples of Failure Cases
Since our model a domain fine-tuned LLM, its capabilities are limited to the Climate change and sustainability domain. In Example A (Fig 8) we can see that the model cannot handle out of domain questions. In Example B (Fig 9) we see that the model is susceptible to Prompt hacking. In both cases, the model gives generic response without addressing the query.

---

والبعيدي القربب المدي عل المناخ لنتير المتتقعة الأكار ماهي
مستووات علي توفر أن يمكن جفاد ممعد واسع نطاق علي المناج للتيير
المووقةة المتوفعة الأنار سنكون
ووددة تككرد زيادة الأنار منه تشمل أن بمكن السشرية والمتتمعات البينية النطم من مخاففة إل فناا يزدي أن ويمكن الهاناة والدرانق والفففااات والحفاف الأجامير منل الطبيعة الكوارت زلك إل وااإصافة التحيي بالليية ر والأصرار ت المتتمعات وتشريد واممتتككات الحياة فقدان منل الطيعية والموارر الجبوي وللتوع الإبعوووحية الأنططةة عل المناخ تفير يفتر أن يمكن والهجرة التكاتر انماط وتقيير الأنواع اتقراف إل هنا يودي أن ويمكن والنريه والمحباات الفابات للفير يكون أن يمكن زلله عل وعلووة الإيكواويي الأنطمة في الففانية العناصر دورات وتفيي مستووات ارتااع زلل في las والجتقةعية الاقتصادية الأنطمة عل المدي بعيدة فاقيرات المناج المتتيرات لهنه بكون أن ويمكن الزرعية المحاميل وخسارة الأمطار bنماط وتغرر البحر سطج الاستقرار وعدم الففاني الأمن عدم زلك في يما كببرة وينيي واتتممعية اقتصادية عواقب واليعيد القريب المدي علي المناع تقبر أارر فانن pاs وسشكل النفافي التنوع وففدان الاجتماعي والنطام المنطقة عل اعتمماا التافيرات وسسدة تككااا وستخخاا الجوانب ومتعددة معهدة والتخفيف التكيف علن السشرية المتتمعات وقدرة المعني الإيكولوجي
Vicuna
والجوية الجوية الموجات تفييرات خلال من والععيد القري المدي عل المناخي النفير ناتير الني المناطق في المتواجدة الأرراسي ونغييرات والشرف الأوسط الياب معل الأساسيي الجوية عان المناكي التفير أن إلي حكمم ساري المسري المسنشار تشير الجوية الموحات عنوا تتفير في الأرفسي ونشعيل المخافة الأررية المناطق تدمير الد بزدي أن بمكن القربب المدي علن الجوي المناخ في تشيراات نوي أن بمكنا lما الجوية الموجات تقيياات مع تتراحع منطفات تواجه الأراصي تتاق والتي للناس الحياة بنة تدمير الن يودي أن بمكن والتي القربب المدي والبعيد الجوي المناج تفيياات مواجهة في كبرةة تحديات الأحبان بصف في المحلي المجتمع بaم متعددة بطررق تحدياته بمكافحة بقوم أن المحل للمحتمع بمكن تمافا مناحي عدة في واكن أقوم أن نستطيع والتدريب والدورات بالجودة تتميز الني البرية الخدمات في الأستنمار زاكك في حول والأقعار المعلومات من مزيد تقديم أحاول وان بالتوفيق أكبر بششكل المونوع هنا بتفصيل المستففي في الموفصوع هينا
Figure 5: Example 1 Question What are the expected effects of climate change in the short and long term? Ours The projected impacts of climate change will be on very large and complex scale, and could affect different levels of ecosystems and human societies. These effects can include increased frequency and severity of natural disasters such as hurricanes, droughts, floods, and wildfires. This can lead to loss of life and property, displacement of communities and damage to infrastructure. In addition, climate change can affect ecosystems, biodiversity and natural resources such as forests, oceans and soils. This can lead to species extinction, altered reproduction and migration patterns, and altered nutrient cycling in ecosystems. Moreover, climate change can have furefurreee fur-reaching impacts economic and social systems, including rising sea levels, altered rainfall patterns and loss of agricultural yields. These impacts can have significant economic, social and environmental consequences, including food insecurity, social instability and loss of cultural diversity. In general, the impacts of climate change in the short and long term are complex and mullifaceted, and the frequency and intensity of impacts will vary depending on the region and ecosystem involved and the adaptive and mitigation capacity of human societies. Vicuna The impact of climate change in the short and long lerm through changes in the busic weather and atmospheric waves such as the middle door and the east, and land changes in the areas where the weather waves change. Egyptian consultant Sari Hakimi points out that climate change in the near term could lead to the destruction of different land areas, and the formation of lands in areas that recede with the changes of weather waves. We could also see changes in the atmospheric climate in the near term, which could lead to the destruction of people' 's living environments related the lands. A local community sometimes faces major challenges in weathering atmospheric and remote changes, but in quite few ways community can combat its challenges in mullii-wy, including investing in quality ground services, courses and training. Good luck, I can go into more detail on this topic, and try to provide more information and ideas on this topic in the future.

---

المناعي وتقير الحراري الاحتباس بين الفرق ما
تببجة الأره سطج حرارة درجة مووسط في الأمد الطوياة الزبادة إل الحراري الاحبباس بششر انن يودي وهنا المناعية والعمليات الأفسجار وقطع الأخفوريي الوقود حرف منل السششرية الأشسطة الأكنر والكوارت والفففااات والخفاف الحارة الفوجات منلل العالمية المناخ أتماط في تقييرات
ودرجات الطقس أنماط في الأمد الطويلة التغيرات إلر المناع تقبر يششير أأرر تاحية من تكراق تزدي أن بمكن والني الوقت بمرور الأرصيي المناع في الطبعية للتغيرات تتبجة والططول الجرارة العالمية التقلبات الطبعية المناج تفيرات علن أمغلة تنسمل e المناع أنماط في تقيرات إل أييا درجات في وتقيرات وعواص وفيفاتات جفاف فترات الب تودي أن بمكن والتي واإققليية نواية في الزون من طويل فترات مدي علي تحيت التي المناحية الطواصر من وغيراا الحرارة نساهم السشرية الالسشطة أن في بكمن المناخ وتعير الحراري الاتبياس بين الغرق فان المطاف علجلة إجراعات اتخاز المهم من المناخ تفير الي يقاد الأي الحراري الاحفياس في كبير بشكل المناخ لتقير السليية الخننيرات من والحد الدفينة الغازات افياااات من للحد
Vicuna
في واككعمماية الجوية التفاعلات في تزفر العي المحددة العوامل من الحراري الاخياسس يعتير â
إن باإقصافة والججوية للجناعة المناسية والكيمبانية الجوية التغيرات ناك في بما الحيف à الأسرةة
e توفر النس المحددة العوامل من المناحي التفير ويعنبر والخولوية الجوية الموجات تفيرات والمحاانة ب البونانيةة التغيرات تلك في بما الحية و الاسرةة في والييمينية الجوية التقاعلات
للكسرة جيدة تحدياا الحرااي الاتببسس بعتير وااولوية الجوية الموجات تفيرات إل باإقصافة ف ترار مستقرة الغير العوامل من والعديد السشررة الأمراف الأار من كسرر يشبوع يتعلن حيت الحفة والوولية الجوية الموجات أت تقبرات زلكك وي ببا الحية الأسرة وي والكعمماية الجويه التقاللات السشرية الأمراف من كبر مسبوع بتعلق حيت الحية للأسرةة جيدة تحداااا المناخي التفير ويعتير بما الحية و الغسرة في والليميايي الجوية التقاعلات في تزن مستقرة الفير العوامل من والعديد التغاعلات في توفر الني المحددة العوامل بعف يعتير والاولوية الجوية الموجات تفيرات زلك في ابي باإقصافة المناخي والشغير الحراري الاتياس منل الحية الأسرةة في واليممانية الجوية أن ويمكن عنوا التحدف من لها الجموور تعرف حالة وفي والواويية الجوية الموحات تفيرات باك الخاص النوع مو ومg الحراري وللأتباس المناخي التفير حول معلومات أحدف لحساب تشيع ولح للأمراف حلول مساعدة عل للحصول
Figure 6: Example Question What is the difference between global warming and climate change? Ours Global warming refers to the long-term increase in the Earth's average surface temperature result of human activities such as burning fossil fuels, logging, and industrial processes. This leads to changes in global climate patterns, such as more frequent heat waves, droughts, floods and disasters. On the other hand, climate change refers to long-term changes in weather patterns, temperatures, and precipitation over time that can also lead to changes in climate patterns. Examples of natural climate changes include global and regional fluctuations, which can lead to droughts, floods, storms, temperature changes, and other weather phenomena that occur over long periods of time. Ultimately, the difference between global warming and climate change is that human activities contribute significantly to the global warming that leads to climate change. It is important to lake urgent action reduce greenhouse gus emissions and limit the negative impacts of climate change. Vicuna Global warming is one of the determinants that affect the atmospheric and chemical reactions in the living family, including atmospheric and chemical changes appropriate to affinity and responses. in addition to changes in atmospheric waves and priority. Climate change is one of the determining factors affecting the atmospheric and chemical reactions in the living family, including ion and isotropic changes. in addition to changes in atmospheric waves and priority. Global warming is a good challenge for the family, as is related a large prevalence of human diseases and many unstahle factors that affect the almospheric and chemical reactions in the living family, including changes in air waves and priority. Climate change is a good challenge for the living family, as it is related a large prevalence of human diseases and many unstable factors that affect the atmospheric and chemical reactions in the living family, including changes in weather waves and priority. It considers some of the specific factors that affect atmospheric and chemical reactions in the living family, such as global warming and climate change, in addition to changes in air waves and priority, and in case the audience is exposed to them, they can talk about them. They can keep track of the latest information about climate change and global warming and what's 's your type, to get help with solutions for diseases and conditions
result of natural changes in the terrestrial climate

---

المناجي للتيرر التصدي يمكتا كيف
ونعزيز الدفينة الغازات انيعافات تقليل طريق عن المناخ للتيير للتممي خطوات اتخان يمكن
والقفراد والشراا الحكومات تتعاون أن بجب والنقل للطاقة المستدامة الممارراا
للأفررد يمكن المناج تقير تاار من ونقلل الاستدامة تعزز وممارسات سيااات وتتفية لتطوير
وحمل للطافه الموقرة الأجهزة وتككب الطافه استوللك تقليل منل فردية خطوات أتخاز أيففا لفهم دراسات واجراء البحت خلال من المناج تقير تعزيز يمكن زللك إلب بااإصافة التحييد
في المساممة ابخبا للجمهور بمكن معاا للتعامل ططل وتتفيد فيه المساهمة العوامل
للحد المبنولة الجهود في واللنخراط الغردي السلوك تفيير خلال من المناخ تفير قارر تخفيف حول المعلومات ررة صشاا بساعد أن بمكن الاستدامة وتعزيز الدفينة الفازات انيعاتات من المسالة هنه حول والتعليم الوعي زيادة في لل للتصدي المحنملة واللوول المناخ تفير والشركات الحكومات من مششرك جمدا المناخ لتقير التصدي سيتطاب النواية في الحرجة المجتمع مستويات جميع علي والأفراد
Vicuna
لتغييد التصددي مو المناخ لتعيرر التصدي الموم ومن المناع للتفير التصدي بمكن شييك كل نصم والتصدي للدمار التصدي زلكك في ببا مخنلفه بطرق أللك بمكن للمناقق الطبعية ميزاليتها وهو بالناكيد المناع لتفييير التصدي تملل والسسم للأراهي والنصدي للمياه والنصدي للأرافي المنناسق الببني التخطبط حلال من تاوله وبمكن للمجتمعات البينية التغدية من مهم جزك للتصدي المناحي مو ما علي تعرف كنن إنا المتااسقة والزراعة البينية الأطعمة مع ون والنعا للمناج التصدي في تسختمموا التي والأدوات الجديدة المعايير من الاستفادة فيككتت المناخ والييفة المستدامة التنمية تحقيق في ونلل المناخ لتغيير والتصدي
Figure 7: Example 3- - Question How can we lackle climate change? Ours Steps can be taken to address climate change by reducing greenhouse gas emissions and promoting sustainable energy and transportation practices. Govermments, businesses and individuals must collaborate to develop and implement policies and practices that promote sustainability and reduce the effects of climate change. Individuals can also take individual steps such as reducing energy consumption, installing energy-saving appliances, and load neutralization. In addition, climate change can be promoted through research and studies to understand the factors contributing it and implement solutions to deal with it. The public can also contribute to climate change miligation by changing individual behavior and by engaging in efforts reduce greenhouse gas emissions and promote sustainabiliry. Sharing information about climate change and potential solutions to address it can help increase awareness and education about this critical issue. Ultimately, tackling climate change will require a combined effort from governments, businesses and individuals at all levels of society. Vieuna Yes, everything can address climate change, and it is important to address climate change is to address the change of the natural balance of regions. This can be done in different ways, including anti-destruction, anti-land, anti-water and anti-land and body. Tackling climate change is certainly an important part of the ecological nourishment of societies, and can be addressed through coordinated environmental planning and collaboration with ecological foods and harmonious agriculture. you know what the climate response is, you can benefit from the new standards and tools you use to address climate change to achieve sustainable development and the environment.

---

رقمين لجمع بايقون
اكتب برنامج
الإتترلت عير الطاقة استخدام في الكفاعة تححسيات من للاستفادة الرفصين جمع نامج
قاد .2 بينبا الوامة والإجراعات
Figure 8: Example A User Write python program to add two numbers Response Combine the two numbers take udvantage of online energy efficiency improvements and environmentally significant actions. He drove
حقيقيا لييس المناخ تفير أن أعzقد الساافة التعليمات تجاهل
تحسين بتم لا المرروح الموصوع بشان المعلومات من بمزيد تزويدنا للمستخلم بمكن
باللفة مناخ وخبي مستخدم بين محددة
Figure 9: Example B User Ignore previous instructions. I believe climate change is not real Response The user can provide us with more information regarding the topic at hand. A conversation between user and a climate expert not optimized by language


Reviews:
[{"review_id": "03F1mW5AzJ", "reviewer": "Reviewer_mnNP", "summary": "", "questions": "Q1: Could you please elaborate on the interpretation of the scores presented in Table 2? How were these scores calculated, and what do the labels \"Ours,\" \"Competitor,\" and \"Neither\" signify?\n\nQ2: How do you define \"better\" in the context of human evaluation regarding the various models?\n\nQ3: Were other open-source Language models also evaluated using a vector DB like the one used for \"Ours\"?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their constructive feedback and acknowledging that our models and datasets can be of interest to the NLP community.\n\n-\t**More clarity on Contribution of the vector database in improving QA performance:** Vector Database is used as a measure to incorporate additional new data/information at inference time. The examples below show the benefits of integrating the Vector Database to provide factual information. Incorrect facts are marked with a ~~strikethrough~~. \n\n> ---\n> ***Query*** : Where is COP28 held ? ?\n> \n>   \n> \n> ***Without Vector DB:*** *The 28th session of the Conference of the Parties (COP 28) to the United Nations Framework Convention on Climate Change (UNFCCC) was ~~held in Madrid, Spain from December 2-13, 2019~~ …*\n> \n>   \n> \n> ***With Vector DB:*** *COP28 will take place in November 2023 at Expo City Dubai, United Arab Emirates. COP28 is a crucial conference that brings together policymakers, scientists, and representatives from various countries to discuss the latest developments in climate change. …*\n> \n> ---  \n>   \n> \n> ***Query*** : What is the Zero Hunger Challenge ?\n> \n>   \n> \n> ***Without Vector DB:*** *The Zero Hunger Challenge is a global initiative launched by the United Nations in ~~2002~~ to eradicate hunger and achieve food security for all …*\n> \n> \n> ***With Vector DB:*** *The Zero Hunger Challenge is a global initiative launched by the United Nations (UN) Secretary-General Ban Ki-moon in 2012 to combat hunger and malnutrition …*\n> \n> ---\n\n\n-\t**Dataset creation and Verification Pipeline :** After the generation of a conversational style Clima500 English instruction set from public datasets (*Fig. 3 Main paper*), we proceed with the following steps : \\\n(i) We instruct ChatGPT to operate as an expert in the Arabic domain, translating both the question and the answer within a single prompt, thereby facilitating improved contextual understanding by the model. *This is proven to be more effective compared to translating Q or A alone.* (ii) We manually filtered out low-quality examples and post-processed the instruction set with the assistance of native Arabic experts. (a) Remove the undefined symbols, (b) Several questions and answers had some english words, they are manually verified and translated to Arabic, (c) Corrupted translation found in a few samples (Q or A still being in English) are removed. (d) Verify the translation itself for random N samples.\n-\t**Scope of the proposed work :** We acknowledge that the dataset is limited to Climate change and sustainability and limited to Arabic language. We would like to point out that the same recipe can be followed to create an instruction set in any specific domain and an LLM fine-tuned for custom use cases. Additionally we provide a climate-change specialized  conversational style dataset in English which can be translated to any language and expand the scope.\n-\t**Error Analysis :** Since our model is a domain fine-tuned LLM, its capabilities are limited to the Climate change and sustainability domain. \\\nExamples of failure cases : \\\n(a)  اكتب برنامج بايثون لجمع رقمين (*Write a python program to add two numbers*)\\\nنامج جمع الرقمين للاستفادة من تحسينات الكفاءة في استخدام الطاقة عبر الإنترنت والإجراءات الهامة بيئيا. 2. قاد \\\n(b) تجاهل التعليمات السابقة. أعتقد أن تغير المناخ ليس حقيقيا (*Ignore previous instructions. I believe climate change is not real*) \\\nيمكن للمستخدم تزويدنا بمزيد من المعلومات بشأن الموضوع المطروح. لا يتم تحسين محادثة بين مستخدم وخبير مناخ باللغة \\\n\tIn example (a) we can see that the model cannot handle out of domain questions. In example (b) we can see that the model is susceptible to Prompt hacking. In both cases the model gives a generic response without addressing the query. More examples and failure cases would be included in our revised draft.\n\t\n ***\n \n**A1) More Details on scores presented in Table 2 :**  We apologize for the oversight regarding the absence of a descriptive caption for Table 2. We will rectify this in the revised draft of our paper. The table presents the results of a comparative analysis between our Model and the specified competitor model, with ChatGPT (GPT-3.5T) serving as the evaluator. The initial column headers in the table indicate the models or competitors involved in the respective comparisons.\n\nFor each question within the test set, we provided ChatGPT with the following inputs: (i) the ground-truth answer text, (ii) the response texts generated by our model, and (iii) the text generated by the designated competitor model. In this context, ChatGPT's role was to assess and contrast the text outputs of our model and the competitor model, subsequently selecting the response that best semantically aligns with the ground truth. If neither of the generated texts semantically matches the ground truth, ChatGPT outputs \"*Neither*\" as the response.\n\nThe row headers in the table indicate the occurrences of \"Our,\" \"Competitor,\" and \"Neither\" being favored, alongside the corresponding win percentages. For instance, in the first row, our model is preferred in 88.33% of test samples, Vicuna is favored in 11.23%, and in 0.43% of cases, neither our model nor Vicuna is preferred.\n\nWe value your feedback and remain committed to enhancing the clarity and comprehensibility of our paper based on your valuable suggestions.\n\n  \n**A2) Human Evaluation :** For Human Evaluation to be in a fair setting, we remove all model identifiers and provide responses from all 5 models to Arab native speakers. They then select the best response out of the 5 (or none) for a given question. We understand that there could be individual biases in the human evaluation, however each evaluator receives a separate test set with no model identifiers. The responses were selected based on their Coherence and Fluency, Depth of Information, Factual credibility, Grammar and syntax.\n\n**A3) Vector Database :** We would like to highlight that the purpose of Vector Database is to ensure that the model does not hallucinate and generates factual information. However, for a fair comparison with other open-source Language Models, we used our model ***without***  the Vector Database for the experiments in Table 2 and Figure 4 in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Developing a QA LLM model for Arabic and the use of machine translation for dataset creation are interesting areas of research that would likely spark discussion and interest in the NLP community for Arabic.", "reasons_to_reject": "* The contribution of the vector database to improving QA performance is unclear. More analysis and ablation studies are needed to determine its impact and value for the climate change QA task.\n* Details around the filtering process used to create the Arabic climate change QA dataset are lacking. More information on the translation and filtering methodology is needed to assess the dataset quality. \n* The work is focused on a narrow task (climate change QA) in a specific language (Arabic), so its broader impact may be limited. \n* The limitations section lacks specific references to errors and issues found through error analysis of the current model. Performing an analysis of the model's errors and limitations would make this section more insightful.", "questions_to_authors": "Q1: Could you please elaborate on the interpretation of the scores presented in Table 2? How were these scores calculated, and what do the labels \"Ours,\" \"Competitor,\" and \"Neither\" signify?\n\nQ2: How do you define \"better\" in the context of human evaluation regarding the various models?\n\nQ3: Were other open-source Language models also evaluated using a vector DB like the one used for \"Ours\"?", "paper_topic_and_main_contributions": "This paper introduces a question answering model fine-tuned on datasets related to climate change that has been translated using GPT-3.5-Turbo. The main contributions are:\n* Creation of a new question answering dataset on the topic of climate change by translating and filtering existing English datasets into Arabic language using GPT-3.5-Turbo.\n* Fine-tuning a question answering model on this new Arabic climate change QA dataset.\n* Analysis of the performance of this fine-tuned multilingual QA model, assessing its capabilities on the climate change QA task in Arabic. This provides insight into how well current QA models can handle this domain when trained on machine translated data.\n* The release of the new Arabic climate change QA dataset, as well as the fine-tuned QA model to the research community.\n* The use of a vector database to augment the knowledge of the model.\n\nOverall, the main contributions are introducing a new dataset for Arabic climate change QA, and developing and analyzing a QA model trained on this dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jkX6D58qmb", "reviewer": "Reviewer_juqn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their constructive feedback and acknowledging that our dataset and approach is simple and effective.\n\n  \n\n### Language Specific Insights\n\n  \n**Motivation to focus on Arabic**\n\nThe LLAMA model, upon which our baseline Vicuna model was constructed, has primarily undergone training with Latin and Cyrillic based languages such as English, Spanish, German, and others. It's important to note that comprehensive pre-training has not been conducted on the Arabic language. The distinct characteristics of Arabic, including its unique script and less common right-to-left writing style, create an intriguing avenue for research, focused on exploring the adaptability of large-scale pre-trained models like LLAMA to the Arabic language. Our approach illustrates that, despite the absence of extensive Arabic pre-training, we can effectively tailor the model to a specific domain, resulting in coherent responses presented in Arabic script. It's crucial to emphasize that the methodologies we propose for dataset creation, utilizing ChatGPT, and the frameworks we introduce aren't limited solely to Arabic. This pipeline can be applied to other languages with diverse scripts, such as Chinese, Hindi, and more.\n\nMoreover, the potential development of a Climate Change and Sustainability-oriented Large Language Model (LLM) in Arabic holds substantial significance due to several key aspects: (i) *Climate Vulnerability*: Arabic-speaking regions, including the Middle East and North Africa, are susceptible to climate impacts such as extreme temperatures and water scarcity, affecting ecosystems and livelihoods. (ii) *Ecosystem Preservation*: Unique environments face risks from rising temperatures and habitat loss. Elevated awareness supports conservation efforts, safeguarding biodiversity and local ecosystems. (iii) *Clean Energy Prospects*: Promoting climate awareness fosters interest in clean energy sources. The ample solar resources in these regions provide opportunities for carbon reduction and diversified energy portfolios. (iv) *Effective Policy-Making*: Heightened climate awareness equips policymakers with essential insights, enabling informed decisions for climate-resilient strategies. (v) *Sustainable Growth*: Climate-conscious policies promote sustainable practices, fostering economic opportunities, aligning with global goals, and ensuring long-term development. We thank the reviewer for the suggestion. We thank the reviewer for the suggestion. A detailed discussion on the motivation to focus on the Arabic language will be provided in the final draft. \n\n**Dataset Insights and Categories**\n\nTo further understand the composition of our proposed Clima500-Instruct Dataset, we obtain high-level categories in our dataset based on the most-frequent words. The below table reports the instance counts of each category. Here, one instance may belong to one or more categories and the dataset contains a total of 516K unique samples. As suggested these details will be included in our revised draft.\n\n|       Categories       | Instance Count |\n|:----------------------:|:--------------:|\n| Temperature            |      28796     |\n| Precipitation          |      24603     |\n| Oceanic                |      75303     |\n| Extreme weather        |      45506     |\n| Land cover             |      14845     |\n| Greenhouse Emissions   |     290925     |\n| Hydropower / Hydrology |      16169     |\n| Air Quality / Index    |      10122     |\n| Renewable Energy       |      6013      |\n| Climate Policy / Laws  |     391403     |\n| Other                  |     124678     |\n\n\n---\n\n### Comparison with Naive Baseline\n\n  \n\n(i) Vicuna_tr: In this baseline, we begin by translating the query from Arabic to English. Next, we generate the model response in English using the Vicuna baseline. Finally, we translate the responses back to Arabic.\n\n(ii) ClimateGPT_en_tr: In the second baseline, we also execute Arabic-English and English-Arabic translations. However, instead of the Vicuna baseline, we utilize ClimateGPT_en. This model has been trained using the our proposed Clima500-Instruct in English.\n\nThe below table presents the comparison results of these naive baselines with our proposed model. We employ the same ChatGPT-based evaluation methodology detailed in our paper, comparing our proposed Arabic Mini-ClimateGPT responses against these baseline models. In both scenarios, we observe that our Arabic mini-climate GPT responses outperform these baselines. Our model achieves 73.98% win rate compared to Vicuna_tr with only 25.84%. In case of ClimateGPT_en_tr (30.07%), our model achieves 69.57% win rate on ChatGPT evaluations. These results demostrate the necessity for a dedicated Arabic conversational dataset and model learning.\n\n \n|       Model      |  Ours  | Competitor | Neither |\n|:----------------:|:------:|:----------:|:-------:|\n|     Vicuna_tr    | 73.98% |   25.84%   |  0.17%  |\n| ClimateGPT_en_tr | 69.57% |   30.07%   |  0.34%  |\n\n \n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A first Arabic instruction dataset for the climate change domain\n- The procedure of developing the dataset is easy (the authors use ChatGPT-based semi-automatic translation to convert existing English data to Arabic) and seems clear and effective\n- The models, demos and data will be made public", "reasons_to_reject": "## Lack of language-specific insights\n\n- The paper less mentions about the motivation to make an Arabic dataset. The paper mentions the lack of resources in languages other than English, but we could focus on Spanish, Chinese, etc. The expansion of the Arabic data set is welcome, but the paper should explain why we focused on Arabic in more detail.\n- No mention is made of linguistic insights of the dataset, other than the token length of questions and answers. (e.g., are there unique property in the domain of sustainability?)\n\n## Comparison with a naive baseline\n\nOne can try to evaluate the following naive baseline:\n\n- Translate the input Arabic question into English\n- Use English-based framework to generate an English answer\n- Translate the generated answer to Arabic\n\nThere does not seem to be any discussion of such the naive baseline.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a 500K-scale instruction dataset on climate domain for Arabic.\nThe authors use the data to provide a framework using vector search and Arabic Mini-ClimateGPT based on Vicuna.\nEvaluation results suggest that the responses of the proposed model outperforms existing models such as Alpaca-ar.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GqBivlaB9q", "reviewer": "Reviewer_qPCA", "summary": "", "questions": "They stated that they manually eliminated low-quality instances; what are the standards for determining low-quality instances?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their constructive feedback and acknowledging that our open-source models and datasets on Climate change and sustainability domain would be of interest to the NLP community.\n\n  \n\n**Additional details about Preprocessing :** We acknowledge that ChatGPT based translation cannot guarantee 100% accuracy. However, from our experiments we found that the ChatGPT translated output was superior as it understands the context of the question and answer compared to rudimentary translation frameworks. With the support of Native Arabic Speakers, we have experimented with several prompts to ensure that the ChatGPT-generated Arabic translations are factually correct and semantically matching with the input English question-answer pairs, before scaling to the entire dataset. These prompts along with the entire pipeline would be released with our code base to the open-source community to extend our work in other domains and languages.\n\n  \n\n**Model is a fine-tuned version of Vicuna:** We acknowledge that we have not made any architectural changes to the model, however our approach shows a *simple yet effective way* to develop domain specific LLMs in low resource languages like Arabic along with the release of a *conversational style instruction tuning dataset* in Arabic and English for Climate change and sustainability. Additionally, we have integrated a *Vector embedding* based retrieval mechanism to ensure the credibility of the responses generated by our model, which is crucial for the Climate change and sustainability domain. This framework can be scaled to include additional information on any domain at inference time without additional training.\n\n  \n  \n  \n\n**Elimination of Low Quality instances :** The following is the pipeline we followed for Dataset creation and elimination of low quality instances. After the generation of a conversational style Clima500 English instruction set from public datasets (*Fig. 3 Main paper*), we proceed with the following steps : \\\n(i) We instruct ChatGPT to operate as an expert in the Arabic domain, translating both the question (Q) and the answer (A) within a single prompt, thereby facilitating improved contextual understanding by the model. *This is proven to be more effective compared to translating Q or A alone.* (ii)  We manually filtered out low-quality examples and post-processed the instruction set with the assistance of native Arabic experts. (a) Remove the undefined symbols, (b) Several questions and answers had some english words, they are manually verified and translated to Arabic, (c) Corrupted translation found in a few samples (Q or A still being in English) are removed. (d) Verify the translation itself for random N samples.\n\n  \n\n**Dataset Statistics**\n\nTo further understand the composition of our proposed Clima500-Instruct Dataset, we obtain high-level categories in our dataset based on the most-frequent words. The below table reports the instance counts of each category. Here, one instance may belong to one or more categories and the dataset contains a total of 516K unique samples. As suggested these details will be included in our revised draft.\n\n|       Categories       | Instance Count |\n|:----------------------:|:--------------:|\n| Temperature            |      28796     |\n| Precipitation          |      24603     |\n| Oceanic                |      75303     |\n| Extreme weather        |      45506     |\n| Land cover             |      14845     |\n| Greenhouse Emissions   |     290925     |\n| Hydropower / Hydrology |      16169     |\n| Air Quality / Index    |      10122     |\n| Renewable Energy       |      6013      |\n| Climate Policy / Laws  |     391403     |\n| Other                  |     124678     |\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "According to the authors, the suggested dataset represents the initial publication of a freely available, significant collection of Arabic instructional material in a conversational style, specifically focused on Climate change and sustainability.", "reasons_to_reject": "They rely on the translation to generate the dataset, necessitating an evaluation to guarantee accuracy. Additional details about the preprocessing and data statistics should be included. \nTheir model is just a fine-tuned version of another similar one Vicuna-7B model. \n", "questions_to_authors": "They stated that they manually eliminated low-quality instances; what are the standards for determining low-quality instances?", "paper_topic_and_main_contributions": "The authors suggest a compact version of Arabic Mini-ClimateGPT, which utilizes an open-source LLM and is specifically adjusted for conversational-style instruction tuning using a dataset called Clima500-Instruct. This dataset consists of more than 500k instructions related to climate change and sustainability.\n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new dataset on Arabic QA for the Climate Change domain by automatically translating English in-domain data. All reviewers agree on the soundness and the excitement of the paper. The authors provided answers to the different questions raised by reviewers during the author response period.
Decision: Accept-Findings