Evaluating LLMs for Detecting Climate Misinformation: How Aligned are LLMs with Expert Classification of False or Misleading Claims about Climate Change on Social Media?
Authors: Mowafak Allaham, Ayse Deniz Lokmanoglu, P. Sol Hart, Erik C Nisbet


Reviewer paper summaries: The authors benchmark GPT-4 and a fine-tuned GPT-3.5-turbo model against a BERT-based model, and two expert human annotators.They use two datasets: the CARDS dataset containing text from climate skeptic sources, and a custom dataset of highly engaging social media posts. The results show that the fine-tuned GPT-3.5-turbo model surpasses the other models. The authors also show that the finetuned 3.5-Turbo achieves an alignment with expert human annotation comparable to the inter-coder reliability between the two human experts.
The paper highlights the potential of fine-tuned LLMs as tools for civil society organizations to counter misinformation, particularly in content moderation and guarding against LLM-generated misinformation. However, limitations include the focus on English text data and reliance on the CARDS taxonomy, which may introduce biases. The authors acknowledge these limitations and suggest future research directions, such as evaluating the models on high-credibility sources and expanding the taxonomy.
Summary of strengths: - The paper is on a very important topic.
- The paper reads well and explains methods clearly and thoroughly.
Summary of weaknesses: - There is no novel method used in the paper. A short paper could have covered the same materials. 
- Multiple studies have concluded in different domains that a finetuned smaller model would outperform a larger model. 
- Overall contribution is not up to the mark of a full paper.
- Authors discuss very interesting ideas in limitations section, would urge them to make them part of the full paper.
Comments, suggestions and typos: Either frame the paper as a pure climate change communications paper or develop novel techniques which are applied to the climate change domain as an example. In its current form the paper does not add much to either research directions.
There is lot that could be achieved by going in details of skeptics vs contrarian trained GPT models, I would urge authors to continue working with the data and bring novelty to methods/results.
Overall assessment scores: 1.5