Assessing Italian Large Language Models on Energy Feedback Generation: A Human Evaluation Study
Authors: Manuela Sanguinetti, Alessandro Pani, Alessandra Perniciano, Luca Zedda, Andrea Loddo, Maurizio Atzori (Università di Cagliari, 2024)
Topic: Language Models

Abstract
This work presents a comparison of some recently-released instruction-tuned large language models for the Italian language, focusing in particular on their effectiveness in a specific application scenario, i.e., that of delivering energy feedback. This work is part of a larger project aimed at developing a conversational interface for users of a renewable energy community, where clarity and accuracy of the provided feedback are important for proper energy management. This comparison is based on the human evaluation of the output produced by such models using energy data as input. Specifically, the data pertains to information regarding the power flows within a household equipped with a photovoltaic (PV) plant and a battery storage system. The goal of the feedback is precisely that of providing the user with such information in a meaningful way based on the specific aspect they intend to monitor at a given moment (e.g., self-consumption levels, the power generated by the PV panels or imported from the main grid, or the battery state of charge). This evaluation experiment has the two-fold purpose of providing an exploratory analysis of the models' abilities on this specific generation task solely relying on the information and instruction provided in the prompt and as an initial investigation into their potential as reliable tools for generating user-friendly energy feedback in this intended scenario.

Introduction
The provision of energy feedback plays a crucial role in promoting energy efficiency among users. The expression energy feedback (or eco-feedback) covers a wide range of energy-related information. This can include detailed reports on energy usage and production (in the case of renewable energy sources), as well as energy-saving advice, whether generic or user-specific. The primary goal of energy feedback is to allow users to make informed decisions regarding their energy management, thus promoting better conservation practices. A substantial body of literature within the field of Human-Computer Interaction (HCI) has explored various energy feedback mechanisms, primarily focusing on visual or ambient feedback as well as gamification techniques (we refer to the surveys proposed by \citet{Albertarelli-etal-2018} and \citet{Chalal-etal-2022} for further details on these aspects). However, a greater interest has been reported on the delivery of energy feedback through conversational agents AUTHOR. Furthermore, within the field of Natural Language Generation (NLG), several studies prior to the advent of Large Language Models (LLMs) investigated the use of NLG architectures to communicate consumption data. Notable works include those by \citet{Trivino-Sanchez-2015} and \citet{Conde-Clemente-etal-2018}, which used fuzzy sets to tackle data-to-text generation tasks, also tailoring the linguistic description on given consumption profiles. Similarly, \citet{martinez-municio_linguistic_2018} employed fuzzy sets to produce linguistic summaries based on the consumption of specific buildings or groups of buildings, using time series data as input. This work is part of a research project aimed at developing a modular task-oriented conversational agent to inform users about their energy consumption and photovoltaic (PV) production and, more generally, to support better management of their energy resources through text-based energy feedback. The conversational agent will then be deployed and tested within a renewable energy community in Italy, which motivates our specific focus on Italian as the primary language for the interactions. At this stage of the project, we plan to integrate the generative abilities of LLMs into the conversational pipeline. This approach is expected to deliver more varied and dynamic responses instead of predefined, static templates, possibly making the user experience enjoyable. This study was driven by the need to obtain more quantitative insights into the expected performance of such models when tasked with the delivery of energy feedback based on actual energy data. The main objective of this study thus aims to verify how effectively instruction-tuned LLMs currently available for the Italian language can deliver clear and accurate feedback based on energy data provided within a prompt, without relying on more elaborate techniques like fine-tuning or Retrieval Augmented Generation. More specifically, we formulated the following research questions: \item Are the LLMs under study able to produce energy feedback that is 1) informative, 2) comprehensible, and 3) accurate with respect to the provided energy data? \item Are there any major differences among such models with respect to these capabilities? To answer these questions, we conducted an exploratory analysis by manually evaluating some of these Italian LLMs, organizing the study around criteria designed to quantify these specific aspects. This work closely aligns with a recent initiative that has been launched within the Italian NLP community, i.e., CALAMITA}, a campaign aimed at evaluating the capabilities of Italian (or multilingual, but including Italian) LLMs on specific tasks in zero or few-shot settings. Unlike the latter, however, our study relies solely on human judgments rather than automatic metrics. The main challenges of a manual approach include the absence of standardized practices and evaluation criteria AUTHOR, as well as the lack of systematic documentation AUTHOR, which hinders the reproducibility of such studies.} In light of these challenges, the intended contributions of this paper are outlined below: \item A small-scale human evaluation of several Italian LLMs on a specific task. \item The description of a protocol for human evaluation inspired by the good practices recommended in recent literature AUTHOR. To this end, we also make available the evaluation dataset, with the ratings assigned by the evaluators in a non-aggregated form. The remainder of this paper describes how this study was designed and carried out, with a discussion of the results obtained and the main limitations of the work.

Conclusion
This study provides an initial assessment of several Italian language models' ability to generate effective energy feedback. %The evaluation focused on three key dimensions—informativeness, comprehensibility, and accuracy—organized into five evaluation criteria. The results indicate that while the models generally perform well, particularly in terms of comprehensibility and accuracy, there is greater variability regarding informativeness. Among the tested models, results show that, except for LLaMAntino2-7B-UltraChat, the remaining ones provide comparable performances. However, it is important to highlight the limitations of this study. First, this is a small-scale study, as it involves a limited number of models and evaluators. Concerning the former issue, we also point out that the study was restricted to models available on HuggingFace, excluding potentially relevant models from external sources, such as Fauno} and Camoscio AUTHOR. A more systematic study should consider these models as well, in order to provide a more comprehensive evaluation over the Italian LLMs' landscape. As for the pool of evaluators, it is important to note a significant bias in both their personal backgrounds and demographics. All the judges have a background in computer science and varying degrees of familiarity with the topics at hand. Furthermore, there is a gender imbalance (1 female and 3 male judges) and a lack of age diversity, as all four judges fall within the 24–30 age range. In light of these considerations, a more systematic comparison as the one envisioned above would benefit from a broader and more diverse pool of evaluators. This would not only increase the reliability of the comparison but also provide a deeper understanding of potential correlations between socio-demographic factors, prior knowledge of technology and energy-related concepts, and the differing perceptions of the evaluation criteria considered in our study. Common approaches to address the lack of human participants include the use of crowdsourcing platforms, with a careful design of participation criteria that would ensure a better gender and demographic balance. Alternatively, a user study involving prospective users of the conversational agent could be conducted; this would ultimately enable to gather valuable insights on the type of feedback expected by the target audience. Finally, an extended evaluation framework should also include an analysis of the statistical power of the sample size to ensure more robust conclusions. Despite these limitations, this work offers a preliminary overview and aims to pave the way for future research on this aspect, also stressing the importance of more standardized human evaluation practices. As a matter of fact, the evaluation protocol we designed draws heavily from methodologies recommended in more general literature pertaining to human evaluation within generation and summarization tasks. Our approach thus aims to ensure that the core principles of the experiment are flexible enough to be easily replicated or adapted for a wider range of different domains. This work has been developed within the framework of the project e.INS- Ecosystem of Innovation for Next Generation Sardinia (cod. ECS 00000038) funded by the Italian Ministry for Research and Education (MUR) under the National Recovery and Resilience Plan (NRRP) - MISSION 4 COMPONENT 2, "From research to business" INVESTMENT 1.5, "Creation and strengthening of Ecosystems of innovation" and construction of "Territorial R\&D Leaders". This work was also partially funded under the National Recovery and Resilience Plan (NRRP) - Mission 4 Component 2 Investment 1.3, Project code PE0000021, “Network 4 Energy Sustainable Transition--NEST”. \appendix \newpage