ExpressivityArena: Can LLMs Express Information Implicitly?
Authors: Joshua Tint, Som Sagar, Bimsara Pathiraja, Aditya Taparia, Kelly Raines, Caleb Liu, Ransalu Senanayake


Reviewer paper summaries: This paper examines the ability of Large Language Models (LLMs) to express implicit information, beyond simple facts introducing ExpressivityArena as a framework to evaluate the expressiveness of LLMs through creative and logical tasks such as poetry and programming. The experiments show that LLMs can generate implicit expressive content but with some limitations, especially in domains apparently less expressive like programming. The results suggest that although LLMs can be expressive, there is still much room for improvement regarding implicit and nuanced communication.
Summary of strengths: - **Comprehensive Framework**: The paper presents ExpressivityArena, a detailed framework to evaluate the expressivity of LLMs generated content, covering various example domains like poetry, coding, and professions. It is very interesting the different implicit signals chosen and crafted among these different domain such as emotion for poetry and programming style for coding. Moreover, it is very useful the possibility to use this framework to automatically generate and evaluate different samples in various domains of interest.
- **Different Experiments**: The study conducts multiple experiments to test expressivity in different contexts, for example in both single-generated text and in a multi-turn dialogue, providing a thorough analysis of LLM capabilities. It is appreciable the study conducted in Experiment 3 which seems that in a conversation the LLMs investigated tend to lose the starting prompted emotional signals across time, while they tend to explicit the signal when the conversation is between two models prompted with a profession.
Summary of weaknesses: - **Automated Grading**: It uses an LLM as an automated grader to evaluate expressivity. If I understood correctly, automated grading is justified via Experiment 1, but the dataset seems too small (5 examples) to claim that a grader LLM could be good or not in evaluating implicit expressive signals of the outputs of tested LLMs
- **Experiment 2 – Programming**: The dataset is made by 90 examples only, maybe a bigger dataset could be a better choice to have more robust and representative results to put in the Experiment 2 section.
Comments, suggestions and typos: - Aside from evaluating the ability of tested LLMs to express implicit information, in this framework could be interesting to investigate the capability of grader LLMs to recognize the nuanced signals in a text. Is the tested LLM good at expressing the implicit information? Or is the grader LLM bad at recognizing the implicit information? Like the chicken and the egg problem!
- Another point is that when a grader LLM is prompted to guess the implicit signal of a text, it could be insightful to investigate the behavior of a model adding a void (or ‘none’) class that indicates no implicit signal is detected in the text. In this way, the grader LLM could abstain.
Overall assessment scores: 3.5