## 3 Methodology
### Target-Level Sentiment Classification with ChatGPT and Gemini  
The recent rise of LLMs has been highly notable, showcasing their exceptional capacity to execute a wide range of text-oriented tasks [71]. Presently, OpenAI's ChatGPT ([[https://chatgpt.com/](https://chatgpt.com/)]([https://chatgpt.com/](https://chatgpt.com/)) (Versions available as of 30 June 2024)) and Google's Gemini ([[https://gemini.google.com/app](https://gemini.google.com/app)]([https://gemini.google.com/app](https://gemini.google.com/app)) (Versions available as of 30 June 2024)) are two highly esteemed and widely used LLMs. The current study utilized multiple versions of the ChatGPT and Gemini models, such as ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Gemini 1 Pro, Gemini 1.5 Flash, and Gemini 1.5 Pro, to perform TLFSA. No fine tuning of the models was conducted, as the objective of this study was to evaluate the zero-shot performance of these advanced generative AI tools on the given task. Interactions with ChatGPT-3.5, ChatGPT-4, and ChatGPT-4o were carried out using the ChatGPT user interface (UI) in its default configuration, accessible to all users, with a preset temperature of 0.7. Additionally, access to Gemini models (Gemini 1 Pro, Gemini 1.5 Flash, and Gemini 1.5 Pro) was obtained via the Gemini API. The recommended temperature settings provided by the API were followed: 0.9 for Gemini 1 Pro and 1.0 for both Gemini 1.5 Flash and Gemini 1.5 Pro.  
To evaluate the performance of several ChatGPT and Gemini models in TLFSA, we utilized a zero-shot prompting method (we also explored the few-shot learning approach by adding a small number of gold-annotated headlines as examples to the prompt, aiming to guide the LLM in performing the task. However, this did not result in significant performance improvements. A more comprehensive evaluation of the few-shot learning capabilities of these models for this task is planned for future work) [72; 73]; that is, the model is instructed with an indication of how to perform a given task without providing any gold annotated example (i.e., a headline and the corresponding manual annotation in our context). More in detail, the following prompt was crafted:  
_Evaluate the sentiment conveyed by the headline with respect to the {target} from an investment perspective. Employ a three-tier scale for this assessment: allocate a score of (\(-1\)) for a negative sentiment, (0) for a neutral sentiment, and (+1) for a positive sentiment. Assign a neutral score of 0 if the headline is vague or does not clearly express positive or negative implications about the {target}._  
By utilizing zero-shot prompting, we exploit the pre-existing training and comprehension of language context and patterns in ChatGPT and Gemini to produce desirable responses without the need for additional training or task-specific fine tuning. This method enables a straightforward evaluation of the intrinsic capacities of these models in sentiment analysis without any alterations, showcasing their potential for effortless integration and utilization in different contexts.  
The performance of all the above models and methods was evaluated using a variety of well-established metrics for sentiment/classification tasks [74; 75; 76], such as precision, recall, F1-measure, and accuracy. Specifically, we report the aggregated performance across the three classes (Positive, Neutral, Negative) using both macro-averaging, where metrics are calculated for each class separately and averaged equally across all classes, and weighted averaging, where metrics are weighted by the number of samples in each class. These metrics provide a comprehensive evaluation of each model's effectiveness in detecting target-level sentiments, even in the presence of a slightly imbalanced class distribution, as seen in our dataset [75; 76].