领域：学术论文 · 应用学科

In an era characterized by rapid digitization and the proliferation of online communication platforms, vast amounts of textual data are generated daily. While the proliferation of online communication provides rich insights, extracting meaning from vast textual data remains resource intensive. Latent content analysis, which involves decoding the underlying meanings, sentiments, and nuances in text, is crucial for understanding social dynamics, informing policy decisions, and guiding business strategies. Automating this process could significantly enhance our ability to respond to societal needs promptly and effectively.
The societal implications of effectively analyzing textual content are profound. Sentiment analysis can reveal public opinion on policies or products, influencing governmental decisions and corporate strategies. Understanding political leanings aids in assessing electoral landscapes and fostering democratic engagement. Detecting emotional intensity and sarcasm in communication is vital for mental health monitoring, customer service, and even national security. Large Language Models (LLMs) offer the potential to perform these analyses at scale, reducing reliance on extensive human labor and accelerating the time to insight.
The field of automated content analysis has evolved significantly over the past few decades. Early computational approaches relied on manual coding schemes applied to small datasets. The advent of machine learning introduced algorithms capable of handling larger datasets with increased efficiency. Traditional models, such as Naïve Bayes and Support Vector Machines, were used for tasks like sentiment classification but often struggled with contextual understanding.
The introduction of deep learning architectures marked a transformative period in natural language processing (NLP). Models utilizing word embeddings captured semantic relationships between words. Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks improved the modeling of sequential data. These advancements enhanced performance in sentiment analysis and emotion detection tasks.
Recent research has also explored integrating multi-modal data (e.g., text with images or audio) to enhance sentiment and effect detection, a challenge that LLMs may benefit from addressing. Thareja addressed the challenges posed by extreme emotional sentiments on social media platforms like Twitter, which can impact users’ mental well-being. Introducing Tweet-SentiNet, a multi-modal framework utilizing both image and text embeddings, the study demonstrated improved sentiment analysis by effectively filtering content with extreme sentiments. Similarly, Li et al. proposed a multi-modal sentiment analysis model based on image and text fusion using a cross-attention mechanism. By extracting features using advanced techniques like ALBert for text and DenseNet121 for images, and then fusing them with cross-attention, their model outperformed baseline models on public datasets, achieving accuracy and F1 scores of over 85%. Akhtar et al. explored a deep multi-task contextual attention framework for multi-modal affect analysis. Recognizing that emotions and sentiments are interdependent, they leveraged the associations among neighboring utterances and their multi-modal information.
Recent approaches broaden the scope of stance detection by integrating knowledge graphs, which help capture political or ideological context in more structured ways. Likewise, boundary-aware frameworks for few‐shot tasks have shown promise in enriching entity‐level interpretations.
Despite these improvements, models have been found to still face challenges in interpreting complex linguistic features such as sarcasm and nuanced emotions. Sarcasm detection, for instance, requires an understanding of contextual cues and sometimes external knowledge beyond the text itself, leading researchers to explore context-aware and multi-modal approaches to enhance detection accuracy. Baruah et al. investigated the impact of conversational context on sarcasm detection using deep-learning (BERT, BiLSTM) NLP models and ML classifier (SVM). They found that incorporating the last utterance in a dialogue significantly improved classifier performance on Twitter datasets, achieving an F-score of 0.743 with BERT. Exploring the distinction between intended and perceived sarcasm, Oprea and Magdy introduced the iSarcasm dataset, which consists of tweets labeled for sarcasm directly by their authors emphasizing the need for datasets that reflect the intended use of sarcasm to improve detection systems.
The introduction of the Transformer architecture and pre-trained language models such as BERT and RoBERTa significantly advanced NLP capabilities. These models utilized attention mechanisms to capture long-range dependencies in text, leading to state-of-the-art results in various tasks.
Beyond Transformer architectures, emerging methods provide complementary solutions. For example, knowledge-graph‐based architectures can improve stance detection tasks, and boundary‐aware LLM designs are increasingly valuable for few‐shot named entity recognition. Integrating both constituency and dependency parse information has proven beneficial for relation extraction.
Large Language Models (LLMs) like GPT-2 and GPT-3 expanded these capabilities by increasing model size and training data. GPT-3, with 175 billion parameters, demonstrated remarkable proficiency in zero-shot and few-shot learning scenarios, performing well on tasks it was not explicitly trained for.
Recent studies have explored LLMs in sentiment analysis and related tasks. Chang & Bergen investigated the use of GPT-3 for sentiment classification and found that it performed competitively with fine-tuned models on specific datasets. Similarly, Floridi and Chiriatti discussed the potential of GPT-3 in understanding and generating human-like text, highlighting its applicability in content analysis.
The incorporation of context-aware mechanisms, consideration of intended versus perceived meanings, and the use of multi-modal data represent critical steps toward improving model performance in complex NLP tasks. The development of domain-specific models like PoliBERTweet highlights the potential benefits of customizing language models to better capture specific content areas, such as political discourse. The integration of symbolic reasoning with deep learning in SenticNet 6 further highlights the importance of combining different AI approaches to enhance understanding and interpretation of subtle linguistic features.
However, challenges remain regarding the ethical and practical implications of relying on LLMs. Concerns include model bias, the interpretability of results, and the tendency of LLMs to produce plausible but incorrect or biased outputs. Additionally, studies have shown that while LLMs excel in language tasks, their performance in detecting sarcasm and nuanced emotions is inconsistent.
The consistency of LLMs over time is another area of interest. Although not updated by service providers, models that are prompted on different instances, may produce different outputs on the same input, raising questions about consistency in longitudinal studies. LLMs can be sensitive to input phrasing, leading to different interpretations based on slight changes in wording.
Human annotators have long been the gold standard in content analysis due to their ability to understand context, cultural references, and subtle language cues. Inter-rater reliability metrics such as Krippendorff’s alpha are used to assess consistency among human coders. Comparing LLM performance against human benchmarks is essential to evaluate their viability as substitutes or supplements in content analysis tasks.
While Large Language Models (LLMs) have demonstrated impressive capabilities, there is a notable lack of comprehensive evaluations comparing their performance to human annotators across multiple dimensions of latent content analysis. Existing studies often focus on single tasks or lack extensive statistical analysis of reliability and quality. Additionally, the consistency of LLMs over time and their inter-rater reliability in capturing complex linguistic features remain underexplored. To address these gaps this study formulates the following research questions:
RQ1: How reliably do LLMs compare to human annotators across multiple dimensions of latent content analysis?
Although the field of content analysis has advanced from manual coding schemes, through machine learning introduced algorithms capable of handling larger datasets with increased efficiency, and finally peaked with deep learning architectures, research is still needed especially in interpreting complex linguistic features such as sarcasm and nuanced emotions. On the other hand, other studies show that machines underperform in comparison to humans in certain tasks. One big gap in these studies is the lack of comparison with human annotations – to compare how LLMs and humans annotate complex linguistic features and whether humans are better at these tasks.
Previous studies have focused on specific aspects of content analysis, such as sentiment classification and sarcasm detection, but there is limited research on how the level of reliability between LLMs and humans varies across different dimensions. This is another gap to be addressed in this study.
RQ2: Are LLMs consistent over time when analyzing textual content?
Consistency over time is another underaddressed issue. Models used in different instances can create different outputs in different instances. LLM outputs can vary due to minor prompt modifications, raising concerns about time consistency.
RQ3: To what extent do LLMs provide analysis that is comparable to human analysis in terms of quality?
Human annotators have long been the gold standard in content analysis due to their ability to understand context, cultural references, and subtle language cues. However, the ability of LLMs to learn from the context is being examined, as well as their ability to produce human-like texts is rising, with an aim to replace human annotators with LLMs. Even though these studies are rising in numbers, quality check studies comparing humans and LLMs are still lacking, and this is another gap to be addressed in this study.
RQ4: How do inter-rater reliability and comparability vary across different LLM models?
Previous studies examined some LLM models’ success in annotation, but usually in one or two tasks, and usually using one or two models in this study we aim to close this gap by including multiple models and multiple tasks, and examine all their inter-rater reliability and consistency. Given the rapid development and diversity of LLM architectures—each trained on varying datasets and employing different model sizes—it’s crucial to understand whether these differences translate into variations in content analysis outcomes.
To evaluate the inter-rater reliability and quality of large language models (LLMs) in latent content analysis, we conducted a comparative study involving both humans and eight types of LLMs that each responded to presented queries to evaluate content by assigning values to statements. Our objective was to benchmark the performance of LLMs and humans across four key dimensions: sentiment, political leaning, emotional intensity, and sarcasm detection by performing (a) within (internal consistency) and (b) between analyses (comparison of performance).
The ethical approval was acquired from the Ethics Committee prior to this research. All methods were carried out in accordance with relevant guidelines and regulations.

翻译考点：
考点1：“resource intensive”推荐译为“资源密集型”
考点2：“societal needs”推荐译为“社会需要/需求”
考点3：“insight”推荐译为“了解/见解”
考点4：“benefit”推荐译为“受益于”
考点5：“inconsistent”推荐译为“时好时坏的”或“不稳定的”
考点6："constituency parse"推荐译为“组块分析”/“成分句法分析”