SenticVec: Toward Robust and Human-Centric Neurosymbolic Sentiment Analysis (2024)

Abstract: Recent developments in natural language processing were enabled by deep neural networks, which excel in various tasks through strong data fitting and latent feature modeling abilities. However, certain challenges linked to deep nets and supervised deep learning deserve considerations, e.g., extensive computing resources, knowledge forgetting, etc. Previous research attempted to tackle these challenges individually through irrelative techniques. However, they do not instigate fundamental shifts in the learning paradigm. In this work, we propose a novel neurosymbolic method for sentiment analysis to tackle these issues. We also propose a novel sentiment-pragmatic knowledge base that places emphasis on human subjectivity within varying domain annotations. We conducted extensive experiments to show that our neurosymbolic framework for sentiment analysis stands out for its lightweight nature, robustness across domains and languages, efficient few-shot training, and rapid convergence.

heading: 1 Introduction
text: Deep neural networks have demonstrated remarkable capabilities in the natural language processing (NLP) domain because of their strong capacity to capture latent features and fit data. Currently, the advancement of large language models (LLMs) shows that when the neural networks become very large and deep, sufficient pre-training grants them impressive capability, making them a staple as foundation models for a variety of tasks (Bommasani et al., 2021). Despite the fascination the academic community holds for LLMs (Mao et al., 2024a), a pressing question remains: is the relentless pursuit of model enlargement truly the paramount focus for NLP? As models become larger and deeper, many machine learning problems arise. For instance, larger models normally require more computing resources to train, resulting in more carbon emission (Patterson et al., 2021).
Traditional supervised learning, one of the most significant deep learning paradigms, suffers from issues including knowledge forgetting (Kemker et al., 2018), extensive data annotation efforts (Aggarwal et al., 2018), weak cross-domain (Singhal et al., 2023) and cross-lingual (Wang et al., 2021a) adaption, and data-centric learning pipelines (Singh, 2023). For NLP, the need for deep neural networks is usually due to the complexity of modeling language, i.e., syntax (Zhang et al., 2023a) and semantics (Mao et al., 2023a) exhibiting different patterns in different contexts.
As such, the learning of complex word sequence patterns is achieved by making the model deeper and adding different learning mechanisms (Hochreiter and Schmidhuber, 1997; Vaswani et al., 2017). Furthermore, models trained with standard annotation sets where labels were aggregated based on the majority opinion of annotators could potentially disregard nuanced individual variations (Zhu et al., 2024). In subjective tasks such as affective computing, a uniform decision-making mechanism may struggle to adequately cater to individual subjective experiences. The judgment of sentiment and emotional states from different people can vary in different contexts or domains, necessitating humancentric systems.
In this research, we investigate whether the coupling of syntactic and pragmatic features can alleviate the need for semantic features, thereby significantly reducing the number of model parameters needed to retain semantic knowledge (Yeo et al., 2024). Hence, such an approach potentially tackles the challenges prevalent in deep neural networks, including heavy reliance on extensive downstream training data and computing resources (Anas et al., 2024), slow training rates, limited adaptability across domains and languages, knowledge forgetting, and the ignorance of humans’ subjectivity.
4851
Previous sentiment analysis works target to address one of these issues with different techniques, e.g., few-shot learning for eliminating the hungry of label data (He et al., 2022b); pruning neural networks or using lightweight encoders to reduce the number of learning parameters and computing resources (Ni et al., 2023); using pre-training (Devlin et al., 2019; Mao et al., 2024b) to improve the generalization of a model in different domains and languages; leveraging continues learning techniques to mitigate the knowledge forgetting (He et al., 2022a). However, these approaches do not instigate fundamental shifts in the deep learning paradigm, nor do they comprehensively resolve all the previously mentioned issues.
To alleviate the reliance on large neural networks and downstream training data while improving model performance in cross-domain and cross-language tasks, we propose a neurosymbolic method and a knowledge base for sentiment analysis1. Our hypothesis is that in affective computing, models can reduce the need for semantic learning by learning pragmatic knowledge and syntactic patterns, thereby reducing the number of learning parameters. and mitigating issues associated with deep neural networks.
The advantage is threefold: a) The exclusion of semantic learning substantially reduces the parameters of neural networks as it obviates the necessity for the model to grasp the meaning of an extensive vocabulary across diverse contexts. b) Incorporating multi-dimensional symbolic representations (sentiment-pragmatic knowledge) from our knowledge base enables the construction of a humancentric system that circumvents the knowledgeforgetting issue, where different dimensions signify subjective judgments from different contributors, offering the flexibility to augment the symbolic representations with new dimensions acquired from new data without changing the existing elements. c) By leveraging the combination of syntactic and pragmatic features as input, it is possible to alleviate issues related to learning and inference across diverse domains and languages, primarily stemming from semantic variations in different linguistic environments. This aligns with our plans to enhance current sub-symbolic AI approaches through task decomposition, panalogy, symbol grounding, and more (Cambria et al., 2023).
1 https://github.com/senticnet/senticvec
Our neurosymbolic framework for sentiment analysis consists of two components, a knowledge base and a neurosymbolic model. The former, termed SenticVec, consists of sentiment-pragmatic representations for 388,158 lexical units, i.e., tokens. Each representation is in vector form, where each element is a sentiment score ranging from -1 (negative) to 1 (positive), pre-trained from a public sentiment dataset. The sentiment score is automatically generated via an explainable encoder (Han et al., 2022) that signifies the contribution of each lexical unit to the sentiment prediction.
Lexical units that contribute more will receive more extreme scores. The sentiment vector of a lexical unit consists of multiple sentiment scores, learned and computed using multiple labeled sentiment datasets. The intuition is that different annotators, annotation task instructions, and text domains can yield different annotation outcomes. For a human-centric system with cross-domain robustness, it is essential to acknowledge these sentimental variations; therefore, we opt for multidimensional representations, integrating subjectivity instead of relying on conventional unified sentiment scores to depict sentiment-pragmatic features.
Our neurosymbolic model consists of a 3-layer multi-layer perceptron (MLP) to learn the pragmatic representations and PoS patterns. We use an existing PoS tagger to generate PoS tag sequences from the inputs. The motivation is that 1) the size of PoS tags is smaller than the vocabulary size of words; 2) different languages can share the same PoS tag set; 3) SenticVec provides useful sentiment-pragmatic representations and can be easily mapped to other languages using bilingual dictionaries. Thus, we can use a shallow neural network by leveraging symbolic representations (e.g., PoS tags and SenticVec) to process multilingual sentiment analysis tasks. This is entirely different from previous learning paradigms that were proposed for addressing the aforementioned deep learning issues. We performed thorough experiments to demonstrate the superiority of our approach in conventional supervised learning, cross-domain inference, and multilingual learning and inference for sentiment analysis. Our method achieves comparable performance to RoBERTabase (Liu et al., 2019) while requiring 1/7723 of its training parameters, and enables faster training in the conventional supervised learning evaluation.
In cross-domain inference evaluation, the SenticVec neurosymbolic model demonstrates overall better performance than RoBERTa. In the fewshot multilingual learning evaluation, SenticVec surpasses multilingual Pre-trained Language Models (PLMs) on three languages. In cross-lingual inference evaluation, SenticVec obtains comparable performance to PLMs. Furthermore, incorporating SenticVec knowledge into multilingual PLM consistently brings accuracy gains in all settings.
The contributions of this work include: (1) We develop a novel knowledge base, incorporating sentiment-pragmatic representations that prioritize human subjectivity in annotating diverse corpora from varying domains while ensuring retention of original knowledge with the influx of new labeled data. (2) Our novel neurosymbolic framework for sentiment analysis stands out for its lightweight nature, robustness across domains and languages, efficient few-shot training, and rapid convergence.

heading: 2 Related Work
text: Deep-learning systems have almost become the fixture of NLP tasks. Although they achieved impressive accuracy in a variety of applications, they are not without their vices. Randomly-initialized neural networks suffer from knowledge forgetting (Kemker et al., 2018; Ramasesh et al., 2021). To resolve this, pre-training is employed to help with their ability for generalizing and transferring knowledge. However, prominent PLMs such as RoBERTa and GPT (Radford et al., 2019; Brown et al., 2020) consist of a large number of parameters trained with enormous data, which consumes substantial computational resources (Bender et al., 2021), and leads to concerns about carbon emissions and environmental impact (Patterson et al., 2021; Bannour et al., 2021). On the other hand, lightweight zero-shot and few-shot learning models still face limitations in cross-domain and crosslingual processing, e.g., negative migration, overfitting, and underfitting (Yang, 2021; Nozza, 2021). Neurosymbolic systems for sentiment analysis leverage both subsymbolic and symbolic approaches, e.g., coupling neural networks and knowledge graphs (Cambria et al., 2024; Kocoń et al., 2022; Zhang et al., 2023b). However, existing sentiment knowledge bases (Wiebe et al., 2005; Strapparava and Valitutti, 2004; Baccianella et al., 2010; Cambria et al., 2024) assign a singular score to a lexical unit, disregarding the inherently
differing perceptions people have to subjective contents (Pavlick and Kwiatkowski, 2019; Beck et al., 2020; Troiano et al., 2021), hence failing to capture the human uncertainty and subjectivity. Additionally, these knowledge bases are not domain-specific and mostly monolingual, hence less adaptable in cross-domain and multilingual settings (Hung and Chen, 2016). Furthermore, existing neurosymbolic systems mostly take a knowledge-based approach (Du et al., 2023), thus still dependent on heavy neural networks. The PLMs are the backbone, while the symbolic representations provide complementary knowledge, e.g., sense polarity (Baccianella et al., 2010), facts (Wang et al., 2021b), and commonsense (Cambria et al., 2024) that are not directly learnable from datasets by neural networks. As a result, these neurosymbolic methods did not fundamentally address some of the issues that deep-learning-based methods commonly encounter.

heading: 3 Methodology

heading: 3.1 SenticVec Knowledge Base Development
text: In this section, we will describe how SenticVec is constructed. First, we construct a sentiment classifier with explainable attention for determining the importance of each token to the model’s sentiment prediction. The intuition is that, the more influence a token has on the sentiment inference of the input, the stronger its sentiment intensity is. As a result, the sentiment score of a token is determined by the ground truth sentiment of the input sentences it appears in, and its attention weights in these sentences. Such sentiment scores can be categorized as pragmatic knowledge, because they embody the likelihood of a lexical unit influencing the overall sentiment of a sentence.
Unlike syntactic and semantic knowledge, which pertain to structural characteristics and meanings of words respectively, this pragmatic aspect of sentiment scores demonstrates their relevance in understanding how specific words contribute to the overall sentiment conveyed in a given context. Second, we train the classifier on 19 corpora from different domains separately to acquire 19 sentiment scores for each token to form both general and PoS-tagged vector representations. We apply an averaging technique to fill in sentiment scores when a token is not present in one of the corpora. Additionally, we translate the English SenticVec lexicon into other
languages using off-the-shelf machine translation systems to expand SenticVec for multilingual processing. The resulting SenticVec knowledge base contains entries for 388,158 lexical units.

heading: 3.1.1 Sentiment Score Computation
text: To obtain the sentiment score of a word from a given corpus, we need to measure how salient it is in terms of sentiment for the sentences it appears in. Hence, we design a simple attention-based sentiment classifier consisting of an embedding layer, an explainable attention module, and two Feed Forward Network (FFN) layers for output. The model is trained with a given corpus so that the attention module learns to assign each token with a weight that reflects its salience to sentence sentiment.
To align with our objective of constructing a lexicon, we adopt GloVe embeddings (Pennington et al., 2014) for word representations. This is motivated by the fact that in a static embedding method, the representation of a word is the same for all of its occurrences in a corpus. We denote the GloVe embedding matrix of a given lemmatized input sequence w = (w1, w2, . . . , wL) as G = (g1, g2, . . . , gL).
To acquire attention weights that effectively indicate word salience (Serrano and Smith, 2019), we employ an explainable attention module called Hierarchical Attention Network (HAN, Han et al., 2022), which encodes hidden states with multiple non-linear projections and ranks the most influential tokens based on attention weights. The attention module consists of I HAN layers:
qi, ai = HANi(G), (1)
where vector qi and ai are the yielded hidden state and attention weights at the i-th HAN layer.
The query qI produced by the last layer HANI is then fed into two consecutive FFN layers to produce the sentiment prediction ŷ:
h = ReLU(FFN c1(qI)), (2) ŷ = Softmax(FFN c2(h)). (3)
The model parameters are optimized via crossentropy loss
L = CrossEntropy(ŷ, ỹ). (4)
After sufficient training, the sentiment score sl of a token wl in the context of the input sentence w is computed as the product of its min-max normalized
attention weight produced by the last attention layer HANI of the trained classifier, and the groundtruth sentiment label ỹ of w:
sl = aI,l −min(aI)
max(aI)−min(aI) ỹ, (5)
so that the score sl falls within the range of [−1, 1].

heading: 3.1.2 Sentiment Vector Construction
text: To construct vector representations that reflect the varying sentiment intensities of words in different domains from different annotators, we train the classifier in the previous section on N ∈ {1, 2, . . . , n, . . . , 19} corpora, separately. Our selection of corpora covers different domains including Twitter, product reviews, movie reviews, finance, news, etc., with several corpora for each domain to accommodate human’s innate disagreement and uncertainty in sentiment annotation. The full list of the corpora used can be found in Table 7 in Appendix C. On each corpus, the classifier is trained on the training set until the accuracy of the validation set no longer improves. To avoid the influence of negation on the connection between sentiment and word salience, we prune out all samples that contain not and never.
Given a token appears in the training set of the n-th corpus for T times, its general sentiment score sgn without PoS distinction is computed as
sgn = sn,1 + sn,2 + · · ·+ sn,T
T , (6)
where the sentiment score sn,t in the t-th sentence is computed via Equation 5. As a result, the general sentiment vector representation sg of the word can be assembled as
sg = (sg1, s g 2, . . . , s g N ). (7)
We manually set the representations of the abovementioned negation words to vectors consisting of -1s. Alternatively, if the token is not present in the n-th corpus, we assign a placeholder value N.A. to sgn. If the token appears M times in the training set of the n-th corpus with the PoS tag p, its sentiment score spn under the PoS p is calculated similarly to Equation 6 by averaging its sentiment scores in all M instances. Its PoS-p-annotated sentiment score sp is constructed using the same method as Equation 7. A sample of SenticVec entries associated with a token is illustrated in Figure 1.
After constructing vectors for all tokens in the lexicon, we fill in the N.A. values using an averaging technique. For entries without PoS tags, assume sc is the SenticVec vector of the token c, and scn = N.A.. We find an anchor token b in the lexicon that satisfies three conditions: a) the cosine similarity of their GloVec embeddings is the largest, b) the n-th value of the anchor token’s vector sbn is not N.A., and c) in at least one dimension, the elements in both sc and sb are not N.A.. Subsequently, we fill in the value of scn as
scn = s b n −
(sbd1 − scd1) + · · ·+ (sbdJ − s c dJ )
J . (8)
J is the number of dimensions, where the values in both vectors are not N.A., and {d1, d2, . . . , dJ} is the set of indices of these non-N.A. dimensions. The motivation for Equation 8 is to consider the sentiment shift from different corpora. We limit the value of scn to the range of [-1,1]. A simplified example is shown in Figure 2. For entries with PoS tags, we follow the same procedure except the chosen anchor entry must also have the same PoS tag. We only update the lexicon after all N.A. values are filled in.

heading: 3.1.3 Multilingual Lexicon Acquisition
text: To acquire a multilingual lexicon, we first employ Open Multilingual WordNet (OMW) (Bond and Foster, 2013) to map the lexicon into Spanish, French, and Italian, using PoS annotations as constraints when applicable. We choose the highest ranking lemma from the top synset. For tokens that are not present in OMW, we use off-the-shelf machine translation systems word2word (Choe et al.,
2020) and Googletrans2 for translation. In the case where more than one tokens are translated into the same token in the target language, we average the vectors of the original tokens as the sentiment vector of the token in the target language. When translating a PoS-annotated entry, we translate the token and keep the PoS tag as it is.

heading: 3.2 SenticVec-Based Neurosymbolic Model
text: Utilizing the constructed SenticVec knowledge base, we propose a lightweight, neurosymbolic model that relies on the PoS tag sequence of the input sentence for syntactic features, and the corresponding SenticVec representations for pragmatic features, as illustrated in Figure 3.
First, given an L-long sentence, we employ an off-the-shelf PoS tagger to obtain the PoS sequence p = (p1, . . . , pL). An embedding layer is used to obtain PoS representations:
V = Embed(p). (9)
The sentence’s general SenticVec representation Sg and PoS-annotated representation Sp are constructed by finding the corresponding SenticVec entries to each lemmatized token, which are concatenated to form S. If a token has no corresponding entry, it is represented by a vector of zeros.
2 https://github.com/ssut/py-googletrans
We then concatenate (⊕) V with SenticVec representation S, and pass it to a 3-layer MLP.
H = ReLu(FFNo(V ⊕ S))×2, (10) ŷ = Softmax(FFNo3 (avg(H))), (11)
where avg denotes averaging along the sequence length dimension. The model is optimized via cross-entropy loss as in Equation 4.

heading: 4 Experiments

heading: 4.1 Datasets
text: The Sentiment140 dataset (Go et al., 2009) (Senti.140) consists of 1.6M tweets and their corresponding sentiment labels (positive or negative). We split 10% of the original training set as the validation set. We removed the neutral samples in the original test set, since the training set does not include neutral samples. The Review dataset originates from the Generic Sentiment dataset3, covering reviews from multiple domains. To align with the other datasets for cross-domain experiment setting, we remove all samples labeled as neutral from this dataset. The Microblog dataset (Cortis et al., 2017) originates from SemEval-2017 Task 5, where each microblog is given a polarity score ranging between -1 and 1. We repurpose it by labeling the samples with scores below 0 as negative, the rest as positive. Since the released test set does not provide ground truth labels, we split 20% of the original training set as our test set. We use the original validation set. The Multilingual Tweet dataset (Barbieri et al., 2021) contains 3,033 tweets per language, each labeled as positive, negative, and neutral. For a few-shot learning setting, we re-split each language subset into 5%/5%/90% for training, validation, and testing.

heading: 4.2 Baselines
text: We include the following baselines. Non-pretrained is a 3-layer MLP model using word embeddings trained from scratch. The embedding size is the same as SenticVec. GloVe is a 3-layer MLP model using GloVe embeddings. RoBERTaEmb is a 3-layer MLP model using RoBERTa-base embeddings. RoBERTa consists of a RoBERTabase encoder and a FFN output layer. XLMRoBERTa (Conneau et al., 2019) consists of an XLM-RoBERTa-base encoder and a FFN output
3 https://kaggle.com/datasets/akgeni/
generic-sentiment-multidomain-sentiment-dataset
layer. The XLM-RoBERTa is a multilingual PLM. XLM-Twitter is the same as XLM-RoBERTa but the encoder is XLM-Twitter-base (Barbieri et al., 2021) pretrained with a large multilingual Twitter corpus. To show that SenticVec can also be effective as a complementary knowledge base, we further propose the following knowledge-based systems. RoBERTa+SenticVec concatenates the pooler outputs of the RoBERTa-base encoder with SenticVec representation (S) averaged along the sequence dimension, and feeds them into a FFN for output. XLM-RoBERTa+SenticVec use the same architecture as RoBERTa+SenticVec, except the encoder is XLM-RoBERTa-base.
Table 2 shows the parameter sizes of compared models, where SenticVec and Non-pre-trained have the smallest parameter sizes, constituting approximately 3/5 of the parameter size of the Glove model and approximately 3/10 of the parameter size of the RoBERTaEmb model. Conversely, PLMs significantly surpass SenticVec in terms of parameter size by several orders of magnitude.

heading: 4.3 Setup
text: We use Adam optimizer (Kingma and Ba, 2014). We set the learning rate to 1e-3 which decays by 0.5 every 10 epochs. The models are trained with a batch size of 20 for 50 epochs with early stopping based on validation accuracy. We implement I = 2 layers of HAN for SenticVec construction. We use spaCy4 for PoS tagging and lemmatization. We use the 6B GloVe. The hidden dimension for all MLPs is set to 100. Results are averaged from 5 runs.

heading: 5 Results
text: We evaluate our neurosymobic method in conventional supervised learning and cross-domain inference (Section 5.1); cross-lingual inference (Section 5.2) and multilingual learning (Section 5.3). Our ablation analysis (Appendix A) verifies that as the dimensionality of SenticVec increases, our model can achieve better accuracy. We further compare the convergence curves of our method and the baselines (Appendix B), demonstrating the training speed advantage inherent in our method.

heading: 5.1 Supervised and Cross-domain Evaluation
text: For conventional supervised learning evaluation, we use the training and testing sets from the same datasets. As seen in Table 3 (gray), SenticVec achieves comparable performance as RoBERTa but with much fewer learnable parameters (Table 2). When evaluated using F1 score, SenticVec exceeds RoBERTa on the Senti.140 and Review datasets. This compelling evidence substantiates the potential of a shallow neural network to attain performance comparable to a deep neural networkbased PLM, through the strategic integration of efficient pragmatic features (the SenticVec knowledge base), alongside syntactic features (PoS tags). Training the SenticVec model uses much less computational resources, given it is merely a 3-layer MLP model. Compared to other embedding-based methods, i.e., non-pre-trained embedding baseline, Glove-based model, and RoBERTaEmb-based model, the advantage of SenticVec is even more prominent. It shows the utility of SenticVec compared to other semantic embeddings, especially in the context that SenticVec representations have a lower dimensionality than the pre-trained embeddings. Finally, RoBERTa+SenticVec achieves the best performance, because the in-domain finetuning with deep neural networks together with
4 https://spacy.io/usage/linguistic-features
SenticVec allows the model to learn more complementary domain-specific features. However, this finding does not hold for cross-domain inference, because cross-domain inference may prefer robust general features. Our cross-domain evaluation tasks are conducted using Senti.140, Review, and Microblog. A model is trained with one dataset and evaluated with the rest. Results are shown in Table 3. When using Senti.140 as the training set, SenticVec achieves the best crossdomain performance. Considering that Senti.140 has the biggest training set, we believe that, given sufficient training samples, SenticVec can achieve better results than PLM-based methods. Although RoBERTa+SenticVec is better than SenticVec on Senti.140 testing set, it performs worse than SenticVec during cross-domain evaluation.
This suggests that the augmented semantic features provided by RoBERTa do not synergize effectively with the syntactic and pragmatic features in sentiment analysis tasks when sufficient training data is available. SenticVec knowledge has a higher utility in general feature representation, compared to the features learned by RoBERTa. This can be explained by the well-known overfitting issue of deep neural networks (Rice et al., 2020) when given a large amount of domain-specific training data. Deep neural networks may overfit in the data distribution of a specific domain. As a result, the learned features struggle to generalize to other domains. Unsurprisingly, SenticVec exceeds all embedding-based methods with large margins.
When Review is used as the training data, SenticVec achieves better average cross-domain performance than RoBERTa by a small margin. RoBERTa+SenticVec yields the best average performance, suggesting that the ensemble strategy is more effective on a middle-sized training set. SenticVec still has significant advantages over other embedding-based methods. Lastly, Microblog is used for training, and Senti.140 and Review for testing. SenticVec obtains the highest F1 scores, and RoBERTa+SenticVec outperforms RoBERTa on both metrics, indicating that the SenticVec knowledge helps with model robustness against domain changes and class imbalance. It also suggests that with about 1,000 training instances, employing deep nets with semantic learning can yield overall superior results compared to our shallow neural network based on syntactic and pragmatic features.
To sum up, compared to the embedding-based methods, SenticVec obtains superior performance across all cross-domain evaluation tasks. It shows that the combination of syntactic and pragmatic information achieves better utilities than using semantic information in shallow neural network-based sentiment analysis. Furthermore, when sufficient training data is available, SenticVec outperforms RoBERTa. With relatively small training data, employing both RoBERTa and SenticVec representations can yield improved accuracy. The robustness of our proposed SenticVec-based method can be attributed to its multi-dimensionality. Rather than combining all pre-training data to compute a single sentiment score for a lexical unit, we uphold the importance of diverse human annotations across various annotation tasks and domains, forming multi-dimensional representations.
We regard each pre-training dataset as a repository of knowledge contributed by a specific group of annotators with their subjective judgments, potentially tailored to specific domains. Given a test set from a separate group of annotators, our framework enables models to judiciously select and learn from the sentiment knowledge aligning best with the domain preferences of the test set. Thus, SenticVec can achieve robust inference for data from other domains. The utility of SenticVec, when trained on a large dataset, holds practical importance for cross-domain inference. Since SenticVec relies exclusively on PoS and pragmatic knowledge as input, it can alleviate the influence of semantic variations stemming from different domains.
The effectiveness of SenticVec is predominantly contingent on the PoS patterns it has learned through training. A large dataset likely encompasses a variety of PoS patterns, thereby enabling effective training of the SenticVec model even though the samples are from the same domain. This could potentially provide insight into an effective approach for fine-tuning neurosymbolic models across different domains: Annotated data should be chosen with the objective of enhancing the diversity of PoS patterns in the fine-tuning dataset. Given the relatively limited vocabulary of PoS tags, a viable strategy involves selecting a subset of data that exhibits a wide range of PoS patterns. Annotators can then focus their efforts on annotating this carefully chosen subset of data. Further investigation into this strategy will be a focal point of our future research.

heading: 5.2 Cross-lingual Evaluation
text: In this section, we conduct cross-lingual inference where models trained with few-shot English data, and tested in Spanish, French, and Italian, using the datasets, developed by Barbieri et al. (2021). Table 4 shows that SenticVec exceeds baselines on average. The slightly inferior performance in French compared to the baselines might be because French is less syntactically similar to English. It indicates that the syntactic and pragmatic patterns learned in English can be applied to other languages to some extent. XLM-RoBERTa+SenticVec outperforms the baselines on all languages, suggesting that sentiment concepts learned from the SenticVec knowledge base transfer well on cross-lingual tasks, hence able to bring extra performance gains to PLMs. To sum up, SenticVec is a strong competitor to fine-tuning-based multilingual PLMs in few-shot multilingual and cross-lingual evaluation tasks, albeit being much more lightweight. This is attributed to the advantage of our novel symbolic representations and shallow neural networks.
The PoS tags possess a limited vocabulary and the capability to break language barriers, thus applicable across diverse languages. The shallow neural networks can be easily trained with few-shot data and limited computational resources because of the small PoS tag vocabulary and the limited learnable parameters. The model only needs to learn the dependency between unified PoS tags rather than that between words from different languages.
SenticVec knowledge base can provide useful pragmatic knowledge for sentiment analysis in other languages. The sentiment polarities of emotional concepts, e.g., “happiness” and “sadness”, are generally consistent across languages. Hence, the sentiment-pragmatic knowledge can be easily broadcasted to other languages via the mappings of multilingual lexicons, delivering robust performance in multilingual sentiment analysis. On the other hand, researchers also identified nuanced semantic differences in emotional words (Zhang et al., 2024). Thus, making appropriate SenticVec adjustments for non-English texts is also recommended.
The robust performance observed in crosslingual and few-shot sentiment analysis tasks holds practical significance. Leveraging languages with rich labeled data such as English to pre-train a model allows for its subsequent fine-tuning with a limited set of labeled data in the target language. Another notable application involves utilizing a well-trained model for predicting labels in a zeroshot setting for a language that lacks labeled data entirely. We will study them in future work.

heading: 5.3 Multilingual Evaluation
text: Table 5 shows the results of training and inference with the same target languages, i.e., with the English, Spanish, French, and Italian datasets under a few-shot setup, where 152 samples are used for training. As shown, the difference between the baselines’ best and worst F1 scores are ∆ = 27.27% and ∆ = 20.38% for XLM-RoBERTa and XLM-Twiter, respectively. In contrast, SenticVec achieves an averaged F1 score 5.78% higher than the best baseline XLM-RoBERTa, with ∆ =
10.78%. The higher average F1 score and minimal performance disparity observed across the four languages confirm that our model is more robust in few-shot and multilingual sentiment analysis. Introducing additional XLM-RoBERTa features upon SenticVec knowledge brings extra gains compared to the SenticVec model in some cases, and propels XLM-RoBERTa+SenticVec to exceed the PLM baselines in all target languages.

heading: 6 Conclusion
text: We propose a novel neurosymbolic model and a sentiment-pragmatic knowledge base, SenticVec. SenticVec consists of multi-dimensional sentimentpragmatic representations, making it humancentric and cross-domain efficient. We propose to use PoS tags together with SenticVec for sentiment analysis. The streamlined PoS vocabulary as input markedly reduces the need for deep neural networks and overcomes language barriers. Consequently, employing a shallow neural network for sentiment analysis in multilingual learning and cross-lingual inference scenarios becomes viable, substantially decreasing computational demands. The proposed framework can be adapted to other text classification tasks with multiple labeled datasets, e.g., emotion detection (Mao et al., 2023b), depression detection (Han et al., 2022), and topic classification (Duong et al., 2023), which will be examined in our future works.
Limitations
The proposed framework have the following limitations. First, the quality of non-English entries in the knowledge base would be affected by the capability of the selected machine translation systems. Second, our proposed model is only tested on Latin alphabetic languages. It might not work as well on languages with more complex segmentation.
Ethics Considerations
The SenticVec knowledge base and involved models are based on public corpora that do not contain private data nor offensive content.

heading: Acknowledgments
text: This research/project is supported by the Ministry of Education, Singapore under its MOE Academic Research Fund Tier 2 (STEM RIE2025 Award MOE-T2EP20123-0005), as well as supported by Alibaba Group and NTU Singapore.

heading: A Ablation Analysis
text: We conduct ablation analysis to show that the integration of new knowledge, i.e., more labeled datasets, can improve model performance on conventional supervised learning, cross-domain inference, and cross-lingual inference tasks in sentiment analysis. We used the first n dimension of SenticVec and the proposed shallow neural network to examine these tasks, where n ∈ {1, 5, 10, 15, 19}. n = 19 denotes the full dimension of SenticVec. Table 6 shows that generally as dimensionality increases, the neurosymbolic model achieves higher accuracy and F1 scores across all the examined tasks, among which the full dimensionality setup is the best. Given our method of integrating new knowledge into the knowledge base, which involves expanding the dimensionality of vector representations instead of modifying existing elements, we believe that SenticVec has the potential to alleviate the problem of knowledge forgetting often observed in deep neural networks.

heading: B Convergence Curve
text: As shown in Figure 4, SenticVec demonstrates superior and quicker convergence on Senti.140 compared to both shallow and deep neural network baselines. Conversely, RoBERTa+SenticVec outperforms RoBERTa in both rate and extent of convergence. This observation highlights the efficacy of the sentiment representations derived from the SenticVec knowledge base, expediting convergence process in both shallow and deep neural networks. Consequently, the lightweight SenticVec neurosymbolic model can be trained with significantly reduced computational resources and subsequent carbon emissions in downstream applications.

heading: C Datasets for Knowledge Base Construction
text: The datasets used for SenticVec knowledge base construction, their splits, and statistics are shown in Table 7.

heading: Future Work
text: , 2023), which will be examined in our future works. Limitations
The proposed framework have the following limitations. First, the quality of non-English entries in the knowledge base would be affected by the capability of the selected machine translation systems. Second, our proposed model is only tested on Latin alphabetic languages. It might not work as well on languages with more complex segmentation