NU at WASSA 2024 Empathy and Personality Shared Task: Enhancing Personality Predictions with Knowledge Graphs; A Graphical Neural Network and LightGBM Ensemble Approach (2024)

Abstract: This paper proposes a novel ensemble approach that combines Graph Neural Networks (GNNs) and LightGBM to enhance personality prediction based on the personality Big 5 model. By integrating BERT embeddings from user essays with knowledge graphderived embeddings, our method accurately captures rich semantic and relational information. Additionally, a special loss function that combines Mean Squared Error (MSE), Pearson correlation loss, and contrastive loss to improve model performance is introduced. The proposed ensemble model, made of Graph Convolutional Networks (GCNs), Graph Attention Networks (GATs), and LightGBM, demonstrates superior performance over other models, with significant improvements in prediction accuracy for the Big Five personality traits achieved. Our system officially ranked 2 at the Track 4: PER track.

heading: 1 Introduction
text: Personality prediction is a complex task that benefits from understanding both the semantic content of text and the relationships between entities. Traditional machine learning models often fail to capture this relational information. To address this, we propose a novel ensemble approach integrating BERT embeddings, knowledge graph features, Graph Convolutional Networks (GCNs), Graph Attention Networks (GATs), and LightGBM. In this paper, we describe our participation in WASSA 2024 Shared Track 4: Personality Prediction (PER). This year’s Track 4, as outlined by (Giorgi et al., 2024), is similar to last year’s Shared Track 4 in terms of predicting the Big Five personality traits (OCEAN). However, unlike the 2023 session, where each essay writer was asked to complete the Ten Item Personality Inventory (Barriere et al., 2023), this year’s session does not require this step. Our method aims to enhance prediction accuracy
by leveraging both semantic and relational data. In recent years, psychologists have developed a number of personality-testing questions (Zhang et al., 2022).
The Big Five model, comprising Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism (OCEAN), is widely used for personality assessment (Barriere et al., 2023). Recent studies have shown the effectiveness of incorporating deep learning techniques in personality prediction (Mehta et al., 2020; Digman, 1990). Knowledge graphs (KGs) represent entities and their relationships, providing valuable contextual information (Peng et al., 2023). GNNs, particularly GCNs and GATs, can effectively process graph-structured data by capturing the structural relations between nodes (Zhang Si, 2019).
Graph Neural Networks (GNNs): Graph Neural Networks (GNNs) are employed in various domains such as social analysis, fraud detection (Akoglu et al., 2015), natural language processing, and computer vision due to their ability to capture structural relations between data, providing more insights compared to isolated data analysis (Zhang Si, 2019). Graph Convolutional Networks (GCNs) enhance this by aggregating information from neighboring nodes, enabling comprehensive extraction of interdependent data. BERT has been extensively used in several tasks to generate token or sentence representations enriched with prior knowledge (Osei-Brefo and Liang, 2022). Our main contributions for participating in the WASSA 2024 Shared Track 4: Personality Prediction (PER) are as follows:
• Integrating BERT embeddings with knowledge graph features.
• Development of an ensemble model that combines GCN, GAT, and LightGBM.
412
• Introduction of a novel loss function that combines Mean square error loss, Pearson correlation loss, and contrastive loss.

heading: 2 Related Work
text: Previous studies have explored the correlation between personality traits and empathy perception, highlighting the importance of agreeableness and conscientiousness in predicting empathy (Omitaomu et al., 2022; Melchers MC, 2016; Giorgi et al., 2024). Techniques such as text generation adversarial networks and multitask detection models have been employed to enhance personality prediction (Sun et al., 2018; Tu et al., 2022). Recent advancements include the use of dynamic deep graph convolutional networks and the integration of psychological language dictionaries with Transformer language models for improved personality detection (Yang et al., 2023; Kerz et al., 2022). Our approach builds on these methods by combining BERT embeddings, knowledge graphs, GCN, GAT, and LightGBM in a novel ensemble model.

heading: 3 Methodology
text: Figure 1 depicts the architecture of our proposed personality traits prediction system. As can be seen in figure 1, our methodology encompasses several key steps, which are:
• BERT Pre-training: A BERT pre-training is utilised to accurately represent the personality features extracted from the sentences in an individual’s essays.
• Knowledge Graph Integration: These sentence representations are then combined with vector representations derived from knowledge graphs, which include demographic features and the Interpersonal Reactivity Index (IRI).
• Graph Structure Processing: During the graph structure data processing, we leverage the comprehensive mapping and syntactic analysis capabilities of multi-layer neural networks, specifically Graph Convolutional Networks (GCN) and Graph Attention Networks (GTA), in conjunction with LightGBM.
• Personality Trait Modelling and Prediction: This integrated approach allows the joint modelling and prediction of the individual’s personality traits with high accuracy.

heading: 3.1 Feature Extraction
text: The feature extraction method employed combines BERT embeddings with knowledge graph embeddings to capture both semantic and relational information. The process involved is captured in Algorithm 1 presented in Appendix B.

heading: 3.2 Graphical Neural Network
text: In the OCEAN prediction task, a Graph Convolutional Networks (GCNs) and Graph Attention Networks (GATs) which are two advanced graph neural network models, were employed in conjunction with LightGBM to predict OCEAN traits. LightGBM, a gradient boosting framework, is used to complement the graph neural networks. It excels in handling large-scale data and provides efficient training with lower memory usage.

heading: 3.3 Mathematical Formulation for GCN and GAT
text: For a graph G = (V,E) with node features X: GCN Layer:
H(l+1) = σ ( D̃−1/2ÃD̃−1/2H(l)W (l) )
Where Ã = A + I is the adjacency matrix with added self-loops, D̃ is the degree matrix, W (l) are the trainable weights, and σ is an activation function. The adjacency matrix (Zheng et al., 2023) represents the connections between nodes in the graph. Each element in the matrix indicates whether a pair of nodes is connected, and the addition of self-loops ensures that each node is connected to itself. This is critical for the GCN since it allows the model to consider each node’s own features in addition to its neighbors’ features during convolution. The adjacency matrix produced is shown in Figure 2 in the Appendix A.
GAT Layer:
H (l+1) i = σ
  ∑
j∈N (i) αijWH
(l) j
 
where αij are the attention coefficients computed as:
αij = exp(LeakyReLU(aT [WH(l)i ||WH (l) j ]))∑
k∈N (i) exp(LeakyReLU(aT [WH (l) i ||WH (l) k ]))
with a as the attention mechanism’s weight vector.

heading: 3.4 Ensemble Prediction
text: The ensemble model combines predictions from GCN, GAT, and LightGBM using weighted average. The procedure involved is shown in Algorithm 4 in the Appendix B.

heading: 3.4.1 Proposed Loss Functions
text: The loss function used in our methodology is designed to enhance the learning process by incorporating Mean Squared Error (MSE) loss, Pearson correlation loss, and contrastive learning loss. Each component serves a specific purpose:
• MSE Loss: It focuses on reducing the prediction error by minimizing the difference between predicted and actual values.
• Pearson Loss: Acts as a regularizer to ensure a strong correlation between predictions and targets, enhancing the alignment of predicted and actual OCEAN scores.
• Contrastive Loss: It adds an additional layer of learning by emphasizing the relationships between pairs of examples, which is crucial for capturing subtle differences in text and the combined effects of various features.
This multi-faceted approach improves the overall learning process by balancing error minimization, correlation enhancement, and relationship learning. The proposed loss functions are detailed below:
Mean Squared Error (MSE) Loss:
`m = 1
N
N∑
i=1
(yi − ŷi)2
Contrastive Loss:
`c = 1
N
N∑
i=1
(1− yi) ·D2i + yi ·max(0,m−Di)2
where Di is the Euclidean distance between a pair of samples, yi is the binary label indicating if the samples are similar, and m is the margin.
Pearson Loss: The Pearson correlation coefficient between predictions ŷ and targets y is given by:
ρ = ∑N i=1(ŷi − ¯̂y)(yi − ȳ)√∑N
i=1(ŷi − ¯̂y)2 √∑N i=1(yi − ȳ)2
The Pearson loss `ρ is then given by:
`ρ = 1− ρ
Total Loss:
`T = 1
2 `m + λreg · `ρ + λc · `c
where N is the number of pairs, `m is the Mean Squared Error loss, `c is the Contrastive loss, and `ρ is the Pearson loss. The parameters λreg and λc are the regularization weights for the Pearson loss and Contrastive loss, respectively.

heading: 4 Experiments

heading: 4.1 Data Collection and Preprocessing
text: The dataset used in this study was the Track 4 dataset provided by the organizers, which was subsequently merged with the Task 3 dataset. This merging process involved integrating the essays of
each individual from Task 4, resulting in a comprehensive dataset. The final training and development data included features such as:
• Essays: Detailed personal essays written by the participants.
• Demographic Information: Gender, education, race, age, and income.
• IRI Features: Scores from the Interpersonal Reactivity Index.
• OCEAN Traits: Target scores for the OCEAN personality traits.
This enriched dataset provided a robust foundation for our feature extraction and model training processes.

heading: 4.2 Feature Extraction
text: • BERT Embeddings: We utilized the BERT model to generate embeddings from user essays. These embeddings captured the contextual information and semantic nuances of the text, offering a rich representation of the user’s language usage.
• Knowledge Graph (KG) Construction: For each user, a knowledge graph was constructed by extracting entities and relationships from their essays using spaCy. These entities and relationships were then represented in a directed graph using NetworkX.
• KG Embeddings: The Node2Vec algorithm was employed to generate embeddings for the nodes within the knowledge graph, capturing both structural and relational information.
• Combined Feature Vector: The BERT embeddings were combined with the node embeddings from the knowledge graph to form a comprehensive feature vector for each user.

heading: 4.3 Models Used
text: The models used are the Graph Convolutional Network (GCN) to capture the local neighborhood structure within the feature graph, the Graph Attention Network (GAT) that introduced attention mechanisms to allow the model to weigh the importance of different neighboring nodes, and the LightGBM model, which complemented the two graph models by providing robust predictions based
on the extracted features. These three models leveraged the strengths of each other to improve overall prediction accuracy as an ensemble strategy. The configuration of hyper-parameters for the proposed model is shown in Table 1.

heading: 4.4 Model Evaluation
text: The performance of our models was evaluated using the Pearson correlation coefficient for each OCEAN trait and the average Pearson correlation across all traits. This evaluation metric was chosen because it measures the linear correlation between predicted and actual values, providing insight into the model’s predictive accuracy.

heading: 4.5 Results and Discussion
text: The ensemble model is evaluated using the Pearson correlation coefficient. A significant improvement in prediction accuracy for the Big Five personality traits compared to baseline models was observed for the test data represented as Ensemble(b) model (unofficial test results) in Table 3. Tables 2 and 3 show the comparative performance of different models on the OCEAN traits for the validation set obtained from using 20% of the training dataset and all the samples of the test data provided by the organisers respectively. The ensemble model outperformed the baseline models, demonstrating the effectiveness of combining GNNs and LightGBM. Figures 3, 4 and 5 in Appendix C show the Loss and Pearson correlations plots per epoch for the GCN, GTA and Ensemble models, respectively. The integration of knowledge graph features and BERT embeddings proves to be particularly beneficial.
The main difference in the performance between Ensemble(a) model and Ensemble(b) model is the setting of hyper-parameters. We tuned the hyperparameters of Ensemble(b) model during the postsubmission stage, where different combinations of custom loss functions and optimal weights for the Ensemble models were explored. We selected the best setting of hyper-parameters for Ensemble(b) model. In the context of personality detection, GCNs effectively captured the relationships and contextual information between nodes by leveraging the graph’s topological structure and node characteristics. This capability significantly aided in the prediction of personality traits. The use of a fixed-weight matrix for convolution in GCNs ensured simplicity and scalability. Additionally, the integration of GCN, GATs and LightGBM enhanced this approach by incorporating attention mechanisms. These mechanisms allowed the model to weigh the importance of different neighbors when aggregating information. The ensemble approach’s success underscores the importance of combining different model strengths to achieve better prediction accuracy. The sensitivity analysis (detailed in Appendix D) further demonstrated the impact of different combinations of MSE, Pearson, and contrastive losses on the model’s performance, highlighting the optimal weights for each component.

heading: 5 Conclusion and Future Work
text: This paper has proposed an ensemble approach to leverage BERT embeddings and knowledge graph embeddings for GNNs and LightGBM that significantly enhances personality prediction. The introduction of a specialised loss function that combines MSE, Pearson correlation loss, and contrastive losses was crucial for balancing error minimiza-
tion, correlation enhancement, and the learning of relationships. Future work will involve the refinement of these methods with additional data sources to improve their performance.
Limitations
The process of extracting meaningful features from text using BERT and constructing knowledge graphs requires substantial computational resources. Additionally, the quality of the extracted features can vary depending on the preprocessing and entity extraction methods used, potentially impacting the model’s performance.

heading: A The Adjacency Matrix

heading: B Algorithms
text: Algorithm 1 BERT Embedding Extraction 1: Input: Text data T 2: Initialize BERT tokenizer and model 3: for each text t in T do 4: Tokenize t using the BERT tokenizer 5: Pass the tokenized text through the BERT
model 6: Extract the embeddings from the last hid-
den layer 7: end for 8: return embeddings E 9: Output: BERT embeddings E
Algorithm 2 Knowledge Graph Construction and Embedding
1: Input: Text data T , additional features F 2: for each text t in T do 3: Construct a knowledge graph G from t and F 4: Generate node embeddings using Node2Vec 5: Aggregate node embeddings to form a fixed-size graph embedding 6: end for 7: return embeddings K 8: Output: Knowledge graph embeddings K
Algorithm 3 Ensemble Learning: Training GCN, GAT, and LightGBM
1: Input: Training data D, validation data V 2: Initialize GCN and GAT models 3: for each epoch do 4: Train GCN on D and validate on V 5: Train GAT on D and validate on V 6: Compute training and validation losses 7: end for 8: Save the best performing GCN and GAT mod-
els 9: Initialize LightGBM models for each OCEAN
trait 10: for each trait do 11: Train the LightGBM model on D 12: Validate the model on V 13: Save the best performing LightGBM
model 14: end for 15: return trained GCN, GAT, and LightGBM
models 16: Output: Trained GCN, GAT, and LightGBM
models
Algorithm 4 Ensemble Prediction 1: Input: Models M , test data T 2: for each model m in M do 3: Generate predictions on T 4: end for 5: Average the predictions from all models 6: return ensemble predictions P 7: Output: Ensemble predictions P

heading: C Loss and Pearson Plots

heading: D Sensitivity Analysis of Loss
text: components

heading: Future Work
text: Future work will involve the refinement of these methods with additional data sources to improve their performance. Limitations
The process of extracting meaningful features from text using BERT and constructing knowledge graphs requires substantial computational resources. Additionally, the quality of the extracted features can vary depending on the preprocessing and entity extraction methods used, potentially impacting the model’s performance