<figure><img src="image_1.png" /></figure>

HAL
open science

# Churn Prediction via Multimodal Fusion Learning: Integrating Customer Financial Literacy, Voice, and Behavioral Data

David Hason Rudd, Huan Huo, Md. Rafiqul Islam, Guandong Xu

<figure><img src="image_2.png" /></figure>

## To cite this version:

David Hason Rudd, Huan Huo, Md. Rafiqul Islam, Guandong Xu. Churn Prediction via Multimodal Fusion Learning: Integrating Customer Financial Literacy, Voice, and Behavioral Data. The International Conference on Behavioural and Social Computing, Cyprus University, Oct 2023, Larnaca, Cyprus. hal-04320145

HAL Id: hal-04320145

https://hal.science/hal-04320145v1

Submitted on 4 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Churn Prediction via Multimodal Fusion Learning:
Integrating Customer Financial Literacy, Voice, and
Behavioral Data

1st David Hason Rudd

Faculty of Engineering and IT

The University of Technology Sydney

Sydney, Australia

david.hasonrudd@uts.edu.au

2nd Huan Huo

Faculty of Engineering and IT

The University of Technology Sydney

Sydney, Australia

huan.huo@uts.edu.au

3rd Md Rafiqul Islam

Information Systems (Data Analytics)

Australian Institute of Higher Education (AIH)

Sydney, Australia

r.islam@aih.edu.au

4rd Guandong Xu

Faculty of Engineering and IT

Advanced Analytics institute (AAi)

Sydney, Australia

guandong.xu@uts.edu.au

*Abstract*—In today's competitive landscape, businesses grapple with customer retention. Churn prediction models, although beneficial, often lack accuracy due to the reliance on a single data source. The intricate nature of human behavior and high-dimensional customer data further complicate these efforts. To address these concerns, this paper proposes a multimodal fusion learning model for identifying customer churn risk levels in financial service providers. Our multimodal approach integrates customer sentiments, financial literacy (FL) level, and financial behavioral data, enabling more accurate and bias-free churn prediction models. The proposed FL model utilizes a SMOGN-COREG supervised model to gauge customer FL levels from their financial data. The baseline churn model applies an ensemble artificial neural network and oversampling techniques to predict churn propensity in high-dimensional financial data. We also incorporate a speech emotion recognition model employing a pre-trained CNN-VGG16 to recognize customer emotions based on pitch, energy, and tone. To integrate these diverse features while retaining unique insights, we introduced late and hybrid fusion techniques that complementary boost coordinated multimodal co-learning. Robust metrics were utilized to evaluate the proposed multimodal fusion model and hence the approach's validity, including mean average precision and macro-averaged F1 score. Our novel approach demonstrates a marked improvement in churn prediction, achieving a test accuracy of 91.2%, a Mean Average Precision (MAP) score of 66, and a Macro-Averaged F1 score of 54 through the proposed hybrid fusion learning technique compared with late fusion and baseline models. Furthermore, the analysis demonstrates a positive correlation between negative emotions, low FL scores, and high-risk customers.

***Index Terms***—Churn prediction, multimodal learning, feature fusion, financial literacy, speech emotion recognition, customer behavior

# I. INTRODUCTION

Understanding and predicting customer churn has become paramount for long-term success in an increasingly competitive business landscape [1]. Customer churn, or customer attrition, directly affects a company's revenue and growth

potential, making it essential to be accurately predicted and prevented [2]. Furthermore, the human cognitive system is inherently susceptible to cognitive biases, such as anchoring bias, the availability heuristic, and the bandwagon effect [9]. These biases can significantly skew customer's feeling and decision-making towards products and services. Misguided customer interactions or suboptimal presentations of complex financial products can trigger these biases, prompting a focus on potential risks over service benefits. For financial service providers, understanding these psychological factors is crucial. It allows them to tailor product design, marketing strategies, and customer relationship management (CRM) to mitigate negative biases and churn rate. Therefore, predicting customer emotion and knowledge about products and services is not just advantageous—it's pivotal for businesses striving for success in today's highly competitive marketplace.

Customer historical data recorded by organizations is increasing exponentially, and data analytics specialists contend that the traditional customer churn analysis method, which heavily relied on transactional data and customer demographics, is no longer a robust solution. A more comprehensive approach is required to better deal with the heterogeneous data input in various modalities.

The current study aims to predict customer churn propensity by incorporating diverse modalities for the first time, to our best knowledge. The proposed approach incorporates survey analysis regarding customer financial literacy level, emotion detection from customer voices (CV), and CRM data to offer a more comprehensive representation of churn risk.

The proposed approach utilizes multimodal machine learning (ML) methods for a more holistic understanding of customer churn. This approach is characterized by considering multiple facets of the customer experience, from interactions with customer service to engagement with a product or service.
Multimodality in ML refers to the integration of two or more distinct inputs, recorded in different media formats, within a single ML model. The unique aspect of these inputs is that they cannot be unambiguously mapped into one another using an algorithm [11]. We proposed a fusion learning to force similarities of these inputs into a coordinated feature representation space. Subsequently, a decision-level fusion is applied to rank churn risk.

## A. Sentiment analysis

Customer sentiment analysis can be divided into para linguistic, using CV analysis, or linguistic or text based approaches. Recorded customer service calls can be analyzed to understand customer satisfaction. Although sentiment analysis using natural language processing (NLP) can be used to transcribe and analyze these calls; meaning is often also conveyed by tone, pitch, and loudness. Thus, humans provide and receive considerable meta-information during a conversation apart from the text. This is an inherent problem for text-based sentiment analysis in NLP, whereas non-linguistic emotion detection through speech signal processing is somewhat more robust [3]. Customers negative emotions are often good indicators of product dissatisfaction, and these customers can be considered to have higher churn risk due to their discernible discontent, warranting special attention. To decode emotion from CV, we introduced a voice-based sentiment analysis model [3]. The central innovation of our work lies in the development of a novel hybrid feature map generator algorithm using harmonic and percussive components of Mel Spectrogram that improves speech emotion recognition (SER).

## B. Financial literacy

Customer financial literacy can be a crucial factor to predict customer churn within the financial services industry. Financially literate customers are generally better equipped to comprehend the range of products and services available, enabling them to better select the most suitable option(s) to meet their requirements. This reduces dissatisfaction and subsequent churn probability. In contrast, customers with low financial literacy could struggle to understand the options, and hence make sub-optimal decisions regarding financial products and services, leading to dissatisfaction and potential migration to competing financial institutions perceived to offer superior products. We introduced a SMOGN-COREG model to deal with unlabelled examples via semi-supervised regression (SSR) methods [8]. We could significantly contribute by enhancing prediction precision beyond relying on labeled data in supervised methods.

## C. Multimodal modelling

The current study focused on analyzing synergistic relationships between predictors in each unimodal context, with the objective is to enhance churn prediction by utilizing a coordinated representation space that forces similarity for three input data formats: finance network, CRM, and inbound audio call data. We investigated the effectiveness for each

previously proposed unimodal. Given the identified challenges and opportunities, this study introduces several significant contributions as below:

- In a first-of-its-kind approach, we proposed a multimodal hybrid fusion model that integrates distinct databases, including CRM, CV, and FL data. This enables us to add an essential layer of understanding customer behavior for churn prediction by discerning customer emotion from their vocal attributes and measuring financial literacy level from their account performance and survey data.

- We built a specific coordinated feature representation space and translation matrix that reflects data similarity meaningfully to capture interactions between modalities, with the capability of adding future modalities like textual features in the proposed framework.

- Identifying significant correlations between negative emotions, low FL, and increased churn risk.

Thus, the proposed multimodal churn model leveraged customer segmentation by detecting negative emotions in CVs during interactions with call center operators, subsequently tailoring the response to address specific customer concerns and requirements, and gaining insights into potential dissatisfaction or frustration causes.

## II. RECENT WORKS

Considerable research effort is evolving towards more sophisticated churn prediction methods to better draw advantages from data growth. Hence, multimodal churn modelling will continue to advance, providing ever more refined strategies for churn prediction by encompassing various customer experience facets for holistic customer churn understanding [14].

De Caigny explored the benefits from incorporating textual data into customer churn prediction (CCP) models and highlighted superior performance from convolutional neural networks (CNNs) compared with other text mining methods for CCP, achieving 89.87% accuracy. Outcomes from their study will help guide researchers regarding text mining for improved CCP [4].

Nhi NY and Liu proposed predicting client churn by analyzing unstructured data, e.g. audio calls, in machine learning models. Their approach expanded traditional methods, which tend to focus on structured data, such as demographics and transactions. In contrast, their proposed model combined text mining with a gradient boosting tree algorithm to improve the client churn model predictive performance. This approach achieved significant predictive accuracy improvement, increasing at least 5% across various customer datasets [5].

Kimura highlighted benefits from combining boosting algorithms with hybrid resampling methods for CP, which could significantly enhance prediction effectiveness and practicality in both academic and industry settings. The challenge from imbalanced datasets was explicitly discussed, and showed that although hybrid resampling methods, including synthetic minority over-sampling technique (SMOTE), SMOTE-ENN, and SMOTE Tomek-Links, offer significant potential, they are yet to be widely adopted for CCP [6]. Ahn and Hwang reviewed
churn prediction methodologies, highlighting their chronological progression and underscored the necessity for adaptable and flexible approaches contingent on specific data formats. They also showed that very few previous studies adopted a multimodal approach to address churn prediction [7].

As delineated some recent work in Table I from our comprehensive literature review, many studies primarily focus on using singular structured data sources, such as CRM databases, for churn prediction tasks. Conversely, there needs to be more research that employs multiple data sources for the same purpose. To the best of our knowledge, our current investigation is the first that incorporates three discrete input data sets for the churn prediction problem.

<table><caption>TABLE I
SUMMARY OF THE COMPARATIVE STUDIES – RECENT MODALITIES</caption><thead><tr><th>Ref.</th><th>Input</th><th>Learning</th><th>Prediction</th><th>Industry</th></tr></thead><tbody><tr><td>[19]</td><td>1<sup>a</sup></td><td>Unimodal</td><td>RF</td><td>Finance</td></tr><tr><td>[18]</td><td>1</td><td>Ensemble</td><td>GBT+k-medios</td><td>Telecom</td></tr><tr><td>[17]</td><td>1</td><td>Unimodal</td><td>RF</td><td>Telecom</td></tr><tr><td>[16]</td><td>1</td><td>Ensemble</td><td>DL+LSTM</td><td>Game</td></tr><tr><td>[15]</td><td>1</td><td>Ensemble</td><td>LSTM+HS</td><td>Game</td></tr><tr><td>[5]</td><td>2<sup>b</sup></td><td>Feature Fusion<sup>d</sup></td><td>GBT</td><td>Finance</td></tr><tr><td>[4]</td><td>2</td><td>Feature Fusion</td><td>CNN+Logit</td><td>Finance</td></tr><tr><td>Ours</td><td>3<sup>c</sup></td><td>Hybrid Fusion<sup>e</sup></td><td>CNNs+DL</td><td>Finance</td></tr></tbody></table>

Therefore, the present study undertook a gap analysis and subsequently proposed an suitable methodology for CCP. This provides significant contributions by incorporating churn predictors, such as customer positive and negative emotional feedback from audio calls, and customer financial literacy level from survey data. The proposed synergistic hybrid feature fusion techniques and subsequent studies building on that framework, will mitigate bias for CCP.

### III. METHODOLOGY

Our methodology encompasses three modalities to analyze customer behavior: baseline customer churn, SER, and the FL model, which are elaborated in the subsequent sections.

#### A. Customer financial literacy modelling

Considering substantial impact from inadequate FL on customer churn, our first modality emphasizes FL in particular [8]. We developed a semi-supervised regression model, (SMOGN-COREG) to measure customer financial literacy derived from a large and unlabeled dataset. The proposed method boosts the learning process by generating synthetic samples for minority class. The SMOGN approach was applied during pre-processing and subsequently combined with non-parametric multi learner semi-supervised regression to enhance model performance, balancing the response variable distribution by increasing the number of rare but important cases in the dataset [13]. The model output is FL measurement in a range between 0 and 1.

#### B. Emotion recognition modelling

The second unimodal leverages the voice based sentiment analysis to recognize customer positive and negative emotions during their interaction with the call center [3]. The model employs the Mel Spectrogram to provide insight into voice tone, pitch, and tempo; extracting acoustic features and percussive and harmonic components from the signal frequency domain [12].

We employed a pre-trained CNN to develop a framework decipher emotions from CV, incorporating a feature map generator function to extract harmonic and percussive components via a median filter applied to the signal spectrum's axes. Two feature vectors were subsequently created by averaging these components, and computing the Mel spectrogram logarithm. Finally, we construct a two-dimensional image feature map as input for the pre-trained CNN-VGG16 network to classify emotions into binary outcomes positive and negative classes, represented as 0 and 1, respectively.

#### C. Baseline churn model

The third unimodal evaluates customer churn risk using demographic and financial behavior sourced from CRM platforms [10]. The proposed approach integrates recursive feature elimination (RFE), SMOTE, and ANN with Deep Learning (ANN DL) to construct the churn classifier. Thus, the proposed framework predicts potential churn within the upcoming six months using data from the prior 12-month window. A customer was deemed a 'churner' if they terminated their account within six-month prediction period, and the outcome is encapsulated in a binary variable, 0 or 1 indicates active or terminated account throughout the subsequent six months.

#### D. Proposed multimodal fusion learning

The proposed multimodal is able to analyze customer behavior from a diverse array of data sources highlighting complex interrelationships between different customer behavior facets. The model is designed with different components, including independent unimodal learning, feature representation space, translation or mapping function, and hybrid fusion.

Figure 1, shows how gauge customer financial literacy level, customer emotion, and churn propensity were employed to segment customers as low-risk (loyal or non-churner), Mid-risk (potential churner), and high-risk (high probability churner). For example, a high risk churn customer may exhibit low financial literacy, negative sentiment towards the company's offerings, and heightened propensity to churn.

##### Hybrid fusion:

The proposed model includes hybrid fusion (multi-level fusion) by incorporating early and late fusion. Hence, hybrid fusion generates input data for the baseline churn model by fusing prediction outcomes from FL and SER models for the CRM dataset.

<footnote><sup>a1</sup>: Structured data (e.g., demographic, account, and CRM data)</footnote>

<footnote><sup>b2</sup>: Structured + textual data (e.g., call log script and e-messages)</footnote>

<footnote><sup>c3</sup>: Structured + voice + financial literacy (qualitive data)</footnote>

<footnote><sup>d</sup>: Feature Fusion: Multimodal feature fusion modelling (or early fusion)</footnote>

<footnote><sup>e</sup>: Hybrid Fusion: Multimodal hybrid (early + late) fusion modelling</footnote>
Late fusion:

We established a prototype multimodal learning model with late fusion to evaluate the proposed hybrid fusion method efficacy compared with other methods. Early fusion data was not utilized as churn model input, with all data sources directly provided to the modalities.

E. Feature representation space and translation

Extracting specific features associated with each modal-
ity presents significant challenges due to inherent disparities
in scales, units, and distributions. Therefore, the proposed
multi-modal model fusion these heterogeneous features while
maintaining their unique insights towards churn modeling.
We propose a specific coordinated representation space that
reflects data similarity meaningfully to capture interactions
between modalities with the ultimate goal to understand com-
plementary customer behavior across various touchpoints, such
as call center interaction, FL survey, and historic demographic
data. The proposed coordinated representation space can be
expressed as

$$
f(x_i) \sim c(x_i) \sim v(x_i) \tag{1}
$$

where $x_i$ is the modalities and functions $f, v$, and $c$ rep-
resenting independent unimodal learning networks, including
FL, SER, and baseline churn model into the coordinated
representation space; and $\sim$ is similarity or coordination in the
projection into this space. We employ a translation dictionary
to map features extracted from each modality while preserv-
ing similarity and meaningful patterns. The mapping matrix
is represented through several logical propositions defined
by experts, each corresponding to different condition(s) on
the unimodal features. Each unimodal output data structure
is mapped into two level numeric nominal variables based
on a predetermined prediction confidence $P$($pred$) threshold,
defining a coordinated representation space by logical propo-
sition(s). Constant weights are assigned to each matrix array
$C_i$, $F_i$ and $V_i$ depending on the unimodal feature weight in
multimodal learning, which can be expressed as,

$$
\forall x_i, P1(x_i) : f(x_i) < P(pred) \Rightarrow F_i = 1; \neg P1(x_i) \Rightarrow 0
$$

$$
\forall x_i, P2(x_i) : c(x_i) \le P(pred) \Rightarrow C_i = 0; \neg P2(x_i) \Rightarrow 2
$$

$$
\forall x_i, P3(x_i) : v(x_i) \in \{\text{'Happiness'}, \text{'Neutral'}\} \Rightarrow V_i = 0;
$$

$$
\neg P3(x_i) : v(x_i) \in \{\text{'Sadness'}, \text{'Anger'}\} \Rightarrow V_i = 1
$$

where $P1(x_i)$, $P2(x_i)$, and $P3(x_i)$ denote propositions
for each unimodal feature, respectively. Estimating suitable
weights for each feature indicator is challenging. This study
assigned weights to the indicators guided by expert knowledge
regarding churn modeling and superannuation fund business
intricacies. Therefore, implicit feature translation alignment,
often incorporated as an intermediate step in mapping data, is
imperative for any subsequent task.

F. Hybrid fusion strategy

The proposed approach employed hybrid fusion to lever-
age complementary knowledge from unimodal representations
while ensuring logical coherence. This incorporates model and
decision-level fusion, creating a multi-level fusion framework.
We enforced constant fusion weight $C_i + F_i + V_i \ge 0$ and
$F_i = V_i$ by assuming fusion weights for FL and SER models
had equal influential contribution to enhance baseline model
performance. Therefore, a decision fusion $D_i$ was defined to
integrate complementary information and hence rank the churn
risk as low, mid, or high. The proposed decision fusion matrix
$D_i$ can be expressed as

$$
D_i = \begin{bmatrix} I_{\text{lowrisk}} \\ I_{\text{midrisk}} \\ I_{\text{highrisk}} \end{bmatrix} = I * \begin{bmatrix} C_1 + F_1 + V_1 \\ C_2 + F_2 + V_2 \\ \vdots \\ C_i + F_i + V_i \end{bmatrix} \quad (2)
$$

subject to fusion conditions

• Low risk Churner:

$$
\begin{align*}
I_{\text{lowrisk}} ={}& I(D_i = 0) \cdot I(C_i = 0) \cdot I(F_i = 0) \cdot I(V_i = 0) \\
& + I(D_i = 1) \cdot I(C_i = 0) \cdot I(F_i = 1) \cdot I(V_i = 0) \\
& + I(D_i = 1) \cdot I(C_i = 0) \cdot I(F_i = 0) \cdot I(V_i = 1)
\end{align*}
$$

If $I_{\text{lowrisk}} = 1$, customer is classified as low risk.

• Mid risk churner:

$$
\begin{align*}
I_{\text{midrisk}} ={}& I(D_i = 2) \cdot I(C_i = 2) \cdot I(F_i = 0) \cdot I(V_i = 0) \\
& + I(D_i = 2) \cdot I(C_i = 0) \cdot I(F_i = 1) \cdot I(V_i = 1)
\end{align*}
$$

If $I_{\text{midrisk}} = 1$, customer is classified as mid risk.

• High risk churner:

$$
\begin{align*}
I_{\text{highrisk}} ={}& I(D_i = 4) \cdot I(C_i = 2) \cdot I(F_i = 1) \cdot I(V_i = 1) \\
& + I(D_i = 3) \cdot I(C_i = 2) \cdot I(F_i = 1) \cdot I(V_i = 0) \\
& + I(D_i = 3) \cdot I(C_i = 2) \cdot I(F_i = 0) \cdot I(V_i = 1)
\end{align*}
$$

If $I_{\text{highrisk}} = 1$, customer is classified as high risk.

Where $D_i \in \{0, 4\}$ is a fusion output weight that embodies
ranked risk for each member $i$. $C_i$, represents the fundamental
customer churn understanding based on historical patterns; $F_i$
is quantitative financial behavior understanding, i.e., customer
FL; $V_i$ is a qualitative measure, reflecting customer emotional
state and satisfaction level derived from their phone commu-
nication with call center; $I(\cdot)$ is an indicator function where
$I(\cdot) = 1$ if the condition in the brackets is true and 0 otherwise.

Thus, the logical operators are designed such that exactly
one of $I_{\text{lowrisk}}$, $I_{\text{midrisk}}$, and $I_{\text{highrisk}}$ should = 1 for a given
customer risk level query. This ensures each customer is
classified into exactly one risk category.

The synergistic integration occurs when these modalities
fuse distinct learnings to generate a multimodal co-learning
allows a coordinated representation of two modalities FL and
SER influence complementary the training of another. Thus,
this co-learning strategy mitigates the limitations of relying on
a single source of information, thereby enhancing the model's
robustness.
<figure><img src="image_4.png" /><figcaption>Fig. 1. Proposed multimodal hybrid fusion learning method workflow to integrate various modalities</figcaption></figure>

## IV. EXPERIMENTS

### A. Data Sources

We employed real-world data from multiple sources to evaluate the proposed approach efficiency, including customer historic data from finance networks, demographic data from CRM platforms, and voice data, as shown in Table II.

Customer voices were replaced with similar voice from standard emotion databases to maintain privacy while preserving emotional content in the recorded voice files from inbound calls. The proposed method built a CV database leveraging the correlation between negative emotions and high risk churn customers with low financial literacy, as well as the association between positive emotions and low risk churn customers. The EMODB database, a standardized resource for categorizing emotions based on voice recordings, was utilized to label customer emotions [14]. The FL database was built by combining financial network data and an online customer survey to measure FL level. After initial data processing, training data was enhanced to (4154, 140). Table II shows the various data sources and their specification.

<table><caption>TABLE II
DATABASES DESCRIPTION</caption><thead><tr><th rowspan="2">Sub<br>Database</th><th colspan="3">Specification</th></tr><tr><th>No. datasets</th><th>Sources</th><th>Attributes / Size</th></tr></thead><tbody><tr><td>SDB1</td><td>1</td><td>Financial Networks</td><td>68 / 64K</td></tr><tr><td>SDB2</td><td>12</td><td>CRM</td><td>87 / 294m</td></tr><tr><td>SDB3</td><td>1<sup>a</sup></td><td>Audio</td><td>4 / 110min. voice</td></tr></tbody></table>

### B. Evaluation criteria

The mean average precision (MAP) metric was used to objectively evaluate the proposed method solving quality for churn ranking,

$$ MAP = \frac{1}{Q} \sum_{q=1}^{Q} \frac{1}{m_q} \sum_{k=1}^{n} P(k) \cdot rel(k) \quad (3) $$

where Q is the total number of queries; q is the current query, m_q is the relevant churn risk level for the q-th query;

$P(k)$ is the precision at cut-off k in the list; $rel(k)$ is an indicator function such $rel(k) = 1$ if the churn risk at rank k is a relevant document, and $rel(k) = 0$ otherwise. We calculated MAP by taking the mean of average precision (AP) across all risk levels (low, mid, high) for queries or instances. MAP = 1 means the system can find all the churn risk levels accurately. The AP was computed for an individual customer risk rank query as follows.

- Calculate the precision for each relevant risk rank retrieved, i.e., the proportion of relevant retrieved risk ranks up to that point, using constant weight for each risk rank.

- Calculate the mean AP across all queries. This provides an overall measure for system performance across a risk rank query set.

We also employed the macro-averaged F1 score (MA-F1), since the datasets were typically considerably imbalanced. It was imperative to evaluate smaller class performance with equivalent weighting to larger classes, and F1 score can be calculated independently for each class,

$$ \text{Macro-Averaged F1 Score} = \frac{1}{N} \sum_{i=1}^{N} \text{F1 Score}_i \quad (4) $$

where $F1 - score_i$ indicates F1 for the $i_{th}$ class and N is the number of classes.

### C. Correlation analysis

Figure 2(a) shows the linear relationship between modalities and the proposed multimodal model output. Thus, the proposed financial literacy and churn unimodal exhibits approximately balanced influences on model performance; whereas the SER unimodal exhibits strong positive correlation = 0.66, confirming significant impact on the multimodal model performance.

Figure 2(b) shows weightings for low FL and negative emotions (large circles). Thus, clients who renewed their accounts annually exhibited high FL and positive feedback; whereas customers with negative feelings did not renew their accounts or exited.

<footnote><sup>a</sup>EMODB database</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 2. a) Strong positive relationship between customer emotion recognition and multimodal learning performance. b) Relation of customer emotion feedback and financial literacy level with customer tenure.</figcaption></figure>

## D. Validation and comparison

To validate the effectiveness of the proposed method, we investigated the impact of modality integration on performance. Various evaluation metrics were used, including test accuracy, recall, F1-score, AUC, MAP, and MA-F1. The Figure 3 (a) shows the highest performance is achieved in hybrid fusion method with contribution of all modalities (FL+Churn+SER) with test accuracy of 91.2%.

Moreover, as shown in Figure 3(a) FL has the greatest impact on performance compared with SER. Figure 3(b) shows that the hybrid fusion approach improved churn risk distribution by shifting customers from low into mid and high risk classes.

<figure><img src="image_6.png" /><figcaption>Fig. 3. (a) Multimodal model performance is significantly improved by combining early and late fusion (i.e., hybrid fusion) considering all features; (b) Risk levels for various multimodal fusion methods, more customers identified as mid and high risk using the proposed hybrid fusion method.</figcaption></figure>

Figure 4 compares ROC curves for late and hybrid fusion. Higher AUC in Fig. 4 (b) confirms the advantage of integrating multiple modalities, leveraging multi-level fusion benefits.

<figure><img src="image_7.png" /><figcaption>Fig. 4. Hybrid fusion exhibits higher AUC than other methods considered.</figcaption></figure>

Table III compares risk identification accuracy using MAP and MA F1 metrics. Multimodal learning with hybrid fusion strategy achieved significantly improved results; MAP score = 66 and MA-F1 = 54.

<table><caption>TABLE III
METRICS FOR THE DIFFERENT FUSION METHODS CONSIDERED.</caption><thead><tr><th rowspan="2">Metrics<sup>a</sup></th><th colspan="3">Fusion Method</th></tr><tr><th>None</th><th>DF (without LF)</th><th>HF (LF+DF)</th></tr></thead><tbody><tr><td>MAP % ± STD</td><td>51 ± 0.8</td><td>65 ± 0.7</td><td>66 ± 0.1</td></tr><tr><td>MA F1 % ± STD</td><td>47 ± 0.1</td><td>47 ± 1.1</td><td>54 ± 0.6</td></tr></tbody></table>

<sup>a</sup>A higher value implies superior result

Python code for our proposed framework with further result visualization is available on our GitHub repository<fnref n="1" />. This resource simplifies the reproduction and enhancement of our study's experimental results.

## V. DISCUSSION

This study proposed a multimodal fusion learning model for predicting customer churn. To the authors' best knowledge, this for the first time an attempt to integrate diverse modalities including customer's voice (CV), financial literacy (FL) survey, and CRM record data to predict churn risk at three levels (low, mid, and high). These modalities provide distinct insights, thereby contributing to improved churn prediction precision.

The FL model utilized a SMOGN-COREG supervised model to infer customer FL from their financial behavior data. The baseline churn model, developed on a robust SMOTE and ensemble ANN algorithm combination, accurately predicted churn propensity in the context of massive and high dimensionality data. The proposed SER model employed a pre-trained CNN-VGG16 model to discern customer emotions from their vocal attributes, providing an additional layer to help understand customer behavior. Our findings highlight significant correlations between negative emotions, low FL, and increased churn risk. Furthermore, comparative analysis underscored the superiority of the hybrid fusion technique, achieving a MAP score of 66 and a test accuracy of 91.2%, thus outperforming both non-fusion and late fusion approaches.

## VI. CONCLUSION AND FUTURE WORKS

Despite the promising results, the proposed approach has limitations; in particular, the coordinated representation in the multimodal fusion method could lose valuable intermodal information. Future study will explore a joint representation approach, concatenating features from different modalities before commencing learning, providing the model with a broader, richer data set. This could enhance the learning process efficacy, particularly when employing a powerful model such as Deep ANN.

The second area of potential future work involves extending the multimodal approach by integrating textual features as a fourth modality. It can be represented as $t(x_i)$ a textual learning unimodal and embedded into existing coordinated representation space which is expressed in equation (1). This would broaden the model's scope and enhance the multifaceted analyses of customer behavior.

<footnote n="1">¹https://github.com/DavidHason/multimodal_churn_model</footnote>
ACKNOWLEDGMENT

This work is partially supported by the Australian Research
Council under grant number: DP22010371, LE220100078,
DP200101374 and LP170100891

REFERENCES

[1] Chen, Z., & Fan, Z. (2011, June). Building comprehensible customer churn prediction models: a multiple kernel support vector machines approach. In ICSSSM11 (pp. 1–4). IEEE.

[2] Lemmens, A., & Gupta, S. (2020). Managing churn to maximize profits. Marketing Science, 39(5), 956–973.

[3] Author names removed

[4] De Caigny, A., Coussement, K., De Bock, K. W., & Lessmann, S. (2020). Incorporating textual information in customer churn prediction models based on a convolutional neural network. International Journal of Forecasting, 36(4), 1563-1578.

[5] Vo, N. N., Liu, S., Brownlow, J., Chu, C., Culbert, B., & Xu, G. (2018). Client churn prediction with call log analysis. In Database Systems for Advanced Applications: 23rd International Conference, DASFAA 2018, Gold Coast, QLD, Australia, May 21-24, 2018, Proceedings, Part II 23 (pp. 752-763). Springer International Publishing.

[6] Kimura, T. (2022). CUSTOMER CHURN PREDICTION WITH HYBRID RESAMPLING AND ENSEMBLE LEARNING. Journal of Management Information & Decision Sciences, 25(1) (pp. 1 – 23).

[7] Ahn, J., Hwang, J., Kim, D., Choi, H., & Kang, S. (2020). A survey on churn analysis in various business domains. IEEE Access, 8, 220816-220839.

[8] Author names removed

[9] Oechssler, J., Roider, A., & Schmitz, P. W. (2009). Cognitive abilities and behavioral biases. Journal of Economic Behavior & Organization, 72(1), 147-152.

[10] Author names removed

[11] Baltrušaitis, T., Ahuja, C., & Morency, L. P. (2018). Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2), 423-443.

[12] Chu, S., Narayanan, S., & Kuo, C. C. J. (2009). Environmental sound recognition with time-frequency audio features. IEEE Transactions on Audio, Speech, and Language Processing, 17(6), 1142-1158.

[13] Branco, P., Torgo, L., & Ribeiro, R. P. (2017, October). SMOGN: a pre-processing approach for imbalanced regression. In First international workshop on learning with imbalanced domains: Theory and applications (pp. 36-50).

[14] Gopal, P., & MohdNawi, N. B. (2021, December). A Survey on Customer Churn Prediction using Machine Learning and data mining Techniques in E-commerce. In 2021 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE) (pp. 1-8). IEEE.

[15] Kristensen, J. T., & Burelli, P. (2019, August). Combining sequential and aggregated data for churn prediction in casual freemium games. In 2019 IEEE Conference on Games (CoG) (pp. 1-8). IEEE.

[16] Guitart, A., Chen, P. P., & Periáñez, Á. (2018). The winning solution to the IEEE CIG 2017 game data mining competition. Machine Learning and Knowledge Extraction, 1(1), 252-264.

[17] Ullah, I., Raza, B., Malik, A. K., Imran, M., Islam, S. U., & Kim, S. W. (2019). A churn prediction model using random forest: analysis of machine learning techniques for churn prediction and factor identification in telecom sector. IEEE access, 7, 60134-60149.

[18] Liu, R., Ali, S., Bilal, S. F., Sakhawat, Z., Imran, A., Almuhaimeed, A., ... & Sun, G. (2022). An intelligent hybrid scheme for customer churn prediction integrating clustering and classification algorithms. Applied Sciences, 12(18), 9355.

[19] de Lima Lemos, R. A., Silva, T. C., & Tabak, B. M. (2022). Propension to customer churn in a financial institution: A machine learning approach. Neural Computing and Applications, 34(14), 11751-11768.