Insights from Consumer Complaints based on Social Media and CFPB
Tags: Final Report, Project

Phung Lai, Pelin Ayranci

Problem statement and motivation

In today’s modern era, companies frequently receive complaints from customers. Consumer complaints provide insights into issues that people experience with products and services. It provides an opportunity to identify and rectify problems with products and services. As shown in Figures 1 and 2, over the past three months from September 2020 to December 2020, there is a total of 118,936 consumer complaints about different states in the USA, and the most complaints are about credit reporting and credit repair services. Understanding the complaints is crucial to regulate and improve customer satisfaction, laws, and financial decisions. This kind of information is important for them as they can improve some services they offer and take action about specific complaints. A good complaint-management strategy will result in a good customer-servicer relationship outcome. For example, if they found out that customers are mostly complaining about credit card reporting or services while purchasing from a specific website, banks can prioritize improving the functionality that is causing the error. The Consumer Financial Protection Bureau (CFPB) (https://www.consumerfinance.gov/data-research/consumer-complaints/) is a popular source of consumer complaint data. The Bureau frequently publishes the consumer’s narrative description of their experience after removing customers’ personal information. However, this Bureau database is not a statistical sample of consumers’ experiences in the marketplace. Complaints are not necessarily representative of all consumers’ experiences. Therefore, in our project, we aim at 1) Collecting data from other sources, such as social media, and Twitter in particular, then we can compare the collected data with CFPB data; 2) Predicting type of complaints to classify complaints into different categories, which help banks or financial companies easily keep track of them; 3) building graph for visualization and analysis based on collected data from social media; 3) explaining the key issues in complaints that contribute to the classification; and 4) explaining and analyzing the similarities and differences between our collected data via social media and the Bureau data (Consumer Financial ProtectionBureau — CFPB).

Figure 1: Map shading for consumer complaints in the USA from 9/2020 to 12/2020

Related Work

Besides a line of work that analyzes CFPB data [2,4], the major media channels like Twitter and Facebook are increasingly used for consumer complaints. Consumers either directly interact with banks or financial products and report their complaints or indirectly mentions their complaints in their social media accounts [3,5].

Previous work on online complaint behavior can be classified into 4 categories. The first category is to investigate the effects of the volume of online reviews on purchase intentions [7]. Consumers are based on online product reviews to access the credibility of the online product and decide whether to buy the product. Online product reviews have become the most important form of electronic word-of-mouth. Today, almost every e-commerce website and retail company, such as Amazon.com, Bestbuy.com, Walmart.com, etc have customer review systems, and based on the reviews or complaints about a product, other customers can make a decision on buying the product. The second category is to investigate the consequences of online mouth-to-mouth marketing [6]. Classical and office mouth-to-mouth has played a key role in customers’ buying decisions [10]. With the emergence of the Internet, consumer’s options are more based on online reviews. However, there can be many scams or fraud reviews on the online reviews, and whether we can trust an online review even we do not know how they are. Therefore, mouth-to-mouth marketing is still important today. The third category investigates online boycott behavior in detail [8]. Boycott behaviors are often effective in anti-consumption tactics against companies. Boycott movements typically negatively affect consumer decisions and purchase intentions toward the target company and its products. The final category focuses on analyzing and designing strategies to handle consumer complaints in social media [3]. Social media offers a possibility for consumers and other stakeholders to express their opinions related to an organization and its products. Our work follows the final line of researches. Different from existing literature on that category, we will enhance and compare the insights we gather from social media with an offline source, namely Consumer Financial Protection Bureau (CFPB). Also, when classifying a complaint into a specific complaint category, existing work does not explain the words/sentences that contribute to that classification. We will provide an explanation for our classification task. That is helpful to sanitize the classifier and provide insights into the complaints. For example, as shown in Figure 10 we have a very long consumer complaint that is related to fraud. We will provide an explanation of why that complaint is classified as a complaint in the fraud category.

Data Exploration and Methodology

Figure 2: Product the consumer identified in the complaint over USA from 9/2020 to 12/2020

We explored datasets from Twitter and CFPB. We first collect consumer complaint-related data from Twitter by using Tweepy and a set of keywords, such as { “late response”, “wrong information”, “trouble in payment”, “bankrupt”, “wrong modification”, “Bank of America”, “Wells Fargo”, “CitiBank” }. Several examples of consumer complaints were collected from Twitter shown in Figure 3.

Figure3:Examples of consumer complaints collected from Twitter.

For classification or text mining tasks, we process the text as follows. Firstly, we read the text into Python using Pandas library. Secondly, we clean the text to obtain better features by some of the basic pre-processing steps, such as removing stop words, removing punctuation, transform texts into lower case. Thirdly, we vectorize the cleaned text using Glove word embedding [9]. The embeddings are inputs for classification tasks using the GRU network. Note that, here we use the CFPB data that has labels about complaint categories to train the GRU model, and then applied the trained classifier to get labels for each tweet collected.

For graph visualization and analysis tasks, we first define the nodes and edges of the graph. Nodes in our graph are the users and bank names. For instance, Bank of America (BofA) is a node and if a user is complaining about BOFA and mentioning BOFA, it is also a node that has a connection to the BOFA node. Edges are the connection between different nodes, such as user-mentions and user-replies. The graph is built for visualization. The CFPB dataset does not have user information, so we use the schema for building our graph for complaints with NEO4j as shown in Figure 4.

Figure 4: Schema of the Graph.

We are storing the collected data from Twitter in the Gexf graph file. Gexf format involves structures, edges, and all attributes of a graph, and mainly used for graphing projects. The .gexf format allows the creation of structure hierarchy and is usually saved in plain text and this extension is used to describe files containing graph data and visualization. After obtaining .json file from Twitter API, we used the publicly available code2 to get .gexf format. We are storing CFPB data in csv format.

For explanation, we applied Hierarchical Attention Network [11] which uses stacked bidirectional GRUs [1] on the word-level, followed by an attention network to extract words that are critical to the meaning of a sentence, and then to aggregate the words to form a sentence vector. The same mechanism is applied to the generated sentence vectors to obtain important sentences that contribute to the classification of the document.

For comparison to CFPB data, we aim to analyze the differences and similarities that we can conclude from our collected dataset and CFPB. It is noted that in CFPB, all personal information is hidden for privacy protection, and with different scales of data collection, there are probably some violations or conflict conclusion based on the two datasets. We are considering the differences as complementary to each other to provide sufficient insights about customer complaints.

Experimental Results

Figure 5: Complaint Graph

We use NEO4J for visualizing the complaints we get from Twitter and CFPB. Visualization helps us better see the important complaints, the products that receive the most complaints, and the most popular response for a complaint which can be seen in Figure 5. We created the schema for our visualization that included the important nodes as presented in Figure 3. In our initial analysis, consumers mostly have problems with credit reporting. Credit reporting is followed by debt collection-related complaints. Most complaints are directed to Experian company. We conducted large-scale experiments as can be seen from Figure 6.

Figure 6: Large Scale Experiment

We identified 3 large communities of consumer complaints. The largest community with one star represents the mortgage-related complaints community. Customers with pink complaint numbers mostly complain about mortgages. The second community with 2 starts accumulated around credit card-related complaints. Third community with 3 stars accumulated around debt collection-related complaints. Different than our small initial dataset, the largest dataset contains complaints that are mostly related to mortgages.

We found that mortgage-related complaints are mostly related to a specific bank rather than multiple banks. This bank should take action on its mortgage practices as its customers are excessively complaining about its mortgage practices. Secondly, Credit card related complaints are often accompanied by account issues. Several banks are frequently mentioned in this category. Actionable insights for those companies should be to review automatic decline practices due to low balance issues.

Large scale experiment results from Twitter complemented the results from CFPB. The largest category for complaints on Twitter in 3 months period was credit card/ account-related issues. Second, debt collection-related complaints and the third one is online banking. Online banking related issues emerged as a very important topic in Twitter consumer complaints whereas we do not have many online banking-related complaints in the CFPB dataset. Twitter dataset complemented the CFPB dataset and provided a new category for banks. We have specific bank names and complaint category matches however our aim is not to disclose any private company names. We identified that Twitter complaints in Figure 7 recently included a credit reporting issue community.

Figure 7: Twitter Recent Community on Credit Report

We suspect this newly emerged topic in Twitter complaints may be a result of a common credit reporting mistake. We expect this issue to appear in the CFPB dataset as soon as they update the dataset.

Twitter dataset also provides us with hourly and daily real-time updates. On the other hand, the CFPB dataset lacks real-time updates. We can see hourly changes in complaint volumes as well as newly emerged complaint topics. Figure 8 and Figure 9 show the hourly and daily consumer complaint volumes. We also observed that early in the morning consumer complaints tend to peak. We suspect this to be a coincidence and check it with other dates and saw that early morning hours tend to be favorable hours for sharing a complaint.

Figure 8: Consumer complaint collected from Twitter daily volume

Figure 9: Consumer complaint collected from Twitter hourly volume

Explanation experiment categorized the complaints into 16 categories. Some of the largest categories are mortgage, credit card issues, debt collection, and credit reporting. We ran our experiment for explaining the complaints and provided concise explanations of the issue and the important sentences/word for the classification in Figure 10. Categorization of complaints is very important for companies. Accurate categorization helps them address the issues promptly. Classification explanation helps them reconsider/improve the model they chose for classification. Also, financial institutions may use these explanations for regulatory compliance purposes. Particularly, they need to explain the reasons behind their complaint classification or loan approval decisions. This explanation output will be useful for documenting the reasons why the model made a particular classification and brings reliability to the model. Moreover, some complaints are not as short as this complaint. There are complaints that are longer than 50 sentences. If a bank representative wants to address this issue they need to read this 50 sentence complaint and understand it clearly. However, using these short explanations like in Figure 10, the company representative can save time and address the core issue quickly without getting lost in the story of the complaint. Twitter complaints tend to be limited by word limit however we have seen many cases that users rely on multiple tweets to complete their complaint story and increase the word limit that Twitter initially has. Finally, In some cases explanations that we generated understand the core issue better than our team and guide us to understand the real issue behind a complaint. We believe this feature will be especially useful for people who need to address the complaints we identify from Twitter and CFPB.

Figure 10: Example of visualization to get insights about complaints and classifier.

Conclusion

The CFBP data does not disclose any information about people who complain. On the other hand, the collected Twitter data has personal information such as user names. It is not possible to obtain written consent from the users that we use for this study. Unfortunately, Twitter is especially ethically challenging because of the free availability of the data. Another possible ethical issue is that Twitter is where everyone can post their contents, comments, and discussion. It does not have many constraints or policies to bring its communities under an ethical umbrella. Therefore, Twitter is a kind of free speech, the ethics and its data correctness issues can be a problem if using their data for research purposes. To protect user information, in our project, we only use complaint content from the collected tweets, but not user/account information.

We believe explanation tool and complaint categorization inferences will be very useful for banks that are struggling to improve their services. Explanation tool will provide them with explanations which in turn can be used to improve classification accuracy and compliance reports. Insights that are derived from Twitter and CFPB can be used to improve the services and products they offer.

This analysis had several limitations: Firstly we consider tweets in the last 3 months and we saw that complaint topics change frequently. If we were to select a different 3 month period analysis results would be different. That’s why we use the Twitter dataset as a complementary source for CFPB to identify trends in complaints. For example, Twitter helps us identify a new trend in complaints which we didn’t see in the CFPB dataset. Secondly, we wanted to use a comprehensive list of keywords to get all possible complaints on Twitter related to our specific companies. However, keywords evolve and change over time and we may lose information if we can’t identify a keyword that is related to our task. Finally, we tried to match the dates of CFPB complaints and Twitter ones and lose a lot of data points. A larger sample of tweets would give us more insights into the complaints.

Appendix

Phung Lai and Pelin Ayranci contribute to this work equally. Data collection and experiments of social media and CFPB are done together via long skype sessions and screen sharing. The problem statement part is finalized by Phung and the Experiment part is finalized by Pelin during the writing session.

References

[1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate.arXivpreprint arXiv:1409.0473, 2014.

[2] K. Bastani, H. Namavari, and J. Shaffer. Latent Dirichlet allocation (LDA) for topic modeling of the cfpb consumer complaints.Expert Systems with Applications, 127:256–271, 2019.

[3] S. A. Einwiller and S. Steilen. Handling complaints on social network sites–an analysis of complaints and complaint responses on Facebook and Twitter pages of large us companies.Public Relations Review, 41(2):195–204, 2015.

[4] P. Foohey. Calling on the cfpb for help: Telling stories and consumer protection. Law & Contemp. Probs., 80:177, 2017.

[5] M. B. Goodman and S. H. Dekay. How large companies react to negative Facebook comments.CorporateCommunications: An International Journal, 2012.

[6] T. Hennig-Thurau, K. P. Gwinner, G. Walsh, and D. D. Gremler. Electronic word-of-mouth via consumer-opinion platforms: what motivates consumers to articulate themselves on the internet?Journal of interactive marketing,18(1):38–52, 2004

[7] F. R. Jim ́enez and N. A. Mendoza. Too popular to ignore: The influence of online reviews on purchase intentions of search and experience products.Journal of Interactive Marketing, 27(3):226–235, 2013.

[8] S. C. Makarem and H. Jae. Consumer boycott behavior: An exploratory analysis of Twitter feeds. Journal of consumer affairs, 50(1):193–223, 2016.

[9] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.

[10] M. L. Richins and T. Root-Shaffer. The role of evolvement and opinion leadership in consumer word-of-mouth: An implicit model made explicit.ACR North American Advances, 1988.

[11] Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy. Hierarchical attention networks for document classification. InProceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480–1489, 2016