Using Non-negative matrix factorization to classify companies. | by Christophe GEISSLER | Towards Data Science

Companies are complex entities that evolve over time. As a data-scientist involved in investment, I have long been asking myself the question of evaluating the most appropriate dimension for modeling enterprise data: in what space do these things live? No better answer could be found than this one: “Far too many!”. Another formulation of the question is to ask how many independent criteria are sufficient to characterize a company. As an example of criteria, we can think of market capitalization, industrial sector, number of employees, level of current earnings, carbon footprint, opinion of financial analysts, and much more: the number of available criteria easily exceeds one hundred. These criteria are not independent but, on the contrary, linked by a whole network of implicit correlations. This makes the choice of relevant subsets of variables a very hard task.
Investment practitioners apply machine learning to these data in two main ways:
To predict, by means of more or less complex regressions, the future behavior of a company — generally the evolution of its stock market value — from historical data on its different criteria. This is supervised learning.
To automatically group companies into homogeneous aggregates, and thus be able to compare these companies with their neighbors, without proper labels: this exercise is called unsupervised learning. An example of an aggregate very frequently used by the financial industry is the notion of industrial sector.
These two exercises — grouping and prediction — pursue different goals, but both are all the more difficult and unstable in their implementation, as more criteria are used. This is one of the manifestations of the well-known phenomenon in statistical learning, known as the ‘curse of dimensionality’.
Hence the natural need to reduce the size of the problem by reducing the number of criteria used. Easier said than done. It’s even more difficult to achieve in practice if we add an interpretability constraint.
In this article, we will see how the non-negative factorization algorithm (NMF) allows to get closer to these two objectives.
Let’s now formalize the problem a bit.
Suppose that the available data are represented by an X matrix of type (n,f), i.e. n rows and f columns. We assume that these data are positive or null and bounded — this assumption can be relaxed but that is the spirit.
A non-negative factorization of X is an approximation of X by a decomposition of type:
with W of type (n,c) and H of type (f,c), subject to the constraints W ≥ 0, H ≥ 0.Some important points to keep in mind:
This decomposition is an approximation, not an equality. The solution W and H matrices minimize the quadratic error between the real data X and their approximation X^.
The decomposition is not necessarily unique, even with the positivity constraint. One can indeed make unit change matrices in the form of positive diagonal D matrices, and then obtain identical decompositions in the form:
c ≤ min(n,f) is an integer representing the number of selected components. As in K-Means clustering methods, this parameter represents a parametric degree of freedom of the method. Determining the optimal number of components requires an additional criterion.
It is common in the literature to refer to lines of W matrix as factors.
A nice introduction to NMF can be found here [5].
It is usual to consider the lines of X as n observations of objects of the same category (eg customers, patients, companies, etc). Each object has f attributes, and the data contained in the matrix correspond to the values taken by the attributes for the corresponding objects.
The main effect of this decomposition is to decrease the information necessary to describe an observation. The original observations of the X matrix can be recovered, at the cost of an approximation, as linear combinations with positive coefficients — the lines of the H matrix. A reduction in size has thus been achieved: part of the information contained in the original matrix, comprising f data per individual, can be approximately summarized by a smaller set of c data per individual.
Another interesting effect of the non-negative decomposition is the emergence of a natural clustering of observations and attributes. The underlying mathematical principles would take us too far in this paper, but you can find a very clear presentation in this article [2]. Intuitively, observations can be clustered according to the dominant factor, i.e. the factor among the c factors having the highest value. In the same way, the original features can be grouped according to the factor on which they have the greatest influence.
This natural possibility of clustering is closely linked to the condition of positivity of the coefficients. This is an important difference with Principal Component Analysis (PCA), where the factors are required to be orthogonal in pairs, which makes it impossible to control the sign of the coefficients.
The collection and publication of extra-financial corporate data for investors is an industry in its own right, whose players are rating agencies. Each agency has its own criteria and its own rating methodology, but a large number of criteria can be found from one agency to another.
We are interested here in data characterizing the behavior of companies according to the three main pillars of environment (E), social (S) and governance (G). These data are called behavioural scores. They score, on a scale of 0 to 100, the company’s performance according to criteria such as equal pay for men and women, the proportion of independent directors in the board, water pollution, carbon footprint and many others.
These scores are generally reviewed between one and two times a year by the agencies, mainly on a declarative basis by the companies. We will use the information contained in these scores for:
Determine homogeneous groups of companies based on more intrinsic information than the industry sector,
Group variables in clusters in order to reduce the number of influencing factors.
The data used in this article are 77 behavioral scores from Vigeo-Eiris research, a subsidiary of the Moody’s rating agency specializing in extra-financial rating of global listed companies. The numerical experience we present here is based on the scores of the 500 largest European stocks, evaluated on average over the period September 2011– June 2020.
Vigeo’s behavioural scores are grouped into 6 areas:
Environment
Corporate Governance
Social: Business Behaviour
Social: Human rights
Social: Human Resources
Social: Community involvement
More elements about the scoring methodology can be found here.
The variety and granularity of this data makes it possible to envisage a more relevant grouping of companies than one based simply on the industrial sector. We will test this hypothesis by applying an NMF to the score data.
Behavioural scores are not known for all actions in all sectors and therefore the series have missing values.
The scores on the leftmost side of the graph have 100% fill rates. This is the case, for example, for scores related to Business behaviour (starting with C_S) or Corporate Governance (starting with C_G). On the other hand, highly specialized scores such as the treatment of local pollution (named EnV2_6) have the lowest fill rate, which can go down to 0 for some sectors where the topic has low relevance (Finance, Real Estate).
Like a vast variety of methods based on matrix algebra, NMF does not tolerate missing data. We will therefore replace the missing values of each score in the original matrix by the average of this same score, taken over all the companies for which this score is filled in.
We are now almost ready to launch an NMF decomposition. All we have to do now is to choose a value for parameter c. We know very little a priori about the optimal values of this parameter, but we expect the following scheme to be true:
Small values of c ==>Data-saving and imprecise model,
Large values of c ==> Data intensive and more accurate model.
This principle will help us to define an objective function integrating the notions of sparsity, stability and precision. Those concepts are indeed key ingredients for the interpretability of a predictive system, as explained here [3],[4]. For the sake of simplicity, we will leave stability aside in this article, and focus onto sparsity and precision.
To quantify sparsity, let us observe that the initial matrix of size (n,f) has been replaced by two matrices of respective sizes (n, c) and (c, f)). We can therefore define a sparsity score given by the following formula:
This score will of course be minimal for c = 1, in the theoretical case where all observations would be roughly multiples of a single vector.
To quantify precision, it is natural to define the precision score err(c) analogous to the R2 of linear regressions, by the following formula:
We can finally define an interpretability score by combining precision and sparsity:
This score is a number between 0 and 1, with the convention that 0 is the best possible level for interpretability, and 1 the worst level.
The score is calibrated to be 100% maximum when c = 1 or c = c*; the constant c* plays the role of a maximum acceptable number of dimensions, set here at 20.
Here are a few lines of code you will have to write compute an NMF decomposition for a given dimension c:
>>> import numpy as np>>> import pandas as pd>>> from sklearn.decomposition import NMF>>> X = pd.read_csv (scores_file_path)  # reading the score data>>> c = 4>>> model = NMF(n_components=c, init='random', random_state=0)>>> W = model.fit_transform(X)>>> H = model.components_>>> err = model.reconstruction_err_
The matrix H and W are the main outputs of the calculation, they are needed to rebuild the approximating matrix and the error function err(c).
The graph below is obtained by plotting the values of sp(c), err(c) and InterpScore(c).
This graph clearly shows the trade-off between precision and parsimony, with a sharp decrease for c ranging between 2 and 6, followed by a slow, then accelerating increase for larger values of c. In the following, and for the sake of the example, we retain the value c=6 for the number of components.
The most visible benefit that we get from the non-negative factorization is the replacement of the initial 77 features by a reduced subset of 6 positive linear combinations, the meta-features.
This graph shows the distribution of the coefficients (the ‘loadings’) of the 6 new meta-features over the initial 77 features. As expected, these coefficients are spread over several features and the linear combination can be difficult to interpret. This is specially true of meta-feature #1, the coefficients being more and more concentrated as the rank of the meta-feature increases.
This array shows the 6 most influent features of each meta-feature. Interestingly, the topics of these influent features are inter-related. We can therefore assign a main theme to each meta-feature. For example, the story told by feature V1 is mostly about Human Rights (hrts) and Human Resources (hr), while V4 speaks clearly about Environment (env).
NMF has traded dimension and interpretability, by drastically lowering the dimension (from 77 to 6) at the price of getting more fuzzy, but still meaningful features.
Interestingly, the symmetrical formulation of NMF also provides a natural way to group all companies into 6 possible clusters: we simply assign cluster #i to a stock when its largest value is in column i, i.e when its highest score is on the meta-feature #i. Let us take a look on how this new way of grouping companies corresponds to the usual sectors:
The NMF-based clustering brings information that is not contained in the sole sectors. We can read each line of the previous graph as a ‘signature’ of each sector along the various meta-features. For example, the Energy sector is among the lowest scores on the meta-feature #4, associated to the Environment theme.
Remember we have started with an X matrix that contained missing data. We have filled the empty cells with the global means of each columns, and fed our NMF algo with the resulting filled matrix.
Now that we have a clustering of the stocks, why don’t we try to improve the filling of missing data by replacing empty cells by the average of each column, taken over stocks of the same clusters, instead of all stocks? This will provide a new estimate of X, that can be in turn used as input for the factorization. And so on and so forth...
The lines clustering provided by the NMF allows to interpolate the missing values of the data matrix with an improved precision, compared to what can be obtained from mean-based priors. This method is very useful in many domains, the main advantage being the independence with respect to an exogeneous clustering like sectors: is it completely data-driven.
This paper tends to illustrate and support how non-negative matrix factorization may be used as a powerful tool to create natural groups of observations and features. This had long been recognized in life sciences, especially in genomics, where the need to reduce the number of features and keep the analysis interpretable at the same time, is crucial.
Applying NMF on extra-financial data is an innovative way of extracting the valuable information contained in these datasets. We have used the data of a particular provider — Vigeo-Eiris-, but the good news is that this method will be extremely robust when using multiple sources of data, to exhibit fundamental attributes of companies.
[1] Ding, C.H.Q.; Tao, L.; Jordan, M.I. Convex and Semi-Nonnegative Matrix Factorizations (2010). IEEE Trans. PatternAnal. Mach. Intelli. Arch. 2010, 32, 44–55.
[2] P.Fogel, Y. Gaston-Mathé et al., Applications of a Novel Clustering Approach Using Non-Negative Matrix Factorization to EnvironmentalResearch in Public Health (2016), International Journal of Environmental Research and Public Health.
[3] B. Yu and K. Kumbier, Three principles of data science: predictability, computability, and stability (pcs) (2019), arXiv preprint arXiv:1901.08152.
[4] V. Margot. A rigorous method to compare interpretability (2020), arXiv preprint arXiv:2004.01570.
[5] Vigeo-Eiris, website (2020)