Explainable Deep Neural Networks | by Javier Marin | Towards Data Science

Nature is an infinite sphere whose center is everywhere and whose circumference is nowhere.
B. Pascal
For some years, black box machine learning has been criticised for its limits in extracting knowledge from data. Deep Neural Networks (DNNs) are one of the most well-known of the ‘black box’ algorithms. Deep Neural Networks (DNNs) are the most widely used and successful image classification and processing technique today. But when it comes to structured data (the most common data problem) there is still a controversial about its advantage. Efforts have been made in recent years to produce explainable machine learning techniques, with the goal of providing a stronger descriptive approach to algorithms as well as additional information to users, hence enhancing data insights.
In this work we propose a new way to make Deep Neural Networks more comprehensible when converting data encoded into practical knowledge. In each hidden layer representation, the data structure will be examined to gain a clear idea of how a deep neural network transforms data for classification purposes.
The emerging subject of deep learning mathematical analysis [1] has been tasked with answering some “mysterious” facts that appear to be inexplicable using traditional mathematical methodologies. They are attempting to comprehend what a neural network actually does. Deep Neural Networks (DNN) transform data at each layer, producing a new representation as output. DNN attempts to divide data in a classification problem, enhancing this action layer by layer until it reaches an output layer when DNN provides its best possible result. Under the manifold hypothesis (natural data creates lower-dimensional manifolds in its embedding space), this task can be viewed as the separation of lower-dimensional manifolds in a data space. [2][3].
DNN layers are linked by a realization function, Φ (an affine transformation) and a component-wise activation function, ρ. Consider the fully connected feedforward neural network depicted in Figure 2. The network architecture can be described by defining the number of layers N, L, the number of neurons, and the activation function. The network parameters are the weights matrix W and the bias vectors b. Each layer’s output is a new way of describing inputs. This is why they are referred to as representations, as they are essentially abstractions over input data. For each layer, Φ(x , θ)= Wx+b, with parameters θ = (W , b). The weights matrix is W, while the bias vector is b.
The activation function is crucial in determining how neural networks are connected and which information is transmitted from one layer to the next. Finally, it controls information exchange, allowing neural networks to “learn” from data. What they learn and why they learn it remains a difficult topic. Some researchers even argue that DNN can learn something particular about the data, and that this research is shared by several layers. So the argument that a layer learns something and then passes a representation to the next layer, which learns something else, is partially correct. Topology of data
Visualizing high-dimensional data representations via dimensionality reduction is a well-known technique for examining deep learning models. In figure 3 we can see the layers representation for a network with an architecture, a= ((33, 500, 250, 50, 1), ρ).
Aside from dimensionality reduction, there are alternative techniques for visualizing high-dimensional models. Topology analyzes the connection information of elements in a space and deals with qualitative geometric information. Topological data analysis (TDA) uses category theory, algebraic topology, and other pure mathematics methods to enable for a practical investigation of data form [4]. High-dimensional data sets significantly limit our ability to visualize them. This is why TDA can help us in improving our ability to visualize and analyze information. The most frequently observed data topologies include connected components, loops, voids, and so on.
The curse of dimensionality is one of the most significant issues we face while evaluating high-dimensional data representations in deep neural nets. In high-dimensional space, points are highly sparse. When DNN transfers data from one layer to another with different dimension, euclidean distances between points, and the distance of a point to a subset grows. Topology studies only properties of geometric objects which do not depend on the chosen coordinates (are coordinates-free). For that reason, this tool is perfect to analyze deep nets representations.
Topology avoids the quantitative values of the distance functions and replaces them with the notion of “infinite nearness” of a point to a subset in the underlying space [4].
Topology data analysis methods follows a basic routine: we come across a topological space (a representation of a high dimensional dataset ) and we need to find its fundamental group (data relations as links, loops or voids). But our data set is an unfamiliar space and it’s too difficult to look at explicit loops and relations. Then we look for another space that is homotopy equivalent to our [4], and whose fundamental group is much easier to compute. Since both spaces are homotopy equivalent, we know fundamental group in our space is isomorphic to fundamental group in the new space (gives the same data connectivity information). And its a coordinates-free process . Level-one connectivity information is related to data links, level-two connectivity information to data loops and level-three to voids (figure 4 shows level-one in green color and level-two in blue color.
Topological data analysis provides both quantitive methods and tools for qualitative understanding of high-dimensionality data through direct visualization [4]. An example is Mapper algorithm (figure 5). Quantitative data analysis will find the fundamental group of representation, and qualitative data analysis will bring other qualitative insights about data.
Topological signatures of a space are its fundamental groups. In TDA these signatures are the quantitative elements used to characterize the data space.
To intuitively understand what is Data Topology, we invite you to read this wonderful post: The Mathematical Shape of Things to Come from Tang Yau Hoong, published in Quanta Magazine.
As corollary we can say that Topological Data Analysis is about finding connections.
What we propose in this post is that Topological Data Analysis in the DNN representations may help us to understand more about how deep nets works and what information we can get from each layer. To perform a DNN representation analysis we are going to use a dataset made publicly accessible by The Government of India as part of their Drug Discovery Hackathon. It’s also posted in Kaggle. It includes a list of drugs that have been tested and their effectiveness against Sars-CoV-2. Kaggle’s user Agrawal completed this list by adding some molecule’s chemical details obtained from PubChem library. The final dataset includes 100 checked molecules, each with 40 features. Among features we have pIC50 (will be our feature label). It is the negative logarithm of the half maximal inhibitory concentration (IC50). This index is used as a measure of the efficacy of a substance in inhibiting a specific biological or biochemical function. Higher the value, higher is the efficacy as inhibitor. Our goal is to find a classification method for this molecules according it’s effectiveness against Sars-CoV-2. The dataset also contains six blinded molecules, which do not show the pIC50 value and can be used for predictive tasks.
We used the Giotto-tda library to perform TDA computations. Library gtda is a high-performance topological machine learning toolbox written in Python.The real advantage of using gtda, in my opinion, is its direct integration with scikit-learn and support for tabular data, time series, graphs, or pictures.Both assets make it simple to construct machine learning pipelines. For Deep Neural Networks we are using Keras library.
In our experiment we have used a fully connected neural network with architecture, a = ((33, 500, 250, 50, 1), ρ). It is a basic graph with three hidden layers. We have built the network with Keras functional API in order to make the different experiments more reproducible. The functional API can handle models with non-linear topology.
# clean is the pandas data frame with the data, # and 'pIC50' is the label feature.input_dim = len(clean.drop(columns='pIC50').columns) model = Sequential()#The Dense function in Keras constructs a fully connected neural network layer, automatically initializing the weights as biases.#First hidden layermodel.add(Dense(50, activation='relu',            kernel_initializer='random_normal',          kernel_regularizer=regularizers.l2(0.05),                                 input_dim=input_dim))#Second hidden layermodel.add(Dense(40, activation='relu',            kernel_initializer='random_normal',          kernel_regularizer=regularizers.l2(0.05) ))#Third hidden layermodel.add(Dense(20, activation='relu',            kernel_initializer='random_normal',          kernel_regularizer=regularizers.l2(0.05) ))#Output layermodel.add(Dense(1, activation='sigmoid',            kernel_initializer='random_normal',          kernel_regularizer=regularizers.l2(0.05) ))model.compile(optimizer='adam',                loss='binary_crossentropy',             metrics=['accuracy'])
As mentioned earlier, activation functions are critical in mapping data from one layer to another. There are two kinds of activation functions that we’re interested in: invertible (continuous functions with continuous inverses) and not invertible. tanh , sigmoid or softplus, are examples of first-class functions, and ReLU is an example of a non-invertible function. Activation functions can be invertible, but a neural network as a whole, even with invertible activation functions, is not invertible in general.
At this point, it’s worth recapping. We want to look into representation’s topology in a deep neural net. The key question will therefore be whether topological signatures preserve from one representation to the next. The answer can be found in category theory: A construction that converts objects (for example, de DNN representations) from one category to objects from another is functorial if it can be extended to a mapping on morphisms while preserving composites and identity morphisms [5]. Such constructions define morphisms between categories, called functors. Such constructions define morphisms between categories, called functors ( functor, F : C → D between categories C and D ). As a result, the task is limited to determining if data mapping from one layer to another is a functor. A functor may describe an equivalence of categories, in which case the objects in one category can be translated into and reconstructed from the objects of another.
To check functionality in a DNN first, we analyze the accuracy and loss of our DNN architecture (figure 6).
Training accuracy is 0.916 and training accuracy is 0.709. Remember we have a small dataset so accuracy is affected. Now we calculate the topological signatures of every representation(it is, finding fundamental groups in for each layer representation). We do this with gtda library, using Vietori-Rips filtration [4]. Results can be seen in figure 7.
To understand this figure, we could say the following: orange points are connected points, it is points that are “close” in the topological space. The green points represent “loops” of points in the space. Green points close to the diagonal are noise (we could say loops not completely formed). Orange points represents also the “size” of the set. This point is a quantitive way to characterize the space. They are called topological signatures. We can see in figure 7 that topological signatures from the input layer to the third hidden layer are kept. We see 4 remarkable loops and several connected points. There is an orange point in the vertical line away from the rest. Remember we have to do a binary classification, so in the vertical line of orange points we have to see two distant “clusters” of points (one for each category), otherwise the set is not classifiable.
This is clear in the output layer where we can see a point remarkably away from the rest of points, meaning that there are two categories and that one of them is much smaller than the other (one has only one point and the other one has more than 10 points). That’s it¡ DNN has separated the connected points until she got a clear separation, a maximum “distance” or minimum “nearness” between points. It has binary classified the dataset¡ So, the topological signatures seems to be the same but in the output function where we have asked the DNN to manipulate this to make a classification. So the transformation between representations is functorial: composites and identity morphisms are preserved.
We have repeated the experiment varying several hyper-parameters. The most interesting conclusion is that functoriality seems more dependent on network width that on activation function invertibility.
We have experimented with a network architecture a = ((33, 50, 25, 10, 1), ρ). Accuracy and Loss are very similar to the wider architecture (figure 8). Then we performed the same Vietory-Rips filtration (figure 9). We see that topological signatures are quite different from the original data set. In the first hidden layer we see less loops that completely disappear in the second hidden layer (we found basically noise). From the second layer, topological signatures are not mapped.
The neural network but is capable to separate the connected points as it did with a wider architecture. It could seem a little counterintuitive because means that DNN does not need to consider topological signatures to perform an accurate separation. But it is possible in relative simple operations. DNN ‘forces’ data topology by manipulating distances of points in a way that identity morphisms are not preserved.
Another topic is that it may be necessary to consider the equivalence of categories during the ‘learning’ process. When we consider small datasets, despite we are getting relative high accuracy levels, we should try to do it better. Topological data analysis gives us tools to double test network performance in the learning process.
We have seen a quantitative topological approach. But topology offers qualitative understanding tools that can be very interesting. Mapper algorithm introduced in figure 5 is a very interesting ‘visualization’ algorithm that can enhance the performance of dimensionality reduction algorithms. Mapper algorithm can be naturally seen as a clustering algorithm with a conversion to a graph with directed edges. In figure 10 we can see the results of applying mapper algorithm to layer’s representation in the network with architecture a = ((33, 500, 250, 50, 1), ρ) -wide layer - . The nodes are colored with a scale of ‘pIC50' values (our label feature). Despite we have been performing binary classification ( 0 for negative pIC50 values and 1 for positive), we are interested now in a regression point of view to get beer insights. We have implemented mapper algorithm with gtda library. Mapper’s difficulty in choosing hyperparameters filter, cover and cluster data.
""" 1. Define filter function – can be any scikit-learn transformer. It is returning a selection of columns of the data """filter_func = Eccentricity(metric= 'euclidean')""" 2. Define cover """cover = CubicalCover(n_intervals=20, overlap_frac=0.5)""" 3. Choose clustering algorithm – default is DBSCAN """clusterer = DBSCAN(eps=8,                   min_samples=2,                   metric='euclidean')""" 4. Initialise pipeline """pipe_mapper = make_mapper_pipeline(filter_func=filter_func,                                   cover=cover,                                   clusterer=clusterer,                                    verbose=False,                                   n_jobs=-1)""" 5. Plot mapper """plotly_params = {"node_trace": {"marker_colorscale": "RdBu"}}fig = plot_static_mapper_graph(pipe_mapper,                               X,                               layout='fruchterman_reingold',                               color_variable =clean['pIC50'],                               node_scale = 20,                                  plotly_params=plotly_params)fig.show(config={'scrollZoom': True})
We see in figure 10 the mapper algorithm applied for input, hidden and output layers. Mapper in the input layer shows some “loops”, several connected component of different length, and some not-connected components (isolated nodes). We can also see loops in hidden layer 1, but not in the following layers. In the input layer we see blue points (positive value of pIC50) and red points (negative values of pIC50) totally mixed. With this graph plot, we can’t imagine an easy way to separate blue points from red points. But output layer shows what deep neural network have worked this separation: nodes with blue color and red color are very easy to divide because blue color are in the end of the connected component. We could easily cut the connection and we will have a binary classification. We can see we needed three hidden layer to perform this separation. If we look at the third layer, this separation is still not possible so another transformation have been necessary (output layer).
DNN solves the classification problem by connecting as many points as possible. Apart from what we could called the main connected branch, the output layer has only one isolated node and a two-node connected component. Points are more connected than in the previous layers.
Another interesting feature from mapper algorithm is that you can visualize every node, extract the data points and translate insights directly from the original data set (figure 11). We see mapper algorithm with the nodes translated to data points, specifically to a feature that is related to molecule drawing. Last but not least we can visualize the graph for every feature in the input data set and for every neuron inside the hidden layers.
This feature from mapper can be useful for example to monitor how each molecule is placed in the graph in every layer.
There are some works that demonstrate how neural nets transform image data through deep layers in image classification problems. However, there aren’t many examples for structured data classification applications. We’ve seen how Topology Data Analysis can be a great tool for extracting information from hidden layers, allowing us to see the steps a DNN takes to complete its task. We can improve DNN performance by understanding this process, just as we can with image recognition difficulties, giving also additional insights into the data.
We have seen DNN can get the same accuracy with or with or without preserving data identities. This result have been tested in small data sets. Maybe preserving topological signatures can be another metric to check DNN’s accuracy in learning from data.
You can find the code used in this article here.
[1] Berner, J., Grohs, P., Kutyniok, G., & Petersen, P. (2021). The Modern Mathematics of Deep Learning. ArXiv, abs/2105.04026.
[2] C. Olah (2014). Neural Networks, Manifolds, and Topology. C. Olah blog.
[3] C. Fefferman, S. Mitter, & H. Narayanan (2016).Testing the manifold hypothesis. Journal of the Amer. Math. Soc. 29, 4, 983–1049.
[4] G. Carlsson. (2009) Topology and data. Bull. Amer. Math. Soc. 46 , 255–308.
[5] E. Riehl. (2017) Category theory in context. Courier Dover Publications.