Package: revelionn
Retrospective Extraction of Visual and Logical Insights for Ontology-based interpretation of Neural NetworksRevelioNNis an open-source library of post-hoc algorithms for explaining predictions of deep convolutional
neural networks of binary classification using ontologies. The algorithms are based on the construction of mapping
networks linking the internal representations of a convolutional neural network with ontology concepts.
The inspiration for the development of this library was a paper in which this approach to the interpretation of
neural networks was proposed:M. de Sousa Ribeiro and J. Leite, “Aligning Artificial Neural Networks and Ontologies towards Explainable AI,” in 35th AAAI Conference on Artificial Intelligence, AAAI 2021, May 2021, vol. 6A, no. 6, pp. 4932–4940. doi:10.1609/aaai.v35i6.16626.How the library worksThe convolutional neural network, whose predictions need to be explained, is called “main” network. When an image is
passed through it, the output is a probability of some target class, which at the same time is a concept of ontology.
Activations of the “main” network produced as a result of image passing represent input data for the mapping networks.
The outputs of the mapping networks are the probabilities of each of the concepts relevant to the target class, that is,
the concepts that are involved in its definition. Knowing the probabilities of each of the concepts, it becomes possible
to form logical and visual explanations.Logical ExplanationsBy extracting relevant concepts, it is possible to form logical explanations about the belonging of the sample to the
target concept, accompanied by a set of axioms of ontology.
The input image presented in the scheme was taken from theSCDB dataset, with which the “main” network and mapping networks were trained.
This image belongs to classC1. The image is classified asC1if the conceptsHexagon⊓StarorEllipse⊓StarorTriangle⊓Ellipse⊓Starmarkerare present. An example of a logical
explanation by ontological inference for this sample is given below.The image is classified as ['C1'].The following concepts were extracted from the image:['HexStar', 'EllStar', 'NotTEStarmarker', 'Hexagon', 'Star', 'Ellipse', 'NotTriangle', 'NotStarmarker']with the following probabilities:[0.99938893, 0.99976605, 0.9937676684930921, 0.99947304, 0.9999995, 0.99962604, 0.9861229043453932, 0.9810010809451342]Justification for '__input__ Type C1':	(Degree of Belief: 0.99963)__input__ Type has some Star	("0.9999995")__input__ Type has some Ellipse	("0.99962604")(has some Ellipse) and (has some Star) SubClassOf EllStarC1 EquivalentTo EllStar or HexStar or TEStarmarkerEach of the extracted concepts corresponds to a certain probability, which is then used to calculate the degree of
confidence of the justifications. The list of possible justifications is ranked by the degree of trust.
If any concept has not been extracted, then we can say that the opposite concept has been extracted, the name of which
is automatically formed by adding the prefix 'Not'.The above example shows one of the explanations from the list of possible explanations. It can be interpreted as
follows. The concepts ofStarandEllipsewere extracted from the image. Therefore, based on the axiom of ontology that
the conjunction of the conceptsStarandEllipseis a subclass ofEllStar, we can conclude that the image also
representsEllStar. And according to another axiom, theC1target concept is equivalent toEllStar. Thus, the
prediction of the neural network was confirmed by ontological reasoning.Visual ExplanationsVisual explanations mean highlighting positively extracted concepts in the image. Currently, visual explanations are
formed using the occlusion method. Its essence lies in the fact that the input image is systematically overlapped by a
square of a given size with a given step. At each step, the overlapped image is run through the “main” network, and its
activations are run through the mapping network. Thus, by obtaining output probabilities at each step, a saliency map can
be formed.RevelioNN FeaturesMapping NetworksThe library implements two types of mapping networks whose parameters can be flexibly customized by the user.Type of mapping networkFeaturesSingle mapping networkIt is a full-connected neural network, the number of input neurons of which is determined by the number of activations of neurons of the specified convolutional network layers. It has a ReLU activation function in its hidden layers and a sigmoid in its output.It is reasonable to use it to extract only one concept from one or more given convolutional network layers.The user can vary the number of layers and the number of neurons in each layer of this mapping network.Simultaneous mapping networkDue to the features of its architecture, it allows you to extract many concepts simultaneously, receiving activations of all specified layers of the convolutional network at once.It takes into account the features of the 2D image structure and is less prone to overfitting compared to single mapping networks.It also shows good results in semi-supervised learning using semantic loss, which strengthens the relationship between concepts.Extraction AlgorithmsExtraction algorithmType of mapping networkWhat it doesExhaustive searchSingle mapping networkTrains and evaluates mapping networks based on the activations of each of the specified layers of the convolutional networkHeuristic searchSingle mapping networkDue to the heuristic reduction of the set of specified layers, mapping networks are not trained for every combination of layer-conceptSimultaneous extractionSimultaneous mapping networkTrains a mapping network that can simultaneously extract a set of relevant concepts from the entire set of layers of specified typesHow to UseRevelioNN can interpret convolutional binary classification networks that have already been trained without using this
library. It is worth noting that the network class must be inherited from the nn.Module class, that is, your network
must be implemented using PyTorch. The specified model must be converted to RevelioNN format.To use the API, follow these steps:In order to convert your model to RevelioNN format, your network class must be described in a separate file in which
the following variables must also be declared:variable storing the number of channels of the image fed to the network;variable storing the size of the image fed to the network;thetorchvision.transformsmodule object, which represents a transformation over images.Examples of network descriptions are given in themain_net_classesdirectory.Next, you need to initialize your convolutional neural network model.from main_net_classes.resnet18_scdb import ResNet18, NUM_CHANNELS, IMG_SIDE_SIZE, transformation

main_net = ResNet18()
main_net.load_state_dict(torch.load('SCDB_ResNet18_C1.pt'))Importconvert_to_rvl_format()function:from revelionn.utils.model import convert_to_rvl_formatCall this function by passing the data of the previously declared network model as parameters:convert_to_rvl_format(main_net, 'SCDB_ResNet18_C1', 'C1', 'resnet18_scdb', 'ResNet18', 'transformation', IMG_SIDE_SIZE, NUM_CHANNELS)After the main network has been successfully converted to the RevelioNN format, mapping networks can be trained.
Here is an example for training a simultaneous extraction network. Here activations were extracted from all batch normalization layers (the value is 'bn').from revelionn.mapping_trainer import MappingTrainer

device = torch.device('cuda')
trainer = MappingTrainer('SCDB_ResNet18_C1.rvl', os.path.join(root_path, 'main_net_classes'), ['bn'], 20, 100, 
                         os.path.join(root_path, 'trained_models', 'mapping_models'),
                         device, os.path.join(root_path, 'data', 'scdb_custom', 'images'),
                         'C1_mapping_train.csv', 'C1_mapping_val.csv', 'name', 100, 6, None)

trainer.train_simultaneous_model(['HexStar', 'EllStar', 'TEStarmarker', 'Hexagon', 
                                 'Star', 'Ellipse', 'Triangle', 'Starmarker'], 
                                 20, [160, 80, 40, 20], [20, 1])Once the mapping network is trained, you can form logical and visual explanations. To do this, you must first load
the trained network model viaload_mapping_model().from revelionn.utils.model import load_mapping_model

main_module, mapping_module, activation_extractor, transformation, img_size = load_mapping_model(
    os.path.join(root_path, 'trained_models', 'mapping_models', 'C1_20_[160, 80, 40, 20]_[20, 1].rvl'), 
    cur_path, os.path.join(root_path, 'main_net_classes'), device)To form logical explanations using an ontology, one must first extract the concepts relevant to the target concept
from the image, and then transfer the extracted concepts and their probabilities to the reasoning module along with the
ontology. This can be done as follows:from revelionn.utils.explanation import extract_concepts_from_img, explain_target_concept
from ontologies.scdb_ontology import concepts_map
from PIL import Image

image_path = os.path.join(root_path, 'data', 'scdb_custom', 'images', '001236.png')

image = Image.open(image_path)
main_concepts, extracted_concepts, mapping_probabilities = extract_concepts_from_img(main_module,
                                                                                     mapping_module,
                                                                                     image,
                                                                                     transformation)
print(f'\nThe image is classified as {main_concepts}.')
print('\nThe following concepts were extracted from the image:')
print(extracted_concepts)
print('with the following probabilities:')
print(f'{mapping_probabilities}\n')
   
justifications = explain_target_concept(extracted_concepts, mapping_probabilities, concepts_map, 'C1',
                                        os.path.join(root_path, 'ontologies', 'SCDB.owl'), 
                                        os.path.join(root_path, 'temp'))
print(justifications)Visual explanations can be formed as follows:import matplotlib.pyplot as plt
from revelionn.occlusion import perform_occlusion

perform_occlusion(main_module, mapping_module, activation_extractor, transformation, img_size,
                 image_path, window_size=20, stride=5, threads=0)
plt.show()The execution of the listed steps is shown inbasic_example.ipynb.RevelioNN also supports a command line-based interface, i.e. interaction through scripts. A detailed description of how to use each of the scripts can be found in the documentation.InstallationThe simplest way to install RevelioNN is usingpip:pipinstallrevelionn
pipinstallgit+https://github.com/lucadiliello/semantic-loss-pytorch.gitYou can view a list of required dependencies in therequirements.txtfile. You can also install them as follows:pipinstall-rrequirements.txtIt is also worth noting thatJava SE 8must be installed to form logical explanations.Project StructureThe repository includes the following directories:Packagemain_net_classescontains various convolutional neural network architectures that can serve as examples for initializing your network in RevelioNN;Packageontologiescontains examples of ontology files in OWL format, as well as examples of the dictionary of relations of dataset attributes to ontology concepts and examples of the class representing the ontology as a graph;Packageexamplesincludes notebooks that contain practical examples of RevelioNN use;All unit and integration tests can be observed in thetestsdirectory;The sources of the documentation are in thedocsdirectory.DocumentationA detailed RevelioNN description is available inRead the Docs.TestsTo run tests, you can use:pytesttestsPublicationsThe library was used in the following publications:Agafonov A., Ponomarev A. An Experiment on Localization of Ontology Concepts in Deep Convolutional Neural Networks // In the11th International Symposium on Information and Communication Technology (SoICT 2022), 82–87. DOI:10.1145/3568562.3568602Ponomarev A., Agafonov A. Ontology Concept Extraction Algorithm for Deep Neural Networks //Proceedings of the 32nd Conference of Open Innovations Association FRUCT, 221-226. DOI:10.23919/FRUCT56874.2022.9953838Agafonov A., Ponomarev A. Localization of Ontology Concepts in Deep Convolutional Neural Networks //2022 IEEE International Multi-Conference on Engineering, Computer and Information Sciences (SIBIRCON), 160-165. DOI:10.1109/SIBIRCON56155.2022.10016932FundingThe RevelioNN library was developed in the scope of the project 22-11-00214, funded by the Russian Science Foundation (RSF).AcknowledgementsWe thank the developers ofxaitk-saliency,semantic-loss-pytorch,nxontologyandBUNDLE, thanks to whom the development of RevelioNN became possible!Special thanks to the creators of theXTRAINS datasetfor providing the ontology and for inspiring the development of this library!