Package: transformational-measures-tf
Metrics on Neural Network Models in TF 2In this library, there are a set of python classes, developed in tensor flow 2,designed to be used in the calculus of different metrics on neural network models.Besides, there are three metrics developed. These metrics are known as Invariance, FeaturesVariance, and Same Equivariance.Main ClassesModelDataSetTransformationsDataSetIteratorOthers ClassesModelStructureLayerStructureBlockMnistTransformationsDataSetAnalyzerGraphsMetrics ClassesVarianceNormalizedVarianceFeaturesVarianceNormalizedFeaturesVarianceSameEquivarianceNormalizedSameEquivarianceClass ModelIntroductionLet's supose we have a neural network model, as in the figure 1:In detail, a neural network model is conformed by a series of layers as we can see in the figure 2:where every layer has a set of activations that we call $aij$ being i the layer index and j the activation index.Now, let's suposse we are interested in knowing the value of each activation when our model predict an input X. To get these values it is necessary a model that has as outputs not only the output Y but also the activations $aij$.
One of the main features of the class Model is what we have mentioned before, that is to say, our class Model allows us getting the activations values corresponding to the prediction of any input X. This class takes a Keras Model and creates a new model having as outputs the activations for each layer in the Keras Model.Another feature of the Class Model is that its method predict takes as argument a matrix of inputs. It was thougth to deal with metrics that works with samples and transformations, but it is also possible to forget the input meaning and just think of a 2D input.Usagemodel_keras=...#instance of a Keras Modelmodel=Model(model_keras=model_keras)input=...# tensor of rank n>2output=model.predict(input)i=...j=...l=...print("Activations values for input[i][j] in the layer l "+output[(i,j,l)])#Another way of instantiating the class is the nextpath=".../mymodel.h5"#path to the file .h5model=Model(path=path)Methodspredictget_model_structureget_name_layerevaluateMethod: predictArgumentstensor_input: a tensor of rank n, n > 2, where the first two dimentions represent the matrix and the rest dimentions represent the input data.ReturnsReturns an instance of Block. This instance can be indexed for the tuple (i,j,l) where i represents the row, j represents the column corresponding to the input matrix and l represents the l-layer of a model. The instance can be indexed too for the tuple (i,j,k,f) where in this case f represents the feature.
See the description of the Class Block for a better understanding.Method: get_model_structureReturns an instance of ModelStructure with the model structure.Method: get_name_layerArgumentsk:int,the layer indexReturnsReturns the name of hte layer k.Method: evaluateArgumentsx:tf.tensor, a tensor with model inputsy:tf.tensor, a tensor with the desired outputsReturnsReturns a list with the loss and the accuracy for the tuple(x,y)Class DataSetDataSet is an abstract class designed to represent a 2D dataset which will be used by the Iterator to iterate over the model. The implementations of the DataSet are responsability of the user and they are necessary to use the Iterator. An implementation available in this library is the MnistDataSet.Methodsget_data_shapeget_widthget_heightget_matrixtransposeMethod: get_data_shapeReturns the shape of the data in the matrix.Method: get_widthReturns the matrix width.Method: get_heightReturns the matrix height.Method: get_matrixArgumentsrows: a list of indicescolumns: a list of indicesReturnsReturns a submatrix conformed by the rows and columns passed as arguments.Method: transposeTranspose the DataSet.Method: is_transposeReturns if the dataset is transposed or not.Class TransformationsDataSetThis class extends the abstraction of the Class DataSet in order to have a dateset oriented to transformations. It is an abstract class too, that inherits from the class DataSet.
This class was thougth to have groups of transformations. These groups are setting up in the dataset through the set_transformation_group method.MethodsMethod: apply_inverse_transformationIt applies an inverse transform to a tensorArgumentstensor: tf.Tensor. The tensor to apply the inverse transformindex_transformation:int. The index of the transformation to apply the inverseMethod: get_size_transformations_groupsReturns the amount of transformations groups.Method: set_transformation_groupSets the transformations group in the datasetArgumentsgroup: int. The groupn:int. The amount of transformations for this group.Class IteratorIntroductionLet's consider we have a 2D dataset, for example a dataset of samples and transformations, and we want to get every value predicted by the model for each input in the dataset. If we think of all possible values that we will obtain and besides we divide these values into layers, it is possible to imagine some kind of data representation in the way showing in the next figure:Every square represents a matrix with the values of an activation for each input in the dataset. For instance, the first square represents the activation $a11$ and has the values $a11(ti(xj))$ with $i\ \varepsilon \ {1,...,m}$ and $j\ \varepsilon \ {1,...,n}$ where $a11(ti(xj))$ is the value of the activation $a11$ predicted by the model for the input $ti(xj)$.Now that we have a visual representation of the values we are looking for, it is easier to understand what the Iterator Class does. The Iterator gives you the possibility of moving and getting the values through blocks of width and height defined by yourself.For example, let's consider a block of 3x3. The iterator allows you obtaining the data in the following way:Usagemodel=Model(...)dataset=DataSet(...)iterator=Iterator(model,dataset)height=3width=3forheight,blocksiniterator.get_block(height,width):#...forwidth,blockinblocks#...Look at the Variance Class to have another using example!ConstructorArgumentsmodel: an instance of the Model classdataset: an instance of the DataSet classMethodsget_blockget_modelMethod: get_blockArgumentsheight: the height of the blockwidth: the width of the blockReturnsReturns a "vertical" iterator where each item in this iterator is a tuple (height,blocks_iterator) where blocks_iterator is an "horizontal" iterator in which each item is a tuple (width,block_activations) where block_activations is an object that indexes a three dimensional list through a tuple (i,j,l) where i represents the row and j represents the column corresponding to the block and l represents the l-layer of a model. The values in the list are the activations values in the layer l for the input located in the row i and the column j, and width is the width of the block and the height of the block is the height returned previously.RemarkWhen we say "vertical" iterator, we refer to the movement of the iterator in the block, the same happens with we say "horizontal" iterator.Method: get_modelReturns the model in the iterator.Class ModelStructureIntroductionThe goal of this class is to represent the structure of a keras model. On this way, the class ModelStructure has the structure of each layer in the model and it has too a map between layers and layers with features. The last information allows us to define the features variance metric.ConstructorArgumentsmodel: an instance of a keras modelAttributeslayers:list. A list with an instance of LayerStructure for each layernumber_layers:int. The number of layers in the modelmap_layers_features:list. A list containing a map between the layers and the layers with featureslayers_features:list. A list with an instance of LayerStructure for each layer with featuresnumber_layers_features:int. The number of layers with featuresMethods (No one)Class LayerStructureIntroductionThis class contains the information of the layer structure.Attributesnumber_features:int. The number of features in the layershape_feature:tuple. The shape for the features in the layernumber_activations:int. The number of activations in the layershape:tuple. The shape of the layerMethods (No one)Class BlockIntroductionThis class is used by the model to encapsulate the output in a block when it is predicted a block of inputs.The block can be indexed by a tuple of three or four indexes to get a particular output.The tuple (i,j,k) gives the output of the layer k for the input in the block given by the row i and the column j.The tuple (i,j,k,f) gives the output of the feature f in the layer k for the input in the block given by the row i and the column j.Besides, the block has some information of the dataset from which the inputs come.Attributesheight:int. The height of the blockwidth:int. The width of the blockmodel_structure:ModelStructure. The model structureMethodsMethod: set_rows_datasetThis method can be used to set the corresponding dataset rows in the block.Argumentsrows: listMethod: set_columns_datasetThis method can be used to set the corresponding dataset columns in the block.Argumentscolumns: listMethod: set_state_transpose_datasetThis method can be used to set if the dataset is transposed.Argumentsstate: boolMethod: get_rows_datasetReturns the corresponding dataset rows in the block.Method: get_columns_datasetReturns the corresponding dataset columns in the block.Method: is_dataset_transposeReturns the transpose state of the dataset.Class MnistTransformationsDataSetThis class implements the abstract class TransformationsDataSet. This dataset is Mnist with transformations. It has 8 groups of transformations. These groups are:IdentityRotationsX-TranslationsY-TranslationsX,Y-TranslationsRotations and X-Translations at the same timeRotations and Y-Translations at the same timeClass AnalyzerIntroductionThis class tries to analyze if there are relations between the results of a metric and the accuracy. For example, if you take the variance metric and a group of transformations for a dataset, you can see if when the variance increases the accuracy decreases.ConstructorArgumentsmodel:Model.dataset:TransformationsDataSet.iterator:Iterator.MethodsMethod: runThis method run the analyses.Argumentsn:int. The amount of transformations for each group in the dataset.Usagemodel=Model(...)dataset=TransformationsDataSet(...)iterator=Iterator(model,dataset)analyzer=Analyzer(model,dataset,iterator)analyzer.run(100)Class GraphsClass VarianceIntroductionLet's A be an activation in our model. One way to define the variance of A for a DataSet X is as follows:Var(A,X)="The mean of the variance for every row in X", i.e.,Similarly, it can be defined the variance for a Layer in the Model.So, the class variance computes these values for a dataset X and a model M using the Iterator.Usagemodel=Model(...)dataset=DataSet(...)iterator=Iterator(model,dataset)variance=Variance(Iterator)variance.compute()print(variance.variance_layers)ConstructorArgumentsiterator: it is an instance of the Iterator classMethodscomputeMethod: computeArgumentsheight: it is the height of the block to iteratewidth: it is the width of the block to iterateReturnsReturns a tuple (variance_layers,variance_layers_activations) where:variance_layers: it is a list with the variance of every layervariance_layers_activations: it is a list where each element in the list is a list with the variance of every activation in the respective layerClass NormalizedVarianceClass FeaturesVarianceClass NormalizedFeaturesVarianceClass NormalizedSameEquivarianceClass NormalizedFeaturesVariance