Package: metamorphic-relations
Installationpip install metamorphic_relationsGet Started - MNIST ExampleFor more information on all classes and functions seehttps://3200-metamorphic-relations-lib.readthedocs.io/Create Data ObjectStores the data that the metamorphic relations (MRs) are used onfrom keras.datasets import mnist

from metamorphic_relations.Data import Data



MNIST = mnist.load_data()

data = Data(train_x=MNIST[0][0], train_y=MNIST[0][1], 

        test_x=MNIST[1][0], test_y=MNIST[1][1], max_y=10)Create Keras ModelA deep learning keras model to be trained for MNIST image classificationfrom keras import Sequential

from keras.layers import Dense

import keras.layers as layers



MNIST_model = Sequential()

MNIST_model.add(Dense(128, input_shape=MNIST[0][0][0].flatten().shape))

MNIST_model.add(layers.LeakyReLU())

MNIST_model.add(Dense(256))

MNIST_model.add(layers.LeakyReLU())

MNIST_model.add(Dense(256))

MNIST_model.add(layers.LeakyReLU())

MNIST_model.add(Dense(data.max_y))

MNIST_model.compile(optimizer="adam", loss="mse", metrics=['accuracy'])Create Metamorphic Relations (MRs)A list of domain specific MRs (DSMRs) is created for the given dataset using the transforms provided by ImageMRfrom metamorphic_relations.MR import MR

from metamorphic_relations.ImageMR import ImageMR



DSMRs = []

DSMRs += MR.for_all_labels(lambda x: ImageMR.rotate_transform(x, angle=180), [0, 1, 6, 8, 9], [0, 1, 9, 8, 6], "Rotate 180 degrees")

DSMRs += MR.for_all_labels(lambda x: ImageMR.flip_vertical_transform(x), [0, 1, 3, 8], name="Flip vertical")

DSMRs += MR.for_all_labels(lambda x: ImageMR.flip_horizontal_transform(x), [0, 1, 8], name="Flip horizontal")The list of Generic MRs (GMRs) for images is provided by ImageMRGMRs = ImageMR.get_image_GMRs()Create MRModelThe object from which the MRs can be applied to the data and the model can be trainedfrom metamorphic_relations.MRModel import MRModel



MR_model = MRModel(data=data, model=MNIST_model, 

    transform_x=lambda x: x.reshape((x.shape[0], -1)), 

    GMRs=GMRs, DSMRs=DSMRs)Use MRModel to Create Additional DataThe list of MRs are converted to MR objects which can be used to create new dataMR_model.DSMRs.perform_MRs_tree(data.train_x, data.train_y, data.max_y)The maximum number of composite MRs can be increased before perform_MRs_tree() to create more dataMR_model.DSMRs.update_composite(max_composite=2)Use MRModel to Train Keras ModelTrains the given model and returns F1 scores and models for various configurationsfrom metamorphic_relations.Results import ResultsMR_model.compare_MRs() compares each MR individually using all the training dataresults, models = MR_model.compare_MRs()MR_model.compare_MR_sets() compares each set of MR (No MRs, GMRs, DSMRs, GMRs + DSMRs) using all the training dataresults, models = MR_model.compare_MR_sets()MR_model.compare_MR_sets() compares each set of MR (No MRs, GMRs, DSMRs, GMRs + DSMRs) using increasing proportions of the training dataresults, models = MR_model.compare_MR_sets_counts()Save and Load ResultsResult objects can be written to and read from text filesresults, _ = MR_model.compare_MR_sets_counts()

results.write_to_file("Output/MNIST_sets_results.txt")

Results.read_from_file("Output/MNIST_sets_results.txt")Present ResultsUsing compare_MRs() gives individual results for each MR, these can be tabulated with initial number of training instances, actual number used, train and test F1 scoreresults, _ = MR_model.compare_MRs()

results.write_to_file("Output/MNIST_individual_best_results.txt")

Results.read_from_file("Output/MNIST_individual_best_results.txt").print_individual()The results for each set from compare_MR_sets_counts() can be graphed with graph_all()results, _ = MR_model.compare_MR_sets_counts()

results.write_to_file("Output/MNIST_sets_results.txt")

Results.read_from_file("Output/MNIST_sets_results.txt").graph_all("MNIST")