Package: netneural
Neural Network FrameworkA project done at the university JMU Würzburg byfachtertogether withmarja-wandmamadfrhiThe package is being used by the remaining implementation of that project in another repository
calledgesture-detection-neural-networkand further usage of this package can be seen in the implementationThere are two jupyter notebooks explaining how to use the package in itsrepositoryProject by Felix Achter, Marja Wahl, Mohammad FarrahiThis repository is designated to creating, training, and using a neural network. There are five main tasks you can
tackle:Data Preprocessing: get the data ready for being trained onCreating the Network: create your network by using information of your training dataTraining the Network: train the resulting network by optionally choosing number of iterations, learning rate, and
optionally optimizerEvaluation: evaluate the trained network with metrics like accuracy and F1 scorePredicting: use the final network for predicting on unseen dataData PreprocessingScalingYou don't have to scale your data in advance. 
Creating a new NeuralNetwork instance, the Standard Scaler is set as the default scaler.
You can read more about creating a NeuralNetwork instance in section "Creating the Network".Thefeature_scaling.pyfile provides two classes for scaling your data: the StandardScaler and
the NormalScaler class. You can scale your data, before instantiating a neural network, by creating an instance of either
class, fitting the scaler to your data, and then transform your data. The data can be transformed back by using the
inverse_transform() function.from netneural.network.feature_scaling import StandardScaler

standard_scaler = StandardScaler()
standard_scaler.fit(X)  # X being the training data
X_scaled = standard_scaler.transform(X)
X = standard_scaler.inverse_transform(X_scaled)  # returns original XDon't forget to set the right scaler, if you did not use the default StandardScaler, when creating a NeuralNetwork
instance:from netneural.network.feature_scaling import NormalScaler

normal_scaler = NormalScaler()
normal_scaler.fit(X)  # X being the training data
X_scaled = normal_scaler.transform(X)

nn = NeuralNetwork(shape, scaler='normal')EncodingYou don't have to encode your target values in advance.
If no encoder is handed over when instantiating a NeuralNetwork, the data is encoded, if necessary.
You can read more about creating a NeuralNetwork instance in section "Creating the Network".Theone_hot_encoder.pycan be used for creating your own OneHotEncoder instance and encoding
multiclass labels into one hot encoded label lists.from netneural.network.one_hot_encoder import OneHotEncoder

encoder = OneHotEncoder()
y_one_hot = encoder.encode(y)  # y being the target values with multiple labels
unique_labels = encoder.unique_labels  # get unique labels found by encoderDon't forget to pass the created encoder to a NeuralNetwork instance, if you encode the labels in advance:nn = NeuralNetwork(shape, encoder=encoder)Training Test SplitIn order to properly analyse your network's training, you will need to split the data into a training and a test set.
You can do that by the provided functiontrain_test_split()indata_loader.py:# take only the first four outputs, because no validation set is produced
X_train, y_train, X_test, y_test = train_test_split(X, y, train_per=0.8, test_per=0.2, randomized=True)[:4]You can also divide your data into training, test, and validation set:X_train, y_train, X_test, y_test, X_val, y_val = train_test_split(X, y, train_per=0.8, test_per=0.1, valid_per=0.1)Principal Component Analysis (PCA)Additionally, you can perform PCA using methods provided inpca.py.If you do not know anything about PCA you can simply reduce your
feature count by specifying how much variance of data you want to cover with the remaining features:from netneural.pca.pca import PCA

pca = PCA()
X_pca = pca.pca(X, var_per=0.99)If you want to directly specify the number of desired features, you can also do so:X_pca_10 = pca.pca(X, number_pcs=10)You can also specify both, wherenumber_pcswould act like a maximum for the returned constructed features:X_pca = pca.pca(X, var_per=0.99, number_pcs=10)  # returns 10 principal components, if more are needed for covering 99% of varianceYou can also directly work with eigenvectors and eigenvalues:eigenvectors, eigenvalues = pca.get_eigenvectors_and_values(X)

# gets matrix with 'rank' of every attribute in every principal component
ordered_attributes_matrix = pca.analyze_eigenvectors(eigenvectors, attributes)  # attributes is list of original attributes

# gets a list of the original attributes, ordered by its position in each eigenvector
ordered_attributes = pca.compute_order_attributes(ordered_attributes_matrix)

# returns the number of principal components needed for covering the input percentage
n_pc = pca.explained_variance(eigenvalues, 0.99, plots=True)  # if plots is True, plots variance covered per n principal components

X_pca = pca.get_n_dimensions(X, n_pc, eigenvectors)Creating the NetworkFor creating the network, you need to create a new instance of the NeuralNetwork class, provided innn.py.
You have to set several parameters when instantiating.Activation Function: you can choose between sigmoid, tanh, and relu activationArchitecture: you will need to provide the number of neurons per layerWeigths: you don't have to specifiy weights, but you can do so by passing a list of lists, each containing weights
for each layerfrom netneural.network.nn import NeuralNetwork

shape = (input_features, 2, 2, output_classes)  # variables store the number of input features/output classes
nn = NeuralNetwork(shape, activation_function='sigmoid')  # uses randomly assigned weightsIf you want to train a neural network for regression, you will have to set theregressionparameter to true:nn = NeuralNetwork(shape, regression=True)Training the NetworkTraining the network requires several hyperparameters you have to set. Thetrain()function of the NeuralNetwork class
takes severable variables in order to define them:Iterations: choose how many epochs the neural network will train forLearning Rate: decide on the learning rate for trainingTest Data: you have the choice of handing over a test data set in order to receive feedback about the evaluation of
the network on this data setOptimizer: you can choose between the default gradient descent or the Adam optimizerCost function: the network decides on the cost function on its own, whether your last layer has one or multiple nodes,
it uses the Mean Squared Error function or Cross Entropy (with Softmax)Batch Size: if a batch size is set, the model trains batches of the training data of this size every iteration, it
updates its weights after training each batch, and it sees each batch during one iteration. Each iteration, the training
examples are randomly assigned to a batch. If the batch size is not given, the model will normally train on all the training
data per one iterationPlotting: if theplotsvariable is set to True, the function will plot the error, accuracy, and F1 score on training
and eventually test data set after trainingEncode: if you did not encode your data yet (multiclass classification) and did not hand over an encoder during the
creation of the neural network, you need to set this parameter to true, if your multiclass labels are not encoded yetThe function returns the training history on either the training data set or the test data set, which can later be used
for analyzing and visualizing.f1_history, acc_history, err_history = nn.train(X, y, 100, 0.1)  # trains with default settings (optimizer=gradient descent, no test data, no plots)f1_history, acc_history, err_history = nn.train(X, y, 100, 0.1, X_test, y_test, optimizer='adam', batch_size=500, plots=True)  # trains while evaluating on test set, uses Adam optimizer, uses batches, plots resultsFor the format of data sets applies:the data matrix (X) should be formatted data points x featuresthe target values (y) should be formatted class x data points (one row for binary classification)When training a regression neural network, the only metric output afterwards is the history of the Mean Squared Error (MSE):error_history = nn.train(X, y, 100, 0.1)Training SessionsThe functions provided in thesession_util.pycan be used for saving and loading a trained network.
Along with the network the functionsave_session()also takes other parameters, like the unique labels, PCA object,
training history, training data, iterations, and learning rate, in order to easily reenact the documented session.from netneural.session.nn_session_util import save_session

learning_rate = 0.1
iterations = 100
f1_history, acc_history, err_history = nn.train(X, y, iterations, learning_rate)  # we train a neural network

# save_session stores a .pkl file with a name consisting of current time and the final f1 score
save_session(nn, nn.encoder.unique_labels, pca, f1_history[-1])After the file is stored, it can be used to again load the neural network:from netneural.session.nn_session_util import load_from_config

config_file = "config_90_f1_2022-03-25 14-56-20.json"  # path to the config file
nn, pca, f1_history, learning_rate, X_train, y_train, X_test, y_test = load_session_from_config(config_file)You can also use these functions for a regression neural network and if you did not use PCA:save_session(nn, learning_rate, X, y, X_test, y_test, iterations, err_history)  # saves file in 'config_file'

# returns None for values that weren't set, like pca and f1_history
nn, pca = load_from_config(config_file)EvaluationMetricsFor evaluating a trained model intrinsic metrics can be calculated using the methods provided inmetrics.py.
These include accuracy and the F1 score, which essentially compare the predictions of the modelhto the ground truthy.
The output differs, depending if binary or multiclass classification is demanded.from netneural.network.metrics import get_accuracy, get_f1_score

accuracy_per_class, accuracy_mc = get_accuracy(h_multiclass, y_multiclass)  # additionally returns accuracy per class
accuracy = get_accuracy(h, y)  

f1_score_per_class, f1_score_mc = get_f1_score(h_multiclass, y_multiclass)
f1_score = get_f1_score(h, y)For the format of the input values it is required that the rows stand for the number of classes.The error can be calculated using theget_error()function of the NeuralNetwork class:errors = nn.get_error(h,y)  # returns error for every data point in a list
error = errors.mean()  # final errorWhen using a regression neural network, you don't have to take the mean over the error. We use the MSE, which will already
output the mean over the errors:error = nn.get_error(h,y)  # for nn with regression=TruePlottingFor making plotting a little bit easier, two functions are introduced in theplot_lib.pyscript. One that
plots a line and one function that creates a scatter plot.from netneural.network.plot_lib import plot_line, plot_scatter

plot_line(list_to_plot, x_label, y_label, title)  # plots a line
plot_scatter(x_data, y_data, colors, x_label, y_label, title):  # creates scatter plotPredictingFinally, we can use the NeuralNetwork instance for predicting values. This can be done using thepredict()provided in
the NeuralNetwork class:predictions = nn.predict(input_values)The input values of this function need to be anumpy array, also if it is only one value.