# Modular Dynamic Neural Network: A Continual Learning Architecture
## 3 Modular Dynamic Neural Network
### Balanced Training Data  
Research shows that neural networks have higher success rates when trained with balanced data [31], meaning that they should be trained with a similar number of samples per class. As this architecture deals with binary classifiers (true or false), it should aim to use an equal number of true and false samples, leaving us with the problem of deciding what to do when there is a different number of each. The algorithm implemented for the selection of training samples starts by calculating the maximum number of samples per class that can maintain an ideal distribution based on where they are positioned in the network. In this situation, where various groups of classes have their own sub-groups, an ideal distribution does not mean using an equal number of samples per class. It means the same number of samples should be used from each of a _nodes_' children, i.e., if one of these children is an _endpoint_ and one is a _node_, then the same number of samples should be used from each, where the _nodes_' samples are a mix of their children's samples and so on. For example, recalling Figure 3, if a classifier were trained to recognize an _endpoint_ A as true and a _node_ [B, C] as false, and A, B and C all had \(m\) samples each, the balanced distribution of samples for this classifier would be \(m\) samples of A and \(m\) samples of [B, C]. Then, to continue aiming for an equal distribution, the \(m\) samples of [B, C] would consist of \(m/2\) samples of B and \(m/2\) samples of C.  
This example makes the problem appear straightforward, but with different numbers of samples per class and a more complex network structure (more known classes and more nested _nodes_ on the true and false sides), an algorithm is needed for calculating the ideal distribution in any situation and making use of as many samples as possible. Using this equal distribution means classes with fewer samples will reduce the number of samples that can be used from other classes with more samples. So, to minimize this effect, it is recommended to establish a minimum number of samples (MNS) for when classes are added to the network. The currently applied MNS value was determined empirically and was set to 175; nevertheless, when using the MDNN architecture in a real-world situation, there is no knowledge of how many samples will be available per new class, and so a minimum value must be defined. Using MNS = 175 presented good results over the various tests performed, during intermediate tests with the three datasets mentioned before, increased MNS values provided better results. Data augmentation would normally be a solution for increasing the amount of data available for the classes with fewer samples, but as the stored data consist of feature vectors and not raw images (as mentioned), the produced samples would not be reliable. There are two main steps involved in computing the maximum number of samples to obtain a balanced training set: (i) calculate the largest number of samples that allows for the use of the distribution explained above and (ii) recursively divide this number by the number of children in a _node_ until all _endpoints_ are reached. For the first step, (i), the number of sub-modules in the _node_ is multiplied by the number of samples of the sub-module with the largest number of samples. The numbers to be calculated for the sub-modules which are also _nodes_ are calculated in the same way. This means that the algorithm is applied recursively until all the sub-modules belonging to the _node_ in question have been calculated and the _node_ in question itself has also been calculated. When this process is complete, it results in the maximum number of samples that allows for an even distribution of samples for the _node_ in question.