# Modular Dynamic Neural Network: A Continual Learning Architecture
## 3 Modular Dynamic Neural Network
### Modular Dynamic Classification  
As mentioned, there are two different types of modules present in the presented arquitecture: (a) _endpoints_, which are responsible for storing feature data extracted from a single class during training, and (b) _nodes_, which contain references to two or more sub-modules along with a binary classifier (BC) for each one. Each sub-module can either be an _endpoint_ or a _node_. The top part of Figure 3 shows the difference between _nodes_ and _endpoints_ with a simple example of a potential network structure with three classes. Each _node_ module has its own set of binary classifiers, which, in this case, are simple NN with two output values: a certainty value for \"true\" and a certainty value for \"false\" (both between 0 and 1), where the definition of what is true or false depends on the module in question and its location in the network. Each binary classifier represents one of a _nodes_' sub-modules. The binary classifiers that represent _endpoints_ define \"true\" as the only class they are responsible for, and \"false\" as the classes present in all the other modules and sub-modules in parallel with that _endpoint_. The binary classifiers representing _nodes_ consider all of their sub-module classes as \"true\" and all the classes present in the other parallel modules and sub-modules as \"false\". During the classification process, these binary classifiers are used to select which sub-module to continue with and end up defining a type of path that eventually leads to a final prediction. During the training process, they are used similarly to determine the best positions in which to \"insert\" new modules (see Section 3.2). Figure 3: Illustration of the difference between _node_ modules and _endpoint_ modules, using a demonstration of a potential network structure with three classes. The two bottom rows show a different representation of the same example used in the top row, including an analogy with real-world objects. Going back to Figure 3, the bottom two rows show the same example as shown at the top but using a different representation, with the last one including an analogy with real-world objects. Classes B and C are joined as children of a _node_ (node [B, C]) and class A is on its own. This means that when this network was constructed, the algorithms in the training process calculated that classes B and C were similar to each other, grouped them together and trained new binary classifiers to distinguish between A and B \(\cup\) C. Afterwards, a pair of sub-classifiers were trained to specialize in differentiating between classes B and C. To clarify, in this example, there are 4 binary classifiers present: A (against [B, C]), [B, C] (against A), B (against C), and C (against B). Going into further detail, each binary classifier has the same structure, but naturally, different trained weights. In other words, the number of neurons per layer, the number of layers, the connections, the activation functions, etc., are all the same for every BC; the only difference between them is their trained weights. The main reason for this is to ensure consistency and provide all the BCs with an equal chance of success and avoid \"favoritism\". This is important because both the classification and training processes have moments where comparisons are made between the predictions of the different classifiers. The composition of the binary classifiers consists of six fully connected layers with the following numbers of neurons, 128, 64, 64, 32, 32 and 2, respectively, with rectified linear activation functions (ReLU), except for the last layer. The last layer's activation function is a Sigmoid function. This is because the algorithms that use the output of these classifiers only make use of the true values, so there is no reason to let the false values interfere with the true values, which is what would happen in the case of an activation function, e.g., SoftMax. The architecture of the binary classifier, including the numbers of layers and neurons per layer, are determined empirically.