# Designing Trojan Detectors in Neural Networks Using Interactive Simulations
## 3 Methods
### Design of Neural Network Measurements
#### 3.2.3 Neural Network Inefficiency of Encoding Classes  
The use of KL divergence [38] is borrowed from the source coding theorem [37]. KL divergence is a measurement of how inefficient it would be on average to code a histogram of NN layer states per class label using a reference histogram as the true distribution for coding. From coding, the reference histogram is defined below as the outcome of a uniform distribution over states assigned to each label. Figure 4 (bottom) shows example results of KL divergence values derived per layer and per class label that can be used to compare against values obtained from other datasets, e.g. datasets with trojans. The rationale behind choosing entropy-based KL divergence with probability ratios is based on three considerations. First, entropy-based measurement is appropriate because which state is assigned to predicting each class label is a random variable and a set of states assigned to predicting each class label is random. Second, probability-based measurement is needed because training data represent samples from the underlying phenomena. Furthermore, while training data might be imbalanced (the number of samples per class varies), all training class labels are equally important, and the probabilities of classes should be included in the measurement. Third, the divergence measurement reflects the fact that NN efficiency is measured relative to a maximum NN efficiency that is achieved when sets of states utilize the entire network capacity (representation power). _Mathematical definition:_ Formally, let us denote \(Q_{j}=\{q_{ij}\}_{i=1}^{n}\) as a discrete probability distribution function (PDF) of \(n\) measured NN states and \(P_{j}=\{p_{ij}\}_{i=1}^{n}\) as the PDF of reference (ideal) NN states. The probabilities are associated with each state (index \(i\)) and each class label (index \(j\)). The KL divergence per class label \(j\) is defined at each NN layer in Equation (1). \[D_{KL}(Q_{j}\parallel P_{j})=\sum_{i=1}^{n}(q_{ij}*\log_{2}\frac{q_{ij}}{p_{ij }}) \tag{1}\]  
where \(q_{ij}=\frac{count(i,j)}{p_{ij}npts}\) is the measured count of states normalized by the probability \(p_{j}\) of a class label \(j\) and the number of training points \(npts\). The PDF of reference states per class label uniformly utilizes the number of states assigned to predicting each class label (i.e., 2 classes imply \(\frac{1}{2}\) of all states per label). The reference probability distribution is uniform across all assigned states. Thus, all reference probabilities can be computed as  
Figure 5: Example of multiple states in Layer 1 used for predicting one class Label P. Inefficiency can be confirmed by removing two nodes in Layer 1 in the simulation. The NN model accuracy after removal is the same as before. where \(m\) is the number of classes and \(n=2^{mnodes}\) is the maximum number of states (\(nnodes\) is the number of nodes per layer). Equation (1) for the Kullback-Leibler divergence is defined only if for all \(x\), \(p_{ij}=0\) implies \(q_{ij}=0\). Whenever \(q_{ij}=0\), the contribution of the corresponding term is interpreted as zero because \(\lim_{x\to 0}(x*\log_{2}x)=0\) (see Appendix C). The case of \"not defined\" takes place when there are more non-zero states than the number of non-zero reference states (i.e., the cardinality of two sets satisfies the equation: \(|Set(q_{ij}\
eq 0)|>|Set(p_{ij}\
eq 0)|\)). This case indicates that a NN has insufficient representation power to encode input dataset into a class label. _Expected properties of KL divergence:_ KL divergence will satisfy a list of basic properties for varying datasets, features, and NN capacities. For example, given an input dataset and a set of features, KL divergence (inefficiency of class encoding) per layer should increase for an increasing number of nodes per NN layer. In another example, given a NN capacity, KL divergence should decrease for datasets with added noise or trojans.