## 3 Proposed Approach
### Activations Space, Linear SVCs and Concept-Based Neural Fitness Function  
More precisely, given a sub-concept \(c\), and a set \(S\) of possible input instances, we can express \(c\) by partitioning \(S\) into two subsets \(P_{c}\) and \(N_{c}\) containing, respectively, the positive and negative instances with respect to concept \(c\). In [2], concept activation vectors (CAVs) were used to explore the ability of layers of the network when employed to separate input instances belonging to \(P_{c}\) or to \(N_{c}\). Here, we define a similar approach: we consider the vector space obtained by concatenating the activations of all the neurons of a layer, and we train a linear support vector classifier (SVC) [26] to correctly recognize the activations yielded by the instances belonging to \(P_{c}\) and those belonging to \(N_{c}\). The rationality in choosing SVCs and linear kernels as simple concept classifiers, working on the layer activations generated from specific classes of input instances, is inspired by that found in a previous work [27], which compares activations of DNN to signals from the brain, when stimulated with some sensorial input. Formally, given a feedforward layer \(l\), for each \(s\in S\), we compute \(f_{l}(s)\in\mathbb{R}^{n\times m}\), and we flatten its rows by applying a transformation \(f_{l}^{*}\colon\mathbb{R}^{n\times m}\to\mathbb{R}^{t}\), where \(t=nm\) and \(f_{l}(s)_{x,y}=f_{l}^{*}(s)_{nx+y-1}\), for all \(x\in\{1,\ldots,n\}\) and \(y\in\{1,\ldots,m\}\). Then, we define the binary ground truth for the concept \(c\) by means of the two sets \(S_{1}=\{f_{l}^{*}(s)\colon s\in P_{c}\}\) and \(S_{0}=\{f_{l}^{*}(s)\colon s\in N_{c}\}\) and then we train a linear SVC on those activation vectors. This binary classifier \(V_{c,l}\), corresponding to a separating hyperplane for the activations space of layer \(l\), can then be used to determine the direction of the concept \(c\). By definition, a linear SVC \(V_{c,l}\) separates the instances with respect to the sub-concept \(c\), by means of a decision function \(d_{c,l}\colon\mathbb{R}^{t}\to\mathbb{R}\) that represents the signed distance between the hyperplane \(V_{c,l}\) and the point determined by the activations yielded by the instances in the layer \(l\). Specifically, since we are considering binary classification, the SVC predicts an instance \(s\) to be in class 0 or in class 1 depending on whether \(d_{c,l}(f_{l}^{*}(s))\) assumes a negative or a positive value. In this way, an instance \(s\) is predicted to represent or not to represent a sub-concept \(c\) according to where, in the space, the activations generated in layer \(l\) are placed with respect to the separating hyperplane \(V_{c,l}\). Given this premise, we define as the fitness function for our evolutionary algorithm the signed distance \(d_{c,l}(s)\) computed by feeding the network with the individual \(s\) and by considering the SVC \(V_{c,l}\) and the corresponding point in the activations space of the feedforward layer \(l\). Since high-level programming languages usually build source code programs under the control of a context-free grammar, within the scope of this paper, the idea is to apply a grammar-based genetic algorithm in order to evolve programs according to a fitness function that represents the direction of a concept. To this end, DSGE [15] is the easy choice since it allows to efficiently evolve individuals that comply with a given formal grammar. As shown in Figure 2, which reports an example of the decoding process of genotype into a phenotype on the basis of a simple BNF grammar, in DSGE, the genotypes are represented by sequences \(G_{1},\ldots,G_{n}\), each listing the derivation steps of a grammar. At each position \(k\), \(G_{k}\) is referred to a non-terminal symbol of the given context-free grammar, and it is made of a list of integers whose length represents the number of times the \(k\)-th non-terminal is expanded, while the values represent, for each expansion,the index in the grammar of the applied rule for that non-terminal.