## 1 Introduction
The LIME algorithm creates an explanation for an individual prediction by creating a simple, interpretable linear model that approximates the behavior of the opaque model in the vicinity of the prediction. Another significant technique is SHapley Additive exPlanations (SHAPs) [18], which use a game-theoretic approach (Shapley values) to demonstrate feature importance for each prediction. Additional methods include visualization techniques and explanations by example. We provided a broad, critical overview of the current shallow approaches to XAI in [19; 20]. The previously mentioned approaches are considered shallow because they offer explanations at the same abstraction level at which decisions are made, without involving a deeper theoretical investigation or understanding of the model's underlying principles and external declarative knowledge. An inherent limitation of shallow methods is their inability to fully capture the model's underlying complexity. Therefore, it is crucial that these methods be used in conjunction with deeper, more transparent techniques to achieve a more comprehensive understanding of the model's behavior and structure. Genetic Programming (GP) is increasingly used within the field of XAI, predominantly through two primary methodologies that are in line with the current XAI taxonomy, namely intrinsic and post hoc interpretability. The intrinsic interpretability approach involves the development of GP models that are inherently interpretable by design. Such models leverage symbolic components that naturally facilitate human comprehension. On the other hand, post hoc interpretability in GP focuses on explaining (globally and locally) models typically considered opaque by simplifying or approximating their decision-making processes into forms that are more comprehensible to users, such as decision trees or linear models [21]. GP is generally considered to be explainable, particularly when compared to other ML methods (e.g., deep learning). However, more complex evolutionary models can be challenging to understand, and different XAI techniques can be applied in order to make them interpretable [22]. This paper explores the issue of model-driven explainable artificial intelligence. In fact, the proposed approach is related to concepts such as model discovery, model-based reasoning [9], symbolic regression, causal modeling [23], function discovery, and discovery by design [24]. As mentioned before, machine learning is a well-matured discipline, and exploration of even large datasets can be performed in an efficient way, leading to accurate and operational prediction and decision models. On the other hand, most methods tend to produce black-box-type models only. In the cases of numerous practical applications, it is also required to uncover the inner decision/prediction mechanism so that an in-depth understanding of the causal and functional dependencies becomes possible and some justification or responsibility for the decision can be considered. This is so in cases where justification, explanation, and validation of the produced results are required, e.g., for confirmation of correctness, safety, or reliability. Unfortunately, the explanatory mechanisms developed so far, such as LIME or SHAP, are based on the same input data that are the input for machine learning algorithms. This seems to be an in-born limitation and an intrinsic conceptual error, as no expert domain knowledge can come into play. In this paper, the idea of building open, white-box explanatory models is put forward. To do that, we propose employing grammatical evolution tools supported by expert domain knowledge. The resulting models are aimed at explaining the structure and behavior of the decision model in terms of components, connections, and simple functional dependencies.