Indeed, the black-box nature of neural networks is a big issue. This is opposed to a popular machine learning algorithm called a decision tree that is a kind of explainable machine learning. The NN fundamental design is first to mix several weighted input features and, second, to transform that into another feature using a nonlinear transformation. This weighted mixing and nonlinear transformation using activation function layer by layer are largely attributed to making NN a black box.The explainability of NN is a challenging research domain. When it comes to image recognition, we can observe the behavior of the algorithm in each iteration. For convolutional deep neural networks, one can examine a feature map of the convolutional layer to visually find, for a given input, which kernel/part of the NN process and which part of an input image are responsible for a prediction. For example, if you input an image of a car into this NN, which NN parts process wheels, windshield, shape of car etc., can be examined.To explain neural networks, we might use visualization of layer-wise feature relevance propagation, understanding feature interaction, i.e., one can try to understand for what inputs an NN behaves in what ways or how much its accuracy varies for perturbing input features, and what parts of an image an NN gives most attention to.As I said, I work on algorithms called neural trees. These are a type of neural network where neural tree decisions on given inputs propagate through particular branches of the tree, making explainability intrinsic to this class of neural information processing algorithms. You can check my work on Backpropagation Neural Tree, published in Neural Networks (2022) for a deeper look.