# Root Cause Attribution of Delivery Risks via Causal Discovery with Reinforcement Learning
## 2 Methodology
Our goal is to maximize the reward across all possible directed graphs, which is equivalent to solving:  
\[x_{i}:=f_{i}(\mathbf{x_{pa(i)}})+n_{i},\quad i=1,2,\ldots,d, \tag{1}\]0  
The expected return during training can be expressed as:  
\[x_{i}:=f_{i}(\mathbf{x_{pa(i)}})+n_{i},\quad i=1,2,\ldots,d, \tag{1}\]1  
where \(\pi(\cdot\mid s)\) represents the strategy, and \(\varphi\) denotes the neural network parameters for graph generation. During training, random samples are drawn from the dataset \(X\). The encoder output is fed into the critic, which is a simple two-layer feedforward neural network with a Tanh activation function. The critic minimizes the mean squared error between predicted and actual rewards and penalties and is trained using the Adam optimizer. Moving on to the inverse information entropy (IIE), based on [33], the causal strength is defined as:  
\[x_{i}:=f_{i}(\mathbf{x_{pa(i)}})+n_{i},\quad i=1,2,\ldots,d, \tag{1}\]2  
where \(S(p_{x_{1}})\) and \(S(p_{x_{2}})\) represent the information entropies of variables \(x_{1}\) and \(x_{2}\), respectively. For finite datasets, the entropy of the probability distribution for risk factors can be estimated using the entropy estimator, as described by [37; 38]:  
\[x_{i}:=f_{i}(\mathbf{x_{pa(i)}})+n_{i},\quad i=1,2,\ldots,d, \tag{1}\]3  
where \(\psi(n)\) is the digamma function, and \(n\) represents the dimensionality of the dataset \(X\). Based on Equation (10), the IIE causal strength is computed using raw data. However, due to potential differences in dimensionality among the variables in the raw data, the calculated causal strength might deviate from its true value. To account for this, the IIE causal strength for normalized data is given by:  
\[x_{i}:=f_{i}(\mathbf{x_{pa(i)}})+n_{i},\quad i=1,2,\ldots,d, \tag{1}\]4  
Finally, we applied a log transformation to the computed causal strengths to normalize their distribution and stabilize variance. This transformation enhances the interpretability of the results, making it easier to compare the relative impact of different factors on delivery risks:  
\[x_{i}:=f_{i}(\mathbf{x_{pa(i)}})+n_{i},\quad i=1,2,\ldots,d, \tag{1}\]5  
Next, we propose the algorithm and hyperparameter we used for our causal discovery via RL for root cause attribution on late delivery. As can be seen in Algorithm 1, the proposed approach is as follows:  
```
1:Input: Data pertaining to late delivery risk. 2:Step 1: Preprocess the observed data and set hyperparameters, including the number of epochs, learning rate, and scoring function. 3:Step 2: Feed the preprocessed data into the encoder, and pass the encoder outputs along with those from the critic to the decoder. The decoder performs calculations using Equation (3) and samples according to Equation (4). 4:Step 3: Rate the generated directed graph using Equation (4), and return rewards and penalties to the critic. Track the maximum reward using Equation (8). 5:Step 4: Check if the preset number of iterations has been reached. If not, return to Step 2; otherwise, output the directed graph with the maximum score. 6:Step 5: From the final directed graph, calculate causal strengths with log transformation (13), remove edges with a strength less than 0.1, and output the refined causal graph with the associated strengths. 7:Step 6: Generate an explainable delivery risk report from the output graph, tailored for business stakeholders. ```  
**Algorithm 1** Causal Discovery with RL for Late Delivery Risk  
As shown in Table 2, the training of our model was performed with a learning rate of 0.001, 500 epochs, and a batch size of 64, chosen through cross-validation to ensure optimal performance. We used the Adam optimizer for its robustness in managing sparse and noisy gradients, while L2 regularization (\(\lambda=0.0001\)) and a dropout rate of 0.2 were applied to mitigate overfitting. Additionally, we implemented a custom reward function designed to minimize supply chain risks and set a causal strength threshold of 0.05 to filter out weaker relationships.