Title: Deep Evidence Regression for Weibull targets
Venue: NeurIPS 2023
Accepted: no
Keywords: Deep Learning

Probabilistic Methods

Computational Finance

Credit risk management
Abstract: Machine Learning has invariantly found its way into various Credit Risk applications. Due to the intrinsic nature of Credit Risk, quantifying the uncertainty of the predicted risk metrics is essential, and applying uncertainty-aware deep learning models to credit risk settings can be very helpful. In this work, we have explored the application of a scalable UQ-aware deep learning technique, Deep Evidence Regression and applied it to predicting Loss Given Default. We contribute to the literature by extending the Deep Evidence Regression methodology to learning target variables generated by a Weibull process and provide the relevant learning framework. We demonstrate the application of our approach to both simulated and real-world data.

UQ for Credit Risk Management: A deep evidence regression approach

Abstract

11

12

1 Introduction

1.1 Credit Risk Management

Credit risk management is assessing and managing the potential losses that may arise from the 13 failure of borrowers or counterparties to fulfil their financial obligations. In other words, it identifies, 14 measures, and mitigates the risks associated with lending money or extending credit to individuals, 15 businesses, or other organizations. 16

17

18

19

20

21

Credit risk's anticipated loss (EL) comprises three components: Probability of Default (PD), Loss Given Default (LGD), and Exposure at Default (EAD). PD is the likelihood that a borrower will fail to fulfill their financial commitments in the future. LGD refers to the proportion of the outstanding amount that is lost in the event of default. Lastly, EAD refers to the outstanding amount at the time of default.[8]

22

23

24

25

26

LGD prediction is important as accurate prediction of LGD not only supports a healthier and riskless allocation of capital, but is also vital for pricing the security properly. [8] & [14]. There is a large body of literature using advanced statistical and machine learning methods for prediction of LGD [8]. However the machine learning literature on LGD has yet to address an essential aspect, which is the uncertainty surrounding the estimates and predictions.[4].

UQ techniques like Bayesian Neural Network, Monte Carlo Dropout and ensemble methods as 27 outlined in [1] present a natural first step towards quantifying uncertainty. However, almost all these 28

methods are computationally and memory intensive, and require sampling on test data after fitting 29

30

the network, making them difficult to adapt for complex neural network architectures that involve a

31

large number of parameters.

1.2 Deep Evidence Regression 32

The primary inspiration of this work is taken from the work done by Amini et al in [2]. The paper 33 develops a unique approach, Deep Evidence Regression as a scalable and accurate UQ aware deep 34 learning technique for regression problems. This approach predicts the types of uncertainty directly 35 within the neural network structure, by learning prior distributions over the parameters of the target 36 distribution, referred to as evidential distributions. Thus this method is able to quantify uncertainty 37 without extra computations after training, since the estimated parameters of the evidential distri38 bution can be plugged into analytical formulas for epistemic and aleatoric uncertainty, and target 39 predictions. 40

The setup of the problem is to assume that the observations from the target variable, y i are drawn 41 i.i.d. from a Normal distribution with unknown mean and variance parameters θ = µ, σ 2 . With 42 this we can write the log likelihood of the observation as: 43

Learning θ that maximises the above likelihood successfully models the uncertainty in the data, 44 also known as the aleatoric uncertainty. However, this model is oblivious to its predictive epistemic 45 uncertainty. [2]. Epistemic uncertainty, is incorporated by placing higher-order prior distributions 46 over the parameters θ . In particular a Gaussian prior is placed on the unknown mean and an Inverse47 Gamma prior on the unknown variance. 48

Following from above the posterior p ( µ, σ 2 | γ, ν, α, β ) can be approximated as p ( µ | γ, ν ) ∗ 49 p ( σ 2 | α, β ) . Hence: 50

Amini et al [2] thus find the likelihood of target variable given evidential parameters, as: 51

where θ = { µ, σ 2 } . Then a Neural Network is trained t infer, the parameters m = { γ, ν, α, β } , of this higher-order, evidential distribution.

1.3 Weibull distribution

The Weibull distribution is a continuous probability distribution commonly used in reliability analysis to model the failure time of a system or component.[7] The probability density function (PDF) of the Weibull distribution is given by:

where λ > 0 is the scale parameter and k > 0 is the shape parameter. The scale parameter de58 termines the location of the distribution, while the shape parameter controls the rate at which the 59 failure rate changes over time. There is a body of lietrature that explores the application of the 60 weibull distribution to various credit risk applications. [9] [11]. 61

The work by [12] assumes a normal distribution on LGD values. While this assumption might be 62 true in a lot of settings, however it does not follow in the context of Loss Given Default. While 63 normal distribution is symmetric and has a support over entire real line, however the LGD values 64 are restricted to a range of [0 , 1] and might not necessarily be symmetric. 65

52

53

54

55

56

57

Hence in the section below we provide a novel theoretical framework to learn target variables which 66 follow Weibull distribution. We provide the following theoretical results, in the setting of target 67 variables following a Weibull dataset. 68

69

70

71

72

73

74

Log Likelihood

Mean Prediction

Prediction Uncertainty

We also provide results testing our approach on both simulated and real world dataset.

2 Deep Evidence Regression for Weibull Data

2.1 Problem setup

We consider the problem where the observed targets, y i , are drawn iid from a Weibull distribution, 75 with a known shape or rate parameter k and an unknown scale λ . Although ideally we would want 76 to keep both the parameters unknown, however with both λ and k there are no priors with which 77 likelihood can be computed analytically [3]. Hence we have decided to simplify the problem setup 78 by assuming known shape k . 79

For the above setting we want to place priors on the unknown parameter, λ , such that we are able 80 to get solve for the likelihood of y i given the parameters of the prior distribution. Hence similar to 81

work in [17] and [5], we define the following prior. 82

And we place a Inverse Gamma Prior on θ (9)

2.2 Learning Log-Likelihood 83

Hence we can define likelihood of y i given the higher order evidential parameters α, β can be defined 84 as : 85

Hence the log-likelihood for i'th observation is defined as: 86

We set up our neural network to minimise the negative Log-Likelihood plus some regularisation 87 cost, discussed in section below. 88

89

2.3 Mean Prediction and UQ of prediction

Given the main advanatge of Deep Evidence Regression over other UQ aware deep learning methods 90 like Bayesian NN, esembling etc, is due to existence of analytical solution for both predictions and 91 unceratinty from NN output, without the need for sampling. Hence this section details the derivation 92 of mean prediction and total prediction uncertainty. 93

2.3.1 Mean Prediction 94

(25)

Hence to solve for mean prediction we need to find pdf p ( λ ) . Because we know θ = λ k ∼ 95 Γ -1 ( α, β ) , we can use change of variable to find pdf of λ [18]. 96

The mean prediction can thus be simplified as: 97

2.3.2 Prediction Uncertainty 98

We quantify the total uncertainty as V ar ( Z ) with defined as above, i.e. Z = E [ y i ] 99

With E ( λ ) defined as in 26, we only need E ( λ 2 ) (32)

(34)

Similar to approach outlined in 2.3.1, we get: 100

Hence we can write 101

or 102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

2.4 Regularisation Cost

In this section, we outline the process of regularization during training by implementing a regularisation penalty, which involves assigning a high uncertainty cost. The purpose of this penalty is to inflate the uncertainty associated with incorrect predictions, thereby improving the overall efficacy of the model. As followed in [2], the intuition behind the regularisation cost is to increases the variance of prediction in cases where it's unsure. This utility of this approach has been demonstrated in classification setting by [13] and in regression setting by [2].

Hence we define the Regularisation cost for the i'th observation as

where error i = y i -Z i and Z is defined in 27.

Note The regularization cost mentioned earlier has been determined as the most effective through experimentation. However, in order to precisely determine the coefficients of α and β in the regularization cost, further theoretical analysis is required. By conducting a deeper theoretical investigation, we can establish the optimal values for these coefficients, which will enhance the regularization process and improve the overall performance of the model.

2.5 Neural Network training schematic

The learning process is then set up with a deep neural network with two output neurons to predict the parameters of the prior/evidential distribution, α and β . The neural network is trained using the cost function:

where c is the hyper-parameter governing the strength of regularisation. 121

122

123

124

125

126

127

NN output =m = {α,β}

NN Loss = - Llik+c * Lreg

3 Results and Experiments

In this section, we present the results of experiments conducted on both simulated and real data. The aim was to evaluate the performance of our proposed method and compare it with existing methods. The simulated data was generated based on example data given in [2], while the real dataset was obtained from a peer to peer lending company.

3.1 Simulated Data

Here generate a target variable following a Weibull distribution. The target variable is generated as: 128

The train set is comprised of uniformly spaced x ∈ [ -4 , 4] while test set is x ∈ [ -5 , 5] . The value 129 of λ is varied between [0 . 2 , 0 . 4] to test the effect of noise magnitude on the approach. 130

y~x² + 0.250* weibul(shape =1.6)

y~x² + 0.400*weibull(shape = 1.6)

25

25

20

20

15

15

10

10

5

5

Train

Train

0

Test

Test

-4

-2

0

2

4

4

-2

0

2

4

Next we fit both the original deep evidence regression and proposed weibull version of deep evidence 131 regression. Since our approach assumes known k , k is estimated from the training set. 132

133

134

135

136

137

138

139

Comparing the two versions qualitatively, we observe that the original model's predictions exhibit consistent uncertainty regardless of whether the data is within or outside the distribution. In contrast, the proposed version demonstrates improved capability in capturing prediction uncertainty. The proposed model's prediction interval gradually expands beyond the training data range | x | > 4 , indicating its ability to account for uncertainty in Out-of-distribution data. On comparison, for the benchmark version of the model displays a slightly narrower prediction interval at the edges of the training window, contrary to expectations of interval widening.

140

141

142

Quantitatively to assess the performance of our proposed method, we compare it to the benchmark model by evaluating key metrics such as mean squared error (MSE) and negative log-likelihood

(NLL).

We can see that the proposed version exhibits significantly lower negative log likelihood compared 143 to the original Deep Evidence Regression model. This indicates that the proposed model better 144

eps=0.2, c= 0.02782559402207126

benchmark Model

proposed Model

25

True Test

60

Train

Train

2 std PI

Mean Pred

50

0.5 std PI

Mean Pred

20

Id ps 8

1 std PI

40

15

30

20

10

10

0

10

2

-2

eps=0.4, c= 0.1

benchmark Model

proposed Model

Isa, an

25

Train

Mean Pred

40

Train

Mean Pred

2 std PI

0.5 std PI

Id ps 8

1 std PI

20

30

15

20

10

10

aligns with the actual distribution of the data, capturing the uncertainty more accurately. However, 145 despite this improvement, the original model outperforms in capturing the underlying signal beyond 146 the training window, as evidenced by its lower mean squared error (MSE) values. 147

148

149

150

151

152

153

154

155

3.2 Real Data: Loss Given Default for peer to peer lending

In this subsection, we showcase the utility of our proposed learning approach by extending it to the intricate and complex domain of credit risk management in the context of peer to peer lending. Peer-to-peer lending, which is an emerging form of credit aimed at funding borrowers from small lenders and individuals seeking to earn interest on their investments. Through an online platform, borrowers can apply for personal loans, which are typically unsecured and funded by one or more peer investors. The P2P lender acts as a facilitator of the lending process and provides the platform, rather than acting as an actual lender.

Credit risk management is crucial for peer-to-peer lending data as it helps mitigate the potential 156 default risks associated with borrowers, ensuring a healthier loan portfolio and reducing financial 157

losses. By effectively analyzing and managing credit risk, P2P lending platforms can maintain 158 investor confidence, attract more participants, and sustain the long-term viability of the lending 159 ecosystem. 160

The dataset under consideration pertains to peer to peer mortgage lending data during the period 161 of 2007 to 2014 sourced from Kaggle [16]. However, the data does not include the loss given 162 default values. Instead, the recovery rate has been used as a proxy, which is calculated as the ratio of 163 recoveries made to the origination amount. The dataset contains approximately 46 variables denoted 164 as 'x,' which include features such as the time since the loan was issued, debt-to-income ratio (DTI), 165 joint applicant status, and delinquency status, among others. In total, the dataset comprises around 166 23,000 rows. 167

y-train

Distribution ofy

Histogram of y

0.08

Train

Test

0.06

0.07

0.05

0.06

0.05

0.04

0.04

0.03

0.03

0.02

0.02

0.01

0.01

0.00

0.00

0

20

40

09

80

100

120

0

20

40

09

80

100

120

As described in the approach the shape parameter was found as k = 1 . 254 by fitting a Weibull 168 distribution on the train dataset. Also given the sensitivity of both the approaches to regression 169 cost, hyper parameter optimisation was done to arrive at the best regularisation cost. After arriving 170 at the best regularisation cost 10 trials of neural network training were conducted with this best 171 regularization cost for benchmark and proposed model separately. 172

173

174

175

176

177

178

179

180

181

Similar to the synthetic case, we see that the proposed model demonstrates superior performance compared to the benchmark model in terms of mean squared error (MSE) and negative log likelihood. Additionally, it exhibits the ability to generate more accurate prediction intervals. The difference in performance between the benchmark and proposed models is even more pronounced in this case compared to the simulated data, and the benchmark fails even to retrieve the underlying signal, let alone the prediction uncertainty. To reinforce this, the benchmark model was also run with 0 regularisation cost and it was found to not improve the MSE. This behaviour outlines the difficulty of tuning regularisation parameter for benchmark model. It is also worth noting that the benchmark model also predicts ∞ as uncertainty for a significant number of observations.

4 Related Work 182

This work is primarily inspired by the work done by Amini et al [2], which proposes Deep Evidentail 183 Learning approach. Maximillian et al [12] have used the same approach and shown it's utility to 184 Loss given default for bonds from Moody's recovery database. Additionally there's a huge body of 185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

cforbenchmark=0.05263157894736843,proposed=0.5263157894736843

benchmarkModel

proposedModel

200

200

TrueTest

TrueTest

MeanPred

MeanPred

150 -

1stdPI

2stdPI

150 -

1 std PI

2 std PI

100

100

50

50

0

0

-50

50

-100

-100

0

1000

2000

3000

4000

0

1000

2000

3000

4000

prior work on uncertainty estimation [10] [6] and the utilization of neural networks for modeling probability distributions. In another line of work, Bayesian deep learning utilizes priors on network weights estimated with variational inference [1]. There also exists alternative techniques to estimate predictive variance like as dropout and ensembling which require sampling. [1]

5 Conclusion, limitations and future work

Wepropose an improvement over Deep Evidence Regression, specifically targeted to usecases where the target might follow weibull distribution. We then test the proposed method both on simulated and real world dataset in the context of Credit risk management. The proposed model exhibits enhanced suitability for applications in which the target variable originates from a weibull distribution, better capturing the uncertainty characteristics of such data. Although we have specifically tested the model in the credit risk domain, this method can be applicable to wide variety of safety critical regression tasks where the target variable follows a weibull distribution. The proposed approach thus serves as a valuable tool for capturing and quantifying uncertainty in cases characterized by weibull distributions, thereby enhancing the trustworthiness and explainability of model predictions, ultimately leading to improved confidence in the modeling process and the corresponding decision making.

However, we are not sure if the proposed approach would generalise to other distributions apart from Weibull. Additionally, the proposed model has only two outputs, which could limit its flexibility when compared to the benchmark model, which had four outputs from the neural network. In consequence the proposed model requires a deeper network architecture compared to the benchmark model. Furthermore we find that both the models exhibit a high sensitivity to regularization cost, which means that changes in the regularization coefficient can significantly impact the model's performance. The cylical learning rate as outlined in the [15], which proposes varying the learning rate between reasonable boundary values might be of help to mitigate this issue. Overall, these points suggest that both models have their strengths and weaknesses, and selecting the most appropriate model depends on the specific task requirements and considerations.

Considering the wide application of beta distributions in the credit risk domain, there may also be 212 value in further extending the proposed technique to target variables characterized by beta distribu213 tions, as it has the potential to provide valuable insights and improved modeling capabilities in the 214 context of credit risk management. 215

216

217

218

219

220

221

222

223

224

References

Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U. Rajendra Acharya, Vladimir Makarenkov, and Saeid Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion , 76:243-297, 2021.

Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 14927-14937. Curran Associates, Inc., 2020.

Pasquale Erto and Massimiliano Giorgio. A note on using bayes priors for weibull distribution, 225 2013. 226

Paolo Gambetti, Genevi` eve Gauthier, and Fr´ ed´ eric Vrins. Recovery rates: Uncertainty cer227 tainly matters. Journal of Banking & Finance , 106:371-383, 2019. 228

229

230

231

Pseudodifferential Operators (https://stats.stackexchange.com/users/320438/pseudodifferential operators). Find a conjugate prior for the weibull distribution under reparametrization. Cross Validated. URL:https://stats.stackexchange.com/q/594050 (version: 2022-10-30).

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Chin-Diew Lai, D. Murthy, and Min Xie. Weibull distributions and their applications. Springer Handbook of Engineering Statistics , Chapter 3:63-78, 02 2006.

Matthias Nagl, Maximilian Nagl, and Daniel R¨ osch. Quantifying uncertainty of machine learning methods for loss given default. Frontiers in Applied Mathematics and Statistics , 8, 2022.

Ismail Noriszura, Jamil Jaber, and Siti Mohd Ramli. Credit risk assessment using survival analysis for progressive right-censored data: a case study in jordan. Journal of Internet Banking and Commerce , 22:1-18, 05 2017.

Harris Papadopoulos and Haris Haralambous. 2011 special issue: Reliable prediction intervals with regression neural networks. Neural Netw. , 24(8):842-851, oct 2011.

Rebeca Pel´ aez, Ricardo Cao, and Juan M. Vilar. Bootstrap Bandwidth Selection and Confidence Regions for Double Smoothed Default Probability Estimation. Mathematics , 10(9):125, May 2022.

Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Luc Severeijns. Challenging lgd models with machine learning, 2018. 256

Leslie N. Smith. Cyclical learning rates for training neural networks, 2017. 257

Shawn Sun. Loan data for credit risk modeling. https://www.kaggle.com/datasets/ 258 shawnysun/loan-data-for-credit-risk-modeling , 2021. 259

tomka (https://math.stackexchange.com/users/118706/tomka). Conjugate prior for the weibull 260 distribution. Mathematics Stack Exchange. URL:https://math.stackexchange.com/q/3529658 261 (version: 2020-02-04). 262

Wikipedia contributors. Change of variables - Wikipedia, the free encyclopedia, 2023. [On263 line; accessed 1-May-2023]. 264

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to explore the application of a scalable UQ-aware deep learning technique, Deep Evidence Regression, and applies it to predict Loss Given Default. It extends the Deep Evidence Regression methodology to learn target variables generated by a Weibull process and provides the relevant learning framework. By testing on both simulated and real-world datasets in the context of credit risk management, the proposed method exhibits enhanced suitability for applications in which the target variable originates from a Weibull distribution, better capturing the uncertainty characteristics of such data.\n\nSTRENGTHS:\n1.This paper innovatively extends the Deep Evidence Regression methodology to learn target variables generated by a Weibull process and provides the relevant learning framework.\n2.The article provides a clear and coherent description of the proposed method, including a thorough derivation of the relevant formulas.\n\nWEAKNESSES:\n1.The article contains several mathematical formula writing errors, such as the missing \"dθ\" in line 51 and Equation (16), the missing \"-1\" in Equation (10), and the inconsistencies in Equation (31) and Equation (33) with their original definitions.\n2.The figure legends in the article do not indicate which parts of the content reference them or provide explanations for their content.\n3.The experimental section lacks clear exposition, such as the specific settings of parameters k and λ for the Weibull distribution, as well as the target or subject for the MSE metric.\n\nQUESTIONS:\n1.Can the Weibull distribution still be selected if the LGD values are symmetric?\n2.What is the purpose of calculating E[Z] in Section 2.3.1 and Var(Z) in Section 2.3.2? How are the calculated E[Z] and Var(Z) specifically used in the experiment?\n3.How are the parameters k and λ of the Weibull distribution set in the experiment?\n4.What is the specific object of MSE metric in the experiment?\n5.Why is it claimed that the MSE of the benchmark and the proposed method in Table 1 are similar?\n\nLIMITATIONS:\nThe proposed approach serves as a valuable tool for capturing and quantifying uncertainty in cases characterized by Weibull distributions. Therefore, the key to using this method lies in determining whether the data is suitable for the Weibull distribution. Furthermore, the applicability of the proposed method to distributions other than the Weibull distribution requires further investigation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors tackle the problem of uncertainty quantification for predicting credit risks. Concretely, they have applied a scalable UQ-aware deep learning technique, Deep Evidence Regression to predicting Loss Given Default with uncertainty. Authors argue that the conventional methods use for uncertainty quantification are too computationally and memory intensive and therefore they adopt Deep Evidence Regression as their statistical model.\n\n\nThey extend the framework of Deep Evidence Regression to predict targets generated with a Weibull distribution. The original method relies on the normally distributed targets which is not fitting for credit risk applications. Authors therefore re-derive the necessary formulas based on Weibull distributed targets; concretely they focus on log likelihood needed for training and the mean/uncertainty needed for predictions. Beyond that, authors adapt the regularizers found in the original paper to their purpose.\n\nThe new method is tested empirically on a synthetic dataset with points sampled from a Weibull distributions as well as a single real-world dataset focused on peer to peer mortgage lending data with recovery rate has been used as a proxy for Loss Given Default. Both datasets were used to test the Weibull-based approach against the original method relying on Gaussian distributed targets. The results show the modification made by the authors indeed helps with lower MSE and NLL on both test and train datasets.\n\nSTRENGTHS:\n1. Authors tackle an important problem of uncertainty quantification for risk assesment where robust, and efficient measures of uncertainty are crucial to ensure fair treatment of customers. \n2. Authors provide an expansion of a well established method to make it much more attractive to specialized domains such as risk assesment. This work can be directly useful for practicioners in the field of risk assesment and indirectly useful for researchers in other fields who can adapt the Deep Evidential Regression to work with different, field-specific targets distributions.\n\nWEAKNESSES:\n1. This paper provides an incremental improvement over the original Deep Evidential Regression (DER) work. The derivation of DER with Webull distribution instead of original gaussian target distribution is interesting but the majority of what makes the method impactful remains unchanged. \n2. Authors provide very limited empirical evaluation of their work. The single study with synthetic data provides a good proof of concept but gives little assurance of real-world impact of the work. The single real-dataset study is relatively limited and suffers from some issues: 1/ Authors do not have any baselines beyond the original methods while other methods, for example based on conformal predictions or quantile regression, could be competitive. Any other approach for uncertainty quantification would be useful to gauge the difficulty of the task. 2/ Authors only use a single dataset making the evaluation process less robust, it is possible that this method works well only for this particular dataset rather than for a generic class of problems. 3/ Authors only consider NLL and MSE as metrics for their evaluation, there are may more metrics to quantify uncertainty quantification (such as coverage) that could be used to make the evaluation more robust.\n3. The presentation of the initial method is vague, I undrstand that the DER work is presented in its own paper but presenting it in more detail would make this paper more self-contained, especially given the reliance on the original work.\n\nQUESTIONS:\n1. Authors suggest that “the proposed model has only two outputs, which could limit its flexibility when compared to the benchmark model, which had four outputs from the neural network” which is a little unclear to me. I understand that the model (NN) is as flexible as many free parameters (weights) it has rather than outputs. Maybe this refers to the parametrization of the distribution (gaussian with 4 parameters and Weibull with 2) but in any case, I would like to get some clarity on this claim.\n2. The model has a few hyperparameters such as c (regularization constant), it is currently unclear how were they chosen and what HPO procedure was used.\n\nLIMITATIONS:\nAuthors provide a formulation for extending Deep Evidential Regression for problems where targets are Weibull distributed. However, they do not cover the extension of this method to problems with targets sampled from a different distribution. There is only a small class of problems where Weibull is appropriate and authors do not provide evidence that this approach generalizes beyond Loss Given Default estimation. I would appreciate additional real-world experiments thta could show whether this parametrization (choice of target distribution) can work beyond the single dataset chosen in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the utilization and extension of deep evidential regression for uncertainty estimation in credit risk prediction. The approach assumes a Weibull distribution for the target variable (e.g., LGD or a synthetic target). The authors modify the evidential regression mechanism to accommodate targets from this distribution, providing equations to illustrate the training process. Simple experiments are conducted on synthetic and real-world peer-to-peer lending datasets, showcasing potential improvements over vanilla deep evidential regression for credit risk management.\n\nSTRENGTHS:\nThe incorporation and extension of evidential regression-based UQ into finance-related problems is the main contribution and the key strength of this paper.\n\nWEAKNESSES:\nRefer to Questions.\n\nQUESTIONS:\n1. What is the motivation for choosing the Weibull distribution? Can the authors comment on whether the method can be utilized for other off-the-shelf finance datasets where the target is distributed otherwise or has a high degree of heteroskedasticity ?\n\n2. The experimental setup is very limited.  For the synthetic example, the parabolic function is essentially corrupted with noise drawn from a known Weibull distribution.  Therefore the performance of the proposed approach is expected to be better than the baseline.  Can the authors explain why the MSE of the baselines is better than the proposed approach ? It seems counterintuitve. How exactly are the metrics evaluated ?\n\n3. No references to any figure has been made in the paper.  The figure captions are also very generic. The figures as well as the captions need to be enhanced.\n\n4. How does this approach fare against other uncertainty estimation based methods for e.g, Monte-Carlo dropout ? I believe it is straightforward to implement and verify.\n\n5. Does Figure 3 depict the epistemic uncertainty or the total uncertainties around every sample? \n\n6. Eqn 3 and 4 are general descriptions qualifying Eqn 2.  It would be good if they are not written as equations but written as text.  This is the case with many other redundant equations mentioned in the paper.\n\n7. Identified typos:\n\na) Line 9 - approach to - approach on\n\nb) Line 23 - The citations [23], [24] can be provided at the end of the preceding sentence.\n\nc) Line 35 - predicts the types - predicts the type\n\nd) Line 52 - trained t --> trained to\n\ne) Line 56 -  the fullstop can be removed before the citation [7]\n\nf) Line 72 - real world datasets\n\ng) Line 76 - rate parameter $k$\n\nh) Line 91 - spelling of ensembling\n\ni) Eqn 31,  the mean term E[\\lambda] must be squared.\n\nj) Line 107 - increases --> increase\n\nk) Line 183 - Spelling of evidential needs to be correct\n\n8. Discrepancies between title in the paper and the title shown in the submission page ? Why ?\n\nLIMITATIONS:\nThe paper's experimental setup is quite limited and requires further motivation. The demonstrations illustrating how uncertainty quantification can benefit credit risk problems lack sufficient substantiation. Moreover, the paper has a limited exploration of related work, and the quality of references needs improvement. In its current stage, the paper requires significant improvement in terms of better problem motivation and comprehensive qualitative and quantitative evaluations prior to publication.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}