Quantifying Inherent Randomness in Machine Learning Algorithms

Download This Paper Open PDF in Browser Add Paper to My Library Share: Permalink Using these links will ensure access to this page indefinitely Copy URL Quantifying Inherent Randomness in Machine Learning Algorithms 14 Pages Posted: 11 Jul 2022 See all articles by Soham RasteSoham RasteCorporate Model Risk, Wells FargoRahul SinghCorporate Model Risk, Wells FargoJoel VaughanWells Fargo BankVijay NairCorporate Model Risk, Wells Fargo Date Written: June 27, 2022 Abstract Most machine learning (ML) algorithms have several stochastic elements, and their performances are affected by these sources of randomness. This paper uses an empirical study to systematically examine the effects of two sources: randomness in model training and randomness in the partitioning of a dataset into training and test subsets. We quantify and compare the magnitude of the variation in predictive performance for the following ML algorithms: Random Forests (RFs), Gradient Boosting Machines (GBMs), and Feedforward Neural Networks (FFNNs). Among the different algorithms, randomness in model training causes larger variation for FFNNs compared to tree-based methods. This is to be expected as FFNNs have more stochastic elements that are part of their model initialization and training. We also found that random splitting of datasets leads to higher variation compared to the inherent randomness from model training. The variation from data splitting can be a major issue if the original dataset has considerable heterogeneity.There are several implications for practitioners: When performance results of multiple ML algorithms are compared, one must keep in mind that part of any observed differences can be due to inherent randomness. Comparisons of ML algorithms in the literature or with deterministic algorithms typically do not control or account for such variation. So users should be cautious with claims that one algorithm is better than another due to small differences in performances. It is recommended that users provide some measure of variation from the source of randomness, together with the performance metrics, to allow a proper comparison of algorithms. In ML, the random seed used for model initialization and training is an integral part of the algorithm, and it should be reported along with the results so that they can be reproduced. The variation from randomness in data splitting should be controlled by using the same training and test subsets for the different algorithms that are being compared. This is also true if one uses cross-validation to assess performance. Keywords: Model Training, Reproducibility, Variation Suggested Citation: Suggested Citation Raste, Soham and Singh, Rahul and Vaughan, Joel and Nair, Vijayan N., Quantifying Inherent Randomness in Machine Learning Algorithms (June 27, 2022). Available at SSRN: https://ssrn.com/abstract=4146989 Soham Raste (Contact Author) Corporate Model Risk, Wells Fargo ( email ) United States Rahul Singh Corporate Model Risk, Wells Fargo ( email ) 301 South Tryon StreetWells Faro Three 10th FloorCharlotte, NC 28288United States Joel Vaughan Wells Fargo Bank ( email ) United States Vijayan N. Nair Corporate Model Risk, Wells Fargo ( email ) 301 South Tryon StreetWells Faro Three 10th FloorCharlotte, NC 28288United States Download This Paper Open PDF in Browser Do you have a job opening that you would like to promote on SSRN? Place Job Opening Paper statistics Downloads 3 Abstract Views 9 PlumX Metrics Related eJournals Machine Learning eJournal Follow Machine Learning eJournal Subscribe to this fee journal for more curated articles on this topic FOLLOWERS 386 PAPERS 4,551 Feedback Feedback to SSRN Feedback (required) Email (required) Submit If you need immediate assistance, call 877-SSRNHelp (877 777 6435) in the United States, or +1 212 448 2500 outside of the United States, 8:30AM to 6:00PM U.S. Eastern, Monday - Friday. Submit a Paper Section 508 Text Only Pages SSRN Quick Links SSRN Solutions Research Paper Series Conference Papers Partners in Publishing Jobs & Announcements Newsletter Sign Up SSRN Rankings Top Papers Top Authors Top Organizations About SSRN SSRN Objectives Network Directors Presidential Letter Announcements Contact us FAQs Copyright Terms and Conditions Privacy Policy We use cookies to help provide and enhance our service and tailor content. To learn more, visit Cookie Settings. This page was processed by aws-apollo5 in 0.216 seconds