Toward a characterization and systematic evaluation framework for theories and models of human , social , behavioral , and cultural processes

A suite of tools and methodologies has been developed for the characterization and systematic evaluation of human, social, behavioral, and cultural (HSBC) modeling and simulation (M&S) systems. State-of-the-art evaluation methods, which are based on empirical and retrospective validation, Monte Carlo type explorations of the hypotheses space, or visual analytics, may not work well for complex nonlinear and adaptive systems, especially when the space of implicit or explicit parameters is large or when validation data are missing, incomplete, and noisy. This paper focuses on the characterization and evaluation of HSBC theories and models with respect to the ability to tease out causal insights, detect and forewarn about possible emergence, predict predominant or interesting behavior, extract dominant processes, and model extreme behavior. Emergence is defined in various operational contexts, and possible trade-offs with predictability are investigated. The ability to generate actionable insights, for example relatively small advance actions which can reduce the possibility of undesirable outcomes, is critically examined. The simulation test-bed comprises a system of differential equations and disparate implementations of an agent-based model across multiple computational platforms. The evaluation methods range from sensitivity analysis and data mining to process modeling, while the metrics range from information theoretic to statistical. The results suggest that data mining and sensitivity analysis of simulation outputs and observations may offer a way to detect precursors of certain types of emergence, a means to extract causal factors and dominant processes, and possibly offer ways to prevent undesirable outcomes through imperceptible, preemptive actions. In addition, the hypothesis that metrics can be developed to compare and contrast simulations, explore complexity-based emergence, short-term predictability, tradeoffs between emergence versus predictability, and extreme behavior, appears to be supported. However, further tests and investigations are required not only with a wider class of theories, models, and M&S systems, but also with additional observed or simulated data and more comprehensive or better-designed performance measures. *Corresponding Author Contact: Computational Sciences and Engineering Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831-6085; Phone: (865) 241-1305; Email: gangulyar@ornl.gov Credits and Acknowledgments: The first author and several of the co-authors are employed by the Oak Ridge National Laboratory (ORNL), managed by UT Battelle, LLC, for the U.S. Department of Energy under Contract DE-AC0500OR22725. This manuscript was developed based on funding received from the Defense Advanced Research Projects Agency (DARPA) of the US Department of Defense (US DoD). The views expressed in the abstract are those of the authors and do not necessarily represent the opinions of their institutions or of the US DoD. The United States Government retains, and the publisher by accepting the article for publication, acknowledges that the United States Government retains, a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. DARPA Foundation “Task 6” Report Systematic Evaluation Framework 2 of 40