Visualizing subjective decision criteria for system evaluation

The tools and methods used to evaluate complex systems in terms of human systems integration (HSI) include software, hardware, models, simulations, processes, and techniques. Hundreds of such tools currently exist to evaluate complex HSI systems, and the number is continuously growing. Many HSI tools are developed in research settings, or in an ad-hoc fashion to meet specific project goals. As such, there is no standard repository of HSI tools. Additionally, as HSI systems become more complex and the number of HSI dimensions in which decision-makers evaluate systems continues to grow, the HSI cost-benefit trade space that explores the costs and benefits of a system on the HSI dimensions expands. Therefore, it becomes more difficult for decision-makers to determine if and to what degree a system meets their specific HSI criteria. Engineers, managers, and other decision-makers often have to evaluate complex HSI systems, such as the case of acquisitions in which decision-makers need to rate and compare particular systems. Since there are hundreds of tools that currently exist to evaluate the HSI aspects of systems, decision-makers may not be familiar with all of them. Additionally, with the HSI cost-benefit trade space continually expanding, decision-makers need a way to reduce the number of HSI tools, or downselect, from the hundreds of HSI tools to a smaller set of desirable tools. This smaller set of desirable tools is defined by users' criteria, which is often vague and ambiguous. This research investigates the visualization techniques of such downselection tools and how the visual representation of information impacts the decisions of decision-makers. The results of this work show that salient information biases decision-makers' decisions when using decision support tools (DSTs). Also, information aggregation did not appear to have an effect on decision-makers' level of automation bias when using decision support tools. There did not appear to be any correlation between decision-makers' trust in DSTs and the extent to which they rely on the results presented in those tools. Additionally, decision-makers' perception of the ease of use of DSTs is correlated with their trust in those tools, although there appeared to be an influence of age on users' perceptions of such tools. While this thesis focuses on the specific example of downselecting a database of HSI tools for a decision maker, the overall processes and research in this effort can generally be applied to any downselection from a large database of elements to a smaller set of desirable elements using both objective and subjective decision criteria of decision-makers. Thesis Supervisor: Mary L. Cummings Title: Associate Professor of Aeronautics and Astronautics Acknowledgments First, I would like to thank my research advisor, Professor Missy Cummings. Thank you for having me in your lab. Thank you for all your guidance, patience, and understanding during my time here. You have made HAL such a great lab in which to work! I will carry the skills that you have given me through the rest of my career. Thank you to Kris for the feedback that you provided on this thesis. Thank you for your help with the statistics, and all your advice throughout this process. Thank you to all my fellow HALiens. Thank you Yves, Farzan, Andrew, Christin, Jason, and Luca for participating in my pilot and actual experiments, and for all your great feedback. Thank you Sylvain and Lee for helping me find participants for my experiments. Thank you to everyone in the lab for making my time here such a great experience. I look forward to keeping in touch with all of you! To my mother and father, words cannot even begin to describe my gratitude to you. You have been my best role models, so thank you for all the great advice. Thank you for believing in me for all these years. Your love and support have always made it easier to persevere through any obstacles. I am so grateful to have such wonderful parents. Thank you to my sister, Avril, for always believing in me. Thank you for always encouraging me. Thank you for always wanting the best for me. You are truly the best sister for which I could ever ask. Lastly, thank you to my closest friends for all the laughs and great times. Preston, thank you for all your encouragement and for always lending a listening ear. Thank you for always being able to put a smile on my face! Jere and Keandra, thank you for always being there for me, not only during my time at MIT, but for the past 15 years. The friendships that I have with all of you have made the past few years such an enjoyable experience. THIS PAGE INTENTIONALLY LEFT BLANK Table of