Visual Analytics Evaluation Based on Judgment Analysis Theory

In this paper we propose a framework to quantitatively evaluate user awareness and the level of support that visual analytics decision support systems (VADS) provide. For the framework, which has a theoretical underpinning from the field of judgement analysis, we propose a model for VADS system. The framework bridges the gap between judgment analysis and VADS evaluation by conceptually connecting judgment analysis concepts to visual analytic. The proposed approach offers an insights based evaluation to measure the importance and the utility of the insights. We propose to model insights and user findings as random variables that parametrize user decisions. The mixed methodology used in our framework has the potential to study user decision process in real situations while producing results that can be generalized. Our contributions in this work appear in the modeling of VADS system and the evaluation framework we propose which quantifies situation awareness. Other advantages include evaluating collaboration and analyzing joint decisions. Some limitations of the framework are also discussed including the requirement of large testing data. Visual Analytics Evaluation Based on Judgment Analysis Theory Mosab Khayat, Student Member, IEEE, and Arif Ghafoor, Fellow, IEEE Fig. 1: The proposed relationship between visual analytics and judgment analysis. The left part of the figure shows the knowledge generation model [33]. The right part shows the lens model for analyzing user decisions Ys collected over N instances. The user utilizes a set of cues X1,X2,. ,Xk which is found during the analysis to render the final decisions. When evaluating these decisions, we need criterion or ground truth Ye to validate the importance of the set of cues used by the user to answer the decision task. Abstract—In this paper we propose a framework to quantitatively evaluate user awareness and the level of support that visual analytics decision support systems (VADS) provide. The framework bridges the gap between judgment analysis and VADS evaluation by conceptually connecting judgment analysis concepts to visual analytic. The proposed approach offers an insights based evaluation to measure the importance and the utility of the insights. We propose to model insights and user findings as random variables that parametrize user decisions. The mixed methodology used in our framework has the potential to study user decision process in real situations while producing results that can be generalized. The proposed framework can also be used for evaluating collaboration and analyzing joint decisions. Some limitations of the framework are also discussed including the requirement of large testing data.In this paper we propose a framework to quantitatively evaluate user awareness and the level of support that visual analytics decision support systems (VADS) provide. The framework bridges the gap between judgment analysis and VADS evaluation by conceptually connecting judgment analysis concepts to visual analytic. The proposed approach offers an insights based evaluation to measure the importance and the utility of the insights. We propose to model insights and user findings as random variables that parametrize user decisions. The mixed methodology used in our framework has the potential to study user decision process in real situations while producing results that can be generalized. The proposed framework can also be used for evaluating collaboration and analyzing joint decisions. Some limitations of the framework are also discussed including the requirement of large testing data.