Before revision: While there have been many proposals on how to make AI algorithms more transparent , few have attempted to evaluate the impact of AI explanations on human performance on a task using AI. We propose a Twenty-Questions style collaborative image guessing game, Explanation-assisted Guess Which (ExAG)  as a method of evaluating the efficacy of explanations  in the context of Visual Question Answering (VQA) - the task of answering natural language questions on images.
After revision: While there have been many proposals on how to make AI algorithms more transparent , few have attempted to evaluate the impact of AI explanations on human performance in conducting human-AI collaborative tasks. To bridge the gap, we propose a Twenty-Questions style collaborative image guessing game, Explanation-assisted Guess Which (ExAG) as a method of evaluating the efficacy of explanations in the context of Visual Question Answering (VQA) - the task of answering natural language questions on images.