Before revision:  We propose a Twenty-Questions style collaborative image guessing game, Explanation-assisted Guess Which (ExAG)  as a method of evaluating the efficacy of explanations  in the context of Visual Question Answering (VQA) - the task of answering natural language questions on images. We study the effect of VQA agent explanations on the game performance as a function of explanation type and quality. We observe that "helpful" explanations are conducive to game performance (by almost 22\% for "excellent" rated explanation games), and having at least one "correct " explanation is significantly helpful when VQA system answers are mostly noisy (by almost 30\% compared to no explanation games).
After revision:  We propose a Twenty-Questions style collaborative image guessing game, Explanation-assisted Guess Which (ExAG) as a method of evaluating the efficacy of explanations in the context of Visual Question Answering (VQA) - the task of answering natural language questions . We show that when AI explains its answers, users succeed more often in guessing the secret image correctly. Furthermore, we show that while good explanations improve human performance, incorrect explanations can degrade game performance as compared to no-explanation games. Notably, a few correct explanations can readily improve human performance in game rounds where the AI system's answers are mostly noisy (by almost 30\% compared to no explanation games).