Before revision:  In this work, we introduce an evaluation framework to study this hypothesis for benchmark image classification tasks, and make two surprising observations on CIFAR-10 and Imagenet-10 datasets: (a) contrary to conventional wisdom, input gradients of standard models (i.e., trained on the original data) actually highlight irrelevant features over relevant features; (b)however, input gradients of adversarially robust models (i.e., trained on adversarially perturbed data)starkly highlight relevant features over irrelevant features.
After revision:  In this work, we introduce an evaluation framework to study this hypothesis for benchmark image classification tasks, and make two surprising observations on CIFAR-10 and Imagenet-10 datasets: (a) contrary to conventional wisdom, input gradients of standard models (i.e., trained on the original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models (i.e., trained on adversarially perturbed data)starkly highlight relevant features over irrelevant features.