Before revision:  Our observations motivate the need to formalize and verify common assumptions in interpretability , while our ] evaluation framework and synthetic dataset serve as a testbed to rigorously analyze instance-specific interpretability methods.
After revision:  Second, we then introduce BlockMNIST, an MNIST-based semi-real dataset, that by design encodes a priori knowledge of discriminative features. Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between input gradient attributions of standard and robust models. Finally, we theoretically prove that our empirical findings hold on a simplified version of the BlockMNIST dataset. Specifically, we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specific signal coordinates, thus grossly violating assumption (A). Our findings motivate the need to formalize and verify common assumptions in interpretability , while our ] evaluation framework and synthetic dataset serve as a testbed to rigorously analyze instance-specific interpretability methods.