Title: Analysis of High-order Interactions in Shapley value for Model Interpretation
Primary area: interpretability and explainable AI
Keywords: explainable AI, Shapley value, Interaction

Abstract: The Shapley value is a game-theoretic framework for fairly allocating a utility function’s output among participating players, and is often interpreted as the expected marginal contribution of each player under random coalitions. However, when applied to complex functions—such as interpreting black-box models—the Shapley value implicitly encodes not only the individual effects of the players, but also their interactions. In this study, we derive a generalized decomposition of the Shapley value into higher-order interaction terms. We prove that computing the Shapley value is equivalent to decomposing the characteristic function into interaction terms of arbitrary order and evenly distributing each term among the involved players. We also provide an unbiased estimator of interactions via permutation sampling for practical computation. Furthermore, we discuss how the Shapley value implicitly subsumes the higher-order interactions, and how this property limits interpretability when the sampled interactions exhibit large deviations depending on the context. Our analysis, including both formal case studies with mathematical derivations and empirical evaluations on deep neural networks, demonstrates that the variance of lower-order interactions can serve as an indicator of hidden higher-order structures, guiding when such interactions should be further explored. Overall, our framework advances an interaction-based perspective on Shapley value, offering both theoretical and practical directions for interpreting complex model behaviors.