Before revision:  In this work, we develop theory justifying the use of M-estimation (Van der Vaart, 2000), traditionally used with i.i.d data, to provide inferential methods for a large class of estimators -- including least squares and maximum likelihood estimators -- but now with data collected with  (contextual) bandit algorithms.
After revision:  In this work, we develop theory justifying the use of M-estimation (Van der Vaart, 2000), traditionally used with i.i.d data, to provide inferential methods for a large class of estimators -- which includes estimators based on empirical risk minimization as well as maximum likelihood -- but now with data collected with (contextual) bandit algorithms.