Before revision:  We answer this question in the affirmative for both regression and linear contextual bandits.
After revision:  We answer this question in the affirmative for both linear regression and contextual bandits.