Before revision: To balance exploration and exploitation, multi-armed bandit algorithms need to conduct inference on the true mean reward of each arm in every time step using the data collected so far  . However, the history of arms and rewards observed up to that time step is adaptively collected and there are known challenges in conducting inference with non-iid data .
After revision: To balance exploration and exploitation, multi-armed bandit algorithms need to conduct inference on the true mean reward of each arm in every time step using the data collected so far at each step . However, the history of arms and rewards observed up to that time step is adaptively collected and there are known challenges in conducting inference with non-iid data .