The start of each episode must start at a state, and all state-action pairs have a chance of being used as a starting state-action if a stochastic soft policy is being used for exploration.