Why are you so sure ?
In any case, this assertion should appear immediately after the call to SetServerListenState().