Fix flaky scheduler factory test.

Subtle flakiness.