[misc] Fix flickering test by improving the test isolation.

(cherry picked from commit 2939577c8c1e2ab5a69380877b1958c51b2de662)