2.1 Cherry Pick 2862 Leaking Netty threads causes file handle exhaustion

Cherry picking the 2.1 related commit.
Making test more dry as suggested in comments (after close)