Make sure to close the network allocators

It quite a pain to debug when we have 2K goroutines. With this PR, we limit the amount of goroutine during the tests.