Don't remove log handlers since this makes it impossible to customize the logging system.

This effectively reverts bcc284686ce7b994ac47308b52b87d292cfabe2f.