Make logging optional

In performance benchmarks, especially when using lots of short
connections, logging every connect & disconnect can have a significant
impact.