If listener is nil, Close() causes a panic