docker/libcontainer — pull request #193 by hustcat (2014-09-17 09:08:10+00:00)
Title: set tx_queuelen to 0 when create veth device

VETH network device will create only one Qdisc queue default, this will become peformance bottleneck.Set tx_queuelen to 0 when create VETH device, kernel will not create Qdisc queue for VETH device, so this will improve network performance.

gperf outputs are as follows:

Samples: 1M of event 'cycles', Event count (approx.): 237661920980
- 13.97% [kernel] [k] _spin_lock
- _spin_lock
- 37.64% dev_queue_xmit
+ 74.94% br_dev_queue_push_xmit
+ 25.06% neigh_resolve_output
+ 19.41% try_to_wake_up
- 17.95% sch_direct_xmit
- 74.23% __qdisc_run
- 97.47% dev_queue_xmit
+ 84.43% br_dev_queue_push_xmit
+ 15.57% neigh_resolve_output
+ 2.53% net_tx_action
- 25.77% dev_queue_xmit
+ 68.81% br_dev_queue_push_xmit
+ 31.19% neigh_resolve_output
+ 17.53% task_rq_lock
+ 3.71% mod_timer
+ 0.86% enqueue_to_backlog
+ 3.82% [kernel] [k] update_curr
+ 1.95% netserver [.] 0x000000000001dd31
+ 1.86% [kernel] [k] update_cfs_shares
+ 1.81% [kernel] [k] tcp_ack
+ 1.76% [kernel] [k] schedule
+ 1.74% [kernel] [k] _spin_lock_irq
+ 1.67% [kernel] [k] dev_queue_xmit

As we can see, spin lock consume a lot of CPU. When set tx_queuelen to 0, the result are as follows:

Samples: 3M of event 'cycles', Event count (approx.): 464264275376
- 8.23% [kernel] [k] _spin_lock
- _spin_lock
- 54.43% dev_queue_xmit
br_dev_queue_push_xmit
br_forward_finish
__br_forward
br_forward
br_handle_frame_finish
br_handle_frame
__netif_receive_skb
process_backlog
net_rx_action
__do_softirq
call_softirq
- 25.78% sch_direct_xmit
- 86.24% __qdisc_run
+ 95.66% dev_queue_xmit
+ 4.34% net_tx_action
+ 13.76% dev_queue_xmit
+ 7.12% task_rq_lock
+ 3.56% try_to_wake_up
+ 2.29% mod_timer
+ 2.04% ipt_do_table
+ 1.57% enqueue_to_backlog
+ 0.60% tcp_v4_rcv
+ 2.28% netserver [.] 0x000000000001a0ca
+ 2.13% [kernel] [k] tcp_ack
+ 2.00% [kernel] [k] update_curr
+ 1.68% [igb] [k] igb_poll

CPU consumption decreased spin locks.I test with netserver in TCP_RR mode,and performance upgrade from 46W+ to 70W+.

Signed-off-by: Ye Yin