ceph/ceph — pull request #2727 by yuandong1222 (2014-10-15 11:31:23+00:00)
Title: Keep osd opwq worker wake for following op

OSD OpWQ worker thread sleep immediately if no op in pgqueue, so the next op must wake up one thread to process. The context switch between threads cost about 100us per IO.

This patch keep one worker process wake for some time (10ms by default) after it finished all the work, so if the following op comes soon enough, there will be no context switch and it will reduce the time cost by OpWQ from 100us to less than 50us.

Here is test result for 4K Object WriteFull, each case tested 3 times and average 10000 operations:

TimeCost(shard_num = 5, thread_per_shard = 2)
*master: 21.2s, 20.5s, 19.4s
*patched: 19.4s, 16.9s, 17.3s
OpWQ latency(shard_num = 5, thread_per_shard = 2)
*master: 97.3us, 99.7us, 106.7us
*patched: 45.8us, 43.8us, 42.2us

I also tested only one shard for all PGs with more threads,so the hot thread can do work better. And the result is better for OpWQ latency too:

TimeCost(shard_num = 1, thread_per_shard = 8)
*master: 23.7s, 20.6s, 20.4s
*patched: 19.8s, 18.2s, 18.5s
OpWQ latency(shard_num = 1, thread_per_shard =8)
*master: 97.3us, 99.7us, 106.7us
*patched: 25.7us, 26.5us, 18.5us