collie/sheepdog — Issue #27: sheep process starved and eating 100% CPU under heavy IO
Author: douardda
Created: 2012-03-03 13:30:42+00:00
State: open

Hi,

Still playing with sheepdog, running IO benchmarks in my VMs, I've hit another problem (see issue #26) while running dbench in VMs: I have a sheep process eating 100% of a cpu on 2 nodes (and the fs in unresponsible on 2 of my VMs using sheepdog). On these 2 nodes, ``collie`` command does not respond (locked).

On the still working node, I have:

> collie cluster info
> Cluster status: running
> Cluster created at Fri Mar 2 19:14:22 2012
> Epoch Time Version
> 2012-03-02 19:14:22 1 [172.17.3.20:7000, 172.17.3.21:7000, 172.17.3.22:7000]

> $ collie node info
> Id Size Used Use%
> 0 1.8 TB 18 GB 0%
> 1 1.8 TB 19 GB 1%
> 2 1.8 TB 15 GB 0%
>Total 5.3 TB 52 GB 0%
>
>Total virtual image size 40 GB

> $ collie vdi list
> Name Id Size Used Shared Creation time VDI id
> squeeze-test_3 1 10 GB 8.4 GB 956 MB 2012-03-03 01:04 406758
> squeeze-test_2 1 10 GB 8.1 GB 960 MB 2012-03-03 01:04 40690b
> squeeze-test_1 1 10 GB 8.3 GB 964 MB 2012-03-03 01:04 406abe
>s squeeze-test 1 10 GB 1.3 GB 0.0 MB 2012-03-02 19:15 ecf746
> squeeze-test 2 10 GB 0.0 MB 1.3 GB 2012-03-03 01:03 ecf747

A strace on the sheep process shows it is looping on:

> epoll_wait(5, {{EPOLLERR|EPOLLHUP, {u32=67111312, u64=140600116513168}}, {EPOLLERR|EPOLLHUP,{u32=4026533136, u64=140599780967696}}, {EPOLLERR|EPOLLHUP, {u32=4026533408, u64=140599780967968}}, {EPOLLERR|EPOLLHUP, {u32=4026533680, u64=140599780968240}}, {EPOLLERR|EPOLLHUP, {u32=4026533952, u64=140599780968512}}, {EPOLLERR|EPOLLHUP, {u32=4026534224, u64=140599780968784}}, {EPOLLERR|EPOLLHUP, {u32=4026534496, u64=140599780969056}}, {EPOLLERR|EPOLLHUP, {u32=4026534768, u64=140599780969328}}, {EPOLLERR|EPOLLHUP, {u32=4026535040, u64=140599780969600}}, {EPOLLERR|EPOLLHUP, {u32=4026535312, u64=140599780969872}}, {EPOLLERR|EPOLLHUP, {u32=4026535584, u64=140599780970144}}, {EPOLLERR|EPOLLHUP, {u32=67146816, u64=140600116548672}}, {EPOLLERR|EPOLLHUP, {u32=4022399216, u64=140599776833776}}, {EPOLLERR|EPOLLHUP, {u32=4022399488, u64=140599776834048}}, {EPOLLERR|EPOLLHUP, {u32=4022399760, u64=140599776834320}}, {EPOLLERR|EPOLLHUP, {u32=4022400032, u64=140599776834592}}, {EPOLLERR|EPOLLHUP, {u32=4022400304, u64=140599776834864}}, {EPOLLERR|EPOLLHUP, {u32=4022400576, u64=140599776835136}}, {EPOLLERR|EPOLLHUP, {u32=4022400848, u64=140599776835408}}, {EPOLLERR|EPOLLHUP, {u32=4022401120, u64=140599776835680}}, {EPOLLERR|EPOLLHUP, {u32=4022401392, u64=140599776835952}}, {EPOLLERR|EPOLLHUP, {u32=4022401664, u64=140599776836224}}, {EPOLLERR|EPOLLHUP, {u32=4022401936, u64=140599776836496}}, {EPOLLERR|EPOLLHUP, {u32=4022402480, u64=140599776837040}}, {EPOLLERR|EPOLLHUP, {u32=4022402752, u64=140599776837312}}, {EPOLLERR|EPOLLHUP, {u32=67187136, u64=140600116588992}}, {EPOLLERR|EPOLLHUP, {u32=88097008, u64=140600137498864}}, {EPOLLERR|EPOLLHUP, {u32=88097280, u64=140600137499136}}, {EPOLLERR|EPOLLHUP, {u32=4022402208, u64=140599776836768}}, {EPOLLERR|EPOLLHUP, {u32=88097552, u64=140600137499408}}, {EPOLLERR|EPOLLHUP, {u32=88097824, u64=140600137499680}}, {EPOLLERR|EPOLLHUP, {u32=88098096, u64=140600137499952}}}, 128, 1000) = 32

After killing the stuck sheep process on one node, the cluster seems to be back to a normal functionnal state:

>$ collie cluster info
>Cluster status: running
>
>Cluster created at Fri Mar 2 19:14:22 2012
>
>Epoch Time Version
>2012-03-03 14:22:17 3 [172.17.3.20:7000, 172.17.3.21:7000, 172.17.3.22:7000]
>2012-03-03 14:22:17 2 [172.17.3.20:7000, 172.17.3.22:7000]
>2012-03-02 19:14:22 1 [172.17.3.20:7000, 172.17.3.21:7000, 172.17.3.22:7000]

And one of my VMs is back to a functional state (IO unblocked) but **not** the other one (the one running on the node where I killed the sheep process).