cea-hpc/clustershell — Issue #131: clubak is slow with heterogeneous output
Author: thiell
Created: 2012-03-03 16:48:18+00:00
State: open
Labels: Scripts

When running {{{clush}}} and {{{clubak}}} against {{{pdsh}}} and {{{dshbak}}} with a non-homogeneous output like {{{dmesg}}}, ClusterShell performance are really bad.

In the following results, test are running against 124 nodes, for a total of 111,000 different output lines. dmesg output were pre-generated to avoid side-effects.

* Reference time, without gathering:
* {{{pdsh -S -w ... cat /tmp/dmesg.log}}}: 3.39
* {{{clush -w ... cat /tmp/dmesg.log}}}: 3.53

* Using {{{dshbak -c}}} to gather:
* {{{pdsh -S -w ... cat /tmp/dmesg.log | dshbak -c}}}: 3.53 (+26%)
* {{{clush -w ... cat /tmp/dmesg.log | dshbak -c}}}: 5.95 (+68%)

* Using {{{clubak}}} to gather:
* {{{pdsh -S -w ... cat /tmp/dmesg.log | clubak -c}}}: 5.38 (+58%, +25% dshbak)
* {{{clush -w ... cat /tmp/dmesg.log | clubak -c}}}: 6.60 (+86%, +10% dshbak)

* Using internal clush internal gathering:
* {{{clush -bw ... cat /tmp/dmesg.log}}}: 7.06 (+100%, +18% dshbak, +6% clubak)

Two things are shown here:
* clubak is much slower to gather heterogeneous results that dshbak.
* For strange reasons, clush output is slower to be gathered than pdsh output (it seems clush introduces fewer interleaving in its output than pdsh? TBC)

Two ways to work on that:
* MsgTree could be possibly costy in this kind of outputs
* Display.gather seems to cast NodeSet to str and back to NodeSet. This is very expensive and could very slow the process. {{{nodeset_cmp()}}} is very stressful too.