opened
mocchira
username_0
With this PR, the time to finish leofs_test2 has been boosted from 1,900 secs to 1,300 secs on my dev-box (ubuntu 16.04 LTS on virtualbox with vcpu2 and 8GB), which means around 10 minutes has been reduced per one running.

The key points are
- Get rid of all timer:sleep(s) at src/leofs_test_scenario.erl which added unnecessary delays on each test step
- Instead of the above timer:sleep(s), I added timer:sleep to the points where timer:sleep actually was needed such as waiting for the queue being filled up after leofs-adm (recover|rebalance|delete-bucket).
Fix #1057 - Improve the test time
issue
created
yosukehara
username_1
@username_0 LGTM
comment
created
yosukehara
username_1
I realized that the time spent on the test was shortened:

```bash
$ ./leofs_test -b test -d /home/yosuke/dev/projects/leo-project/leofs/package/
 [{bucket,"test"},
  {leofs_dir,"/home/yosuke/dev/projects/leo-project/leofs/package/"}]

::: START :::

 [SCENARIO-1]
   * create_bucket (create a bucket)
   * recover_file (recover data of a file)
        #1
            * 'storage_0@127.0.0.1':[{leo_recovery_node_queue,1}]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #2
            * 'storage_0@127.0.0.1':[{leo_recovery_node_queue,1}]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #3
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_recovery_node_queue,1}]
            * 'storage_3@127.0.0.1':[N/A]
        #2
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,1}]
            * 'storage_3@127.0.0.1':[N/A]
        #3
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_recovery_node_queue,1}]
        #2
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[{leo_per_object_queue,1}]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #2
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,1}]
            * 'storage_3@127.0.0.1':[N/A]
        #2
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
        #1
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
   * get_objects_not_found (get objects_not_found)....................................................................................................|
   * put_zero_byte_objects (put zero byte objects)....................................................................................................|
   * get_objects (get objects)....................................................................................................|
   * put_objects (put objects)....................................................................................................|
   * get_objects (get objects)....................................................................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
   * del_objects (remove objects)....................................................................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
   * mp_upload_normal (multipart upload)
   * mp_upload_normal_in_parallel (multipart upload in parallel)
   * mp_upload_abort (abort multipart upload)
        #1
            * 'storage_0@127.0.0.1':[{leo_delete_dir_queue_1,2}]
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,2}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,3}]
            * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,2}]
        #2
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,1}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,3}]
            * 'storage_3@127.0.0.1':[N/A]
        #3
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
   * mp_upload_invalid_complete (invalid complete multipart upload)
 [SCENARIO-2]
   * put_objects (put objects)....................................................................................................|
   * detach_node (detach a node)
   * watch_mq (watch state of mq)
        #1
            * 'storage_0@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,552}, {leo_per_object_queue,152}]
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,73}, {leo_rebalance_queue,374}]
            * 'storage_2@127.0.0.1':[N/A]
        #2
            * 'storage_0@127.0.0.1':[{leo_sync_by_vnode_id_queue,15}, {leo_rebalance_queue,418}, {leo_per_object_queue,808}]
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,61}, {leo_rebalance_queue,711}, {leo_per_object_queue,206}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,90}]
        #3
            * 'storage_0@127.0.0.1':[{leo_sync_by_vnode_id_queue,5}, {leo_rebalance_queue,313}, {leo_per_object_queue,993}]
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,33}, {leo_rebalance_queue,443}, {leo_per_object_queue,652}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,65}, {leo_rebalance_queue,524}, {leo_per_object_queue,182}]
        #4
            * 'storage_0@127.0.0.1':[{leo_rebalance_queue,67}, {leo_per_object_queue,672}]
            * 'storage_1@127.0.0.1':[{leo_rebalance_queue,3}, {leo_per_object_queue,1277}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,929}, {leo_per_object_queue,584}]
        #5
            * 'storage_0@127.0.0.1':[{leo_rebalance_queue,26}, {leo_per_object_queue,49}]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,11}, {leo_rebalance_queue,32}, {leo_per_object_queue,1354}]
        #6
            * 'storage_0@127.0.0.1':[{leo_per_object_queue,49}]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,279}]
        #7
            * 'storage_0@127.0.0.1':[{leo_per_object_queue,41}]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
        #8
            * 'storage_0@127.0.0.1':[{leo_per_object_queue,5}]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
        #9
            * 'storage_0@127.0.0.1':[N/A]
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
   * get_objects (get objects)....................................................................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
 [SCENARIO-3]
   * put_objects (put objects)....................................................................................................|
   * takeover (takeover)
   * watch_mq (watch state of mq)
        #1
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,175}, {leo_rebalance_queue,71}, {leo_per_object_queue,1359}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,153}, {leo_per_object_queue,2629}]
            * 'storage_4@127.0.0.1':[N/A]
        #2
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,146}, {leo_rebalance_queue,438}, {leo_per_object_queue,1280}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,132}, {leo_rebalance_queue,294}, {leo_per_object_queue,2192}]
            * 'storage_4@127.0.0.1':[N/A]
        #3
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,103}, {leo_rebalance_queue,786}, {leo_per_object_queue,1345}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,83}, {leo_rebalance_queue,722}, {leo_per_object_queue,2174}]
            * 'storage_4@127.0.0.1':[N/A]
        #4
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,68}, {leo_rebalance_queue,1154}, {leo_per_object_queue,1351}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,77}, {leo_rebalance_queue,1060}, {leo_per_object_queue,2140}]
            * 'storage_4@127.0.0.1':[N/A]
        #5
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,59}, {leo_rebalance_queue,1217}, {leo_per_object_queue,1249}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,1217}, {leo_per_object_queue,1930}]
            * 'storage_4@127.0.0.1':[N/A]
        #6
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,17}, {leo_rebalance_queue,1315}, {leo_per_object_queue,1238}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,8}, {leo_rebalance_queue,985}, {leo_per_object_queue,1906}]
            * 'storage_4@127.0.0.1':[N/A]
        #7
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,6}, {leo_rebalance_queue,163}, {leo_per_object_queue,1426}]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,383}]
            * 'storage_4@127.0.0.1':[N/A]
        #8
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
   * get_objects (get objects)....................................................................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
 [SCENARIO-4]
   * put_objects (put objects)....................................................................................................|
   * suspend_node (suspend a node)
   * stop_node (stop a node)
   * get_objects (get objects)....................................................................................................|
   * start_node (start a node)
   * resume_node (resume a node)
   * put_objects (put objects)....................................................................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
   * get_objects (get objects)....................................................................................................|
 [SCENARIO-5]
   * put_objects (put objects)....................................................................................................|
   * attach_node (attach a node)
   * watch_mq (watch state of mq)
        #1
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,77}, {leo_rebalance_queue,500}, {leo_per_object_queue,219}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,83}, {leo_rebalance_queue,262}, {leo_per_object_queue,68}]
        #2
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,118}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,43}, {leo_rebalance_queue,337}, {leo_per_object_queue,786}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,38}, {leo_rebalance_queue,214}, {leo_per_object_queue,527}]
        #3
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,111}, {leo_rebalance_queue,166}, {leo_per_object_queue,19}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,38}, {leo_rebalance_queue,129}, {leo_per_object_queue,814}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,25}, {leo_rebalance_queue,285}, {leo_per_object_queue,487}]
        #4
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,111}, {leo_rebalance_queue,352}, {leo_per_object_queue,21}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,29}, {leo_rebalance_queue,133}, {leo_per_object_queue,482}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,16}, {leo_rebalance_queue,347}, {leo_per_object_queue,434}]
        #5
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,92}, {leo_rebalance_queue,396}, {leo_per_object_queue,96}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,23}, {leo_rebalance_queue,126}, {leo_per_object_queue,215}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_sync_by_vnode_id_queue,12}, {leo_rebalance_queue,129}, {leo_per_object_queue,468}]
        #6
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,92}, {leo_rebalance_queue,507}, {leo_per_object_queue,184}]
            * 'storage_2@127.0.0.1':[{leo_sync_by_vnode_id_queue,11}, {leo_rebalance_queue,61}, {leo_per_object_queue,122}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_rebalance_queue,28}, {leo_per_object_queue,81}]
        #7
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,58}, {leo_rebalance_queue,726}, {leo_per_object_queue,368}]
            * 'storage_2@127.0.0.1':[{leo_rebalance_queue,61}, {leo_per_object_queue,77}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_rebalance_queue,15}]
        #8
            * 'storage_1@127.0.0.1':[{leo_sync_by_vnode_id_queue,19}, {leo_rebalance_queue,706}, {leo_per_object_queue,561}]
            * 'storage_2@127.0.0.1':[{leo_rebalance_queue,61}, {leo_per_object_queue,74}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_rebalance_queue,4}]
        #9
            * 'storage_1@127.0.0.1':[{leo_per_object_queue,616}]
            * 'storage_2@127.0.0.1':[{leo_rebalance_queue,49}, {leo_per_object_queue,40}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_per_object_queue,18}]
        #10
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,27}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #11
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,27}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #12
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
   * get_objects (get objects)....................................................................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
   * compaction (execute data-compaction).......................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
   * get_objects (get objects)....................................................................................................|
 [SCENARIO-6]
   * put_objects (put objects)....................................................................................................|
   * remove_avs (remove avs of a node)
   * recover_node (recover data of a node)
   * watch_mq (watch state of mq)
        #1
            * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_recovery_node_queue,1}]
            * 'storage_4@127.0.0.1':[{leo_per_object_queue,730}]
        #2
            * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}, {leo_per_object_queue,129}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_recovery_node_queue,1}, {leo_per_object_queue,1424}]
            * 'storage_4@127.0.0.1':[{leo_per_object_queue,609}]
        #3
            * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}, {leo_per_object_queue,954}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_per_object_queue,572}]
            * 'storage_4@127.0.0.1':[N/A]
        #4
            * 'storage_1@127.0.0.1':[{leo_per_object_queue,550}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_per_object_queue,424}]
            * 'storage_4@127.0.0.1':[N/A]
        #5
            * 'storage_1@127.0.0.1':[{leo_per_object_queue,232}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #6
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
   * get_objects (get objects)....................................................................................................|
   * check_redundancies (check redundancies of replicas)....................................................................................................|
 [SCENARIO-7]
   * delete_bucket (delete a bucket)
   * watch_mq (watch state of mq)
        #1
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,4633}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,5020}]
            * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,4743}]
            * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,4617}]
        #2
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,3917}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,4354}]
            * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,4220}]
            * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,3816}]
        #3
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,2892}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,3215}]
            * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,3144}]
            * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,2916}]
        #4
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,2424}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,2466}]
            * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,2407}]
            * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,2151}]
        #5
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,1395}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,1545}]
            * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,1514}]
            * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,1247}]
        #6
            * 'storage_1@127.0.0.1':[{leo_delete_dir_queue_1,682}]
            * 'storage_2@127.0.0.1':[{leo_delete_dir_queue_1,873}]
            * 'storage_3@127.0.0.1':[{leo_delete_dir_queue_1,677}]
            * 'storage_4@127.0.0.1':[{leo_delete_dir_queue_1,324}]
        #7
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
   * create_bucket (create a bucket)
   * get_objects_not_found (get objects_not_found)....................................................................................................|
   * put_inconsistent_objects (put inconsistent objects)....................................................................................................|
   * scrub_cluster (scrub the whole cluster)
        #1
            * 'storage_1@127.0.0.1':[{leo_recovery_node_queue,1}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #2
            * 'storage_1@127.0.0.1':[{leo_per_object_queue,4527}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #3
            * 'storage_1@127.0.0.1':[{leo_per_object_queue,2921}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #4
            * 'storage_1@127.0.0.1':[{leo_per_object_queue,1649}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #5
            * 'storage_1@127.0.0.1':[{leo_per_object_queue,174}]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #6
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #1
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,4879}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #2
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,4443}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #3
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,2370}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #4
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[{leo_per_object_queue,57}]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #5
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #1
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_per_object_queue,3948}]
            * 'storage_4@127.0.0.1':[N/A]
        #2
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_per_object_queue,3128}]
            * 'storage_4@127.0.0.1':[N/A]
        #3
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[{leo_per_object_queue,503}]
            * 'storage_4@127.0.0.1':[N/A]
        #4
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
        #1
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_recovery_node_queue,1}]
        #2
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_recovery_node_queue,1}]
        #3
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_per_object_queue,3745}]
        #4
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_per_object_queue,3318}]
        #5
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_per_object_queue,1416}]
        #6
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[{leo_per_object_queue,211}]
        #7
            * 'storage_1@127.0.0.1':[N/A]
            * 'storage_2@127.0.0.1':[N/A]
            * 'storage_3@127.0.0.1':[N/A]
            * 'storage_4@127.0.0.1':[N/A]
   * check_redundancies (check redundancies of replicas)....................................................................................................|
::: Finished (1548sec) :::
```
comment