axkibe/lsyncd — Issue #75: Very high disk usage on init
Author: axkibe
Created: 2011-11-16 15:08:37+00:00
State: open

Originally filed by ro...@d3r.com on 2011-08-03T07:48:29

What steps will reproduce the problem?
1. Start lsyncd with several sync stanzas each watching a large file tree
2. Monitor load averages / disk usage

What is the expected output? What do you see instead?

Expected / Desirable : Lsyncd does the initial scan of directories in serial or with a limited number of threads

Actual Result : Lsyncd starts a thread for every sync stanza which results in very high disk usage when there are several
sync stanzas with large trees

What version of the product are you using? On what operating system?

2.0.4 on Debian Lenny compiled against lenny-backports packages for Lua

Please provide any additional information below.

We have several large asset directories (15G+) for web sites and we would like to use lsyncd to push the assets to a backup server in (almost) realtime. It works great for this. However when we restart lsyncd it starts a thread for each synz stanza to do its initial scan. This doesn't scale well as the more sync points there are, the heavier the
disk activity when lsycnd is restarted.

Could Lsyncd internally limit the number of initial sync threads to be kinder to the disks? Ideally there would be some configuration options to allow the administrator to specify how the initial sync is handled, for example max number of threads, nice value for initial rsync processes?

Alternatively would it be possible to write a layer 3/4 function to control this? A crude implementation might write a marker file to disk for directories that have had an initial sync and discard the init event if the marker exists?

Otherwise Lsyncd is an excellent piece of software. Thank you for all your hard work.