cfengine/core — pull request #1754 by kacfengine (2014-06-13 09:02:58+00:00)
Title: Redmine #2917: Rewrite file monitoring (files changes attribute).

The main commit is this one:

Redmine #2917: Rewrite file monitoring (files changes attribute).

The old implementation suffered from the fact that the database layout
was unsuited for the task. Each was indexed solely by filename,
however we want to do detection of file removals as well. Since, if
the file has vanished from the file system, we don't have the database
key anymore, it was not possible to find vanished files without
traversing the entire database.

While this was possible, you had to choose between two evils. Either
you traverse the entire database once at the end of the run, and
report missing files then, but this has the disadvantage that you
cannot tie the message to any particular promise, nor can you report
it correctly in Mission Portal. The other alternative is to check
after each file promise, but this has the disadvantage that each
promise pays the price of every other file promise combined, since we
have to traverse the entire database, and do a stat for every entry
(if you had one directory with 200000 files, and 200 directories with
just a few files, performance would be on the order of 200x200000
iterations, even if you did not run the promise that had 200000 files
in it).

The solution is to change the database format. The stat and checksum
information remains semantically the same (with a few minor tweaks to
the DB key), but we add a directory listing for each monitored
directory, so that we can look up the list without knowing which file
we are looking for, only the containing directory.

All the information is now also contained within one single database,
instead of the two that existed before. It has a new name and the old
databases will be migrated to the new name and format if they exist.

A lot of reorganizing was also done as part of this, with all "files
changes" related code gathered in one file: files_changes.c.

In addition there are some support commits (note that ce238c739 is the same as in #1748):

commit ce238c7395ad62b4768b5ba17ac15bc323629637
Author: Kristian Amlie
Date: Wed Jun 11 09:21:59 2014

Move several low level file functions to libutils.

This was they can be used from libcompat replacement functions.

commit 737146694ba0ff58607875a199419fa83354663b
Author: Kristian Amlie
Date: Thu Jun 12 14:23:25 2014

Add SeqSoftSort, SeqBinaryLookup and SeqBinaryIndexOf functions.

commit 3e298c5fa1e35b2a9f246d3264be9e15c2c9f357
Author: Kristian Amlie
Date: Thu Jun 12 14:25:50 2014

Move DeleteDirectoryTree and FILE_SEPARATOR to libutils.

They don't depend on anything in libpromises, and this makes them
available for use outside of libpromises as well.

Should be merged together with https://github.com/cfengine/enterprise/pull/195.