Repository: weaverba137/hpsspy

Issue:
Support top-level directories that contain archival files. 
Currently, data to be backed up must reside in a subdirectory of a top-level directory, but there are real-world examples where that is not the case.

Discussion:

Patch:
diff --git a/doc/changes.rst b/doc/changes.rst
index 9ac41fc..5d39237 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -7,9 +7,14 @@ Release Notes
 
 *This release drops support for Python 2.*
 
-* Remove all Python 2 code. (PR `#8`_).
+* Remove all Python 2 code (PR `#8`_).
+* Support fine-grained exclusion in configuration files (PR `#10`_).
+* Avoid commonly-used names for metadata in configuration files (PR `#10`_).
+* Detect newer files on disk that map to older HPSS files (PR `#10`_).
+* Allow top-level directories to contain only files (PR `#10`_).
 
 .. _`#8`: https://github.com/weaverba137/hpsspy/pull/8
+.. _`#10`: https://github.com/weaverba137/hpsspy/pull/10
 
 0.4.2 (2019-01-29)
 ------------------
diff --git a/doc/configuration.rst b/doc/configuration.rst
index 058ec2f..6ee4362 100644
--- a/doc/configuration.rst
+++ b/doc/configuration.rst
@@ -30,12 +30,12 @@ described below.
 Metadata
 ++++++++
 
-The configuration file should contain a top-level keyword ``"config"``.
+The configuration file should contain a top-level keyword ``"__config__"``.
 The value should itself be a :class:`dict`, containing some important
 metadata::
 
     {
-        "config": {
+        "__config__": {
             "root": "/global/project/projectdirs/my_project",
             "hpss_root": "/nersc/projects/my_project",
             "physical_disks": ["my_project"]
@@ -65,17 +65,17 @@ Sections
 
 Inside the root directory, as described above, there may be several top-level
 directories.  For the purposes of this documentation, these are called
-"sections" or "releases".  The terms are interchangable.  Each section
+"sections" or "releases".  The terms are interchangeable.  Each section
 has configuration items that describe its structure::
 
     {
-        "config": {
+        "__config__": {
             "root": "/projects/my_project",
             "hpss_root": "/hpss/projects/my_project",
             "physical_disks": ["my_project"]
         },
         "data": {
-            "exclude": [],
+            "__exclude__": [],
             "d1": {
                 "d1/batch/.*$": "d1/batch.tar",
                 "d1/([^/]+\\.txt)$": "d1/\\1",
@@ -89,14 +89,20 @@ The name of the section is passed on the command-line::
 
     missing_from_hpss config.json data
 
-This would read the data section above.
+This would read the ``"data"`` section above.
 
-Each section should have an ``"exclude"`` keyword, whose value is a list
+Each section should have an ``"__exclude__"`` keyword, whose value is a list
 of files to be ignored.  In the example above, in order to ignore the file
-``/projects/my_project/data/d1/README.html``, the ``"exclude"`` value
+``/projects/my_project/data/d1/README.html``, the ``"__exclude__"`` value
 would be ``["d1/README.html"]``.  Note that this is relative to the
 path ``/projects/my_project/data``, since ``"data"`` is the section being
-processed.
+processed.  Generally, this should only be used for a handful of top-level
+files, like README files.  For more precise exclusion, see the ``"EXCLUDE"``
+statement below.
+
+In the special case where a section contains only files, and no
+subdirectories, the special pseudo-subdirectory ``"__top__"`` can be
+used to contain the configuration.
 
 Mapping File Names to HPSS Archives
 +++++++++++++++++++++++++++++++++++
@@ -129,6 +135,12 @@ In coding terms we describe a portion of a directory tree hierarchy
 using regular expressions to match *files* in that portion.  Then we map
 files that match that regular expression to tape archive files.
 
+Finally, it should be noted that the configuration of each section is
+organized by subdirectory in order to speed up the process of mapping files
+to backup files.  Instead of looking through every possible configuration
+of files, only the configurations in a subdirectory need to be considered
+when examining files in that subdirectory.
+
 Regular Expression Details
 ++++++++++++++++++++++++++
 
@@ -153,6 +165,15 @@ imposes some additional requirements, conventions and idioms:
     and that command will be used to construct it.
   - Any archive file *not* ending in ``.tar`` will simply be copied to
     HPSS as is.
+  - The special string ``"EXCLUDE"`` can be used to prevent backups of
+    parts of a directory tree that might otherwise be archival. For example,
+    ``"d1/data/preproc/.*$" : "EXCLUDE"`` would prevent the ``preproc``
+    directory from being backed up, even if other parts of ``d1/data``
+    were configured for backup.
+  - The special string ``"AUTOMATED"`` behaves the same way as ``"EXCLUDE"``,
+    but is a human-readable way to denote data sets that are backed up by
+    automation independently of :command:`missing_from_hpss`, as opposed
+    to not being backed up at all.
   - When constructing an archive file, :command:`missing_from_hpss` will
     obtain the directory it needs to archive from the name of the *archive*
     file, not the regular expression itself.  This is because regular
diff --git a/hpsspy/data/desi.json b/hpsspy/data/desi.json
index dd4c294..3f38fba 100644
--- a/hpsspy/data/desi.json
+++ b/hpsspy/data/desi.json
@@ -1,11 +1,11 @@
 {
-    "config":{
+    "__config__":{
         "root":"/global/project/projectdirs/desi",
         "hpss_root":"/nersc/projects/desi",
         "physical_disks":["desi"]
     },
     "datachallenge":{
-        "exclude":[],
+        "__exclude__":[],
         "dc2":{
             "dc2/batch/.*$":"dc2/batch.tar",
             "dc2/([^/]+\\.txt)$":"dc2/\\1",
@@ -13,16 +13,16 @@
         }
     },
     "imaging":{
-        "exclude":[]
+        "__exclude__":[]
     },
     "mocks":{
-        "exclude":[]
+        "__exclude__":[]
     },
     "release":{
-        "exclude":[]
+        "__exclude__":[]
     },
     "spectro":{
-        "exclude":[],
+        "__exclude__":[],
         "data":{
         },
         "redux":{
@@ -34,6 +34,6 @@
         }
     },
     "target":{
-        "exclude":[]
+        "__exclude__":[]
     }
 }
diff --git a/hpsspy/data/sdss.json b/hpsspy/data/sdss.json
index c51d6b7..ecab777 100644
--- a/hpsspy/data/sdss.json
+++ b/hpsspy/data/sdss.json
@@ -1,11 +1,11 @@
 {
-    "config":{
+    "__config__":{
         "root":"/clusterfs/riemann/raid006",
         "hpss_root":"/nersc/projects/boss",
         "physical_disks":["raid006","raid000","raid005","raid007","raid008","raid2008","netapp"]
     },
     "dr8":{
-        "exclude":[
+        "__exclude__":[
             "env/index.html",
             "boss/photoObj/frames/301/index.html",
             "sdss/spectro/data/README.html"
@@ -61,7 +61,7 @@
         }
     },
     "dr9":{
-        "exclude":[
+        "__exclude__":[
             "env/index.html"
         ],
         "casload":{
@@ -107,7 +107,7 @@
         }
     },
     "dr10":{
-        "exclude":[
+        "__exclude__":[
             "env/index.html"
         ],
         "casload":{
@@ -151,7 +151,7 @@
         }
     },
     "dr11":{
-        "exclude":[
+        "__exclude__":[
             "env/index.html",
             "boss/lss/index.html"
         ],
@@ -196,7 +196,7 @@
         }
     },
     "dr12":{
-        "exclude":[
+        "__exclude__":[
             "env/index.html",
             "boss/lss/index.html"
         ],
diff --git a/hpsspy/scan.py b/hpsspy/scan.py
index 5229b0d..5615d33 100644
--- a/hpsspy/scan.py
+++ b/hpsspy/scan.py
@@ -43,20 +43,20 @@ def validate_configuration(config):
     except IOError:
         logger.critical("%s does not exist. Try again.", config)
         return 1
-    if 'config' in json_data:
+    if '__config__' in json_data:
         for k in ('root', 'hpss_root', 'physical_disks'):
-            if k not in json_data['config']:
-                logger.warning("%s 'config' section does not contain an " +
+            if k not in json_data['__config__']:
+                logger.warning("%s '__config__' section does not contain an " +
                                "entry for '%s'.", config, k)
     else:
-        logger.critical("%s does not contain a 'config' section.", config)
+        logger.critical("%s does not contain a '__config__' section.", config)
         return 1
     for k in json_data:
-        if k == 'config':
+        if k == '__config__':
             continue
-        if 'exclude' not in json_data[k]:
+        if '__exclude__' not in json_data[k]:
             logger.warning("Section '%s' should at least have an " +
-                           "'exclude' entry.", k)
+                           "'__exclude__' entry.", k)
         try:
             new_map = compile_map(json_data, k)
         except re.error:
@@ -66,15 +66,16 @@ def validate_configuration(config):
     return 0
 
 
-def compile_map(old_map, release):
+def compile_map(old_map, section):
     """Compile the regular expressions in a map.
 
     Parameters
     ----------
     old_map : :class:`dict`
         A dictionary containing regular expressions to compile.
-    release : :class:`str`
+    section : :class:`str`
         An initial key to determine the section of the dictionary of interest.
+        Typically, this will be a top-level directory.
 
     Returns
     -------
@@ -82,26 +83,27 @@ def compile_map(old_map, release):
         A new dictionary containing compiled regular expressions.
     """
     new_map = dict()
-    for key in old_map[release]:
-        if key == 'exclude':
-            new_map[key] = frozenset(old_map[release][key])
+    for key in old_map[section]:
+        if key == '__exclude__':
+            new_map[key] = frozenset(old_map[section][key])
         else:
             foo = list()
-            for r in old_map[release][key]:
-                foo.append((re.compile(r), old_map[release][key][r]))
+            for r in old_map[section][key]:
+                foo.append((re.compile(r), old_map[section][key][r]))
             new_map[key] = tuple(foo)
     return new_map
 
 
-def files_to_hpss(hpss_map_cache, release):
+def files_to_hpss(hpss_map_cache, section):
     """Create a map of files on disk to HPSS files.
 
     Parameters
     ----------
     hpss_map_cache : :class:`str`
         Data file containing the map.
-    release : :class:`str`
-        Release name.
+    section : :class:`str`
+        An initial key to determine the section of the dictionary of interest.
+        Typically, this will be a top-level directory.
 
     Returns
     -------
@@ -123,19 +125,20 @@ def files_to_hpss(hpss_map_cache, release):
             t.close()
         else:
             logger.warning("Returning empty map file!")
-            hpss_map = {"config": {},
-                        "dr8": {"exclude": [], "casload": {}, "apogee": {},
-                                "boss": {}, "sdss": {}},
-                        "dr9": {"exclude": [], "casload": {}, "apogee": {},
-                                "boss": {}, "sdss": {}},
-                        "dr10": {"exclude": [], "casload": {}, "apogee": {},
-                                 "boss": {}, "sdss": {}},
-                        "dr11": {"exclude": [], "casload": {}, "apogee": {},
-                                 "boss": {}, "marvels": {}, "sdss": {}},
-                        "dr12": {"exclude": [], "casload": {}, "apo": {},
+            hpss_map = {"__config__": {},
+                        "dr8": {"__exclude__": [], "casload": {},
+                                "apogee": {}, "boss": {}, "sdss": {}},
+                        "dr9": {"__exclude__": [], "casload": {},
+                                "apogee": {}, "boss": {}, "sdss": {}},
+                        "dr10": {"__exclude__": [], "casload": {},
+                                 "apogee": {}, "boss": {}, "sdss": {}},
+                        "dr11": {"__exclude__": [], "casload": {},
+                                 "apogee": {}, "boss": {}, "marvels": {},
+                                 "sdss": {}},
+                        "dr12": {"__exclude__": [], "casload": {}, "apo": {},
                                  "apogee": {}, "boss": {}, "marvels": {},
                                  "sdss": {}}}
-    return (compile_map(hpss_map, release), hpss_map['config'])
+    return (compile_map(hpss_map, section), hpss_map['__config__'])
 
 
 def find_missing(hpss_map, hpss_files, disk_files_cache, missing_files,
@@ -146,7 +149,7 @@ def find_missing(hpss_map, hpss_files, disk_files_cache, missing_files,
     ----------
     hpss_map : :class:`dict`
         A mapping of file names to HPSS files.
-    hpss_files : :class:`frozenset`
+    hpss_files : :class:`dict`
         The list of actual HPSS files.
     disk_files_cache : :class:`str`
         Name of the disk cache file.
@@ -166,18 +169,23 @@ def find_missing(hpss_map, hpss_files, disk_files_cache, missing_files,
     nfiles = 0
     nmissing = 0
     nmultiple = 0
-    missing = dict()
+    backups = dict()
     pattern_used = dict()
     section_warning = set()
-    with open(disk_files_cache) as t:
+    with open(disk_files_cache, newline='') as t:
         reader = csv.DictReader(t)
         for row in reader:
             f = row['Name']
             nfiles += 1
-            if f in hpss_map["exclude"]:
+            if f in hpss_map["__exclude__"]:
                 logger.info("%s is excluded.", f)
                 continue
             section = f.split('/')[0]
+            if section == f:
+                #
+                # Top-level section containing no subdirectories.
+                #
+                section = '__top__'
             try:
                 s = hpss_map[section]
             except KeyError:
@@ -209,16 +217,38 @@ def find_missing(hpss_map, hpss_files, disk_files_cache, missing_files,
                 m = r[0].match(f)
                 if m is not None:
                     pattern_used[r[0].pattern] += 1
-                    reName = r[0].sub(r[1], f)
-                    if reName not in hpss_files:
-                        if reName in missing:
-                            missing[reName]['files'].append(f)
-                            missing[reName]['size'] += int(row['Size'])
-                        else:
-                            missing[reName] = {'files': [f],
-                                               'size': int(row['Size'])}
-                    logger.debug("%s in %s.", f, reName)
                     mapped += 1
+                    if r[1] == "EXCLUDE":
+                        logger.debug("%s is excluded from backups.", f)
+                    elif r[1] == "AUTOMATED":
+                        logger.debug("%s is backed up by some other " +
+                                     "automated process.", f)
+                    else:
+                        reName = r[0].sub(r[1], f)
+                        logger.debug("%s in %s.", f, reName)
+                        exists = reName in hpss_files
+                        if exists:
+                            newer = int(row['Mtime']) > hpss_files[reName][1]
+                        else:
+                            newer = False
+                        if newer:
+                            logger.warning("%s is newer than %s, " +
+                                           "marking as missing!",
+                                           f, reName)
+                        if reName in backups:
+                            backups[reName]['files'].append(f)
+                            backups[reName]['size'] += int(row['Size'])
+                            #
+                            # 'newer' can change from False to True, but
+                            # it should never change back to False.
+                            #
+                            if newer:
+                                backups[reName]['newer'] = newer
+                        else:
+                            backups[reName] = {'files': [f],
+                                               'size': int(row['Size']),
+                                               'newer': newer,
+                                               'exists': exists}
             if mapped == 0:
                 logger.error("%s is not mapped to any file on HPSS!", f)
                 nmissing += 1
@@ -227,6 +257,16 @@ def find_missing(hpss_map, hpss_files, disk_files_cache, missing_files,
                 nmultiple += 1
             if (nfiles % report) == 0:
                 logger.info("%9d files scanned.", nfiles)
+    #
+    # Eliminate backups that exist and have no newer files on disk.
+    #
+    missing = dict()
+    for k, v in backups.items():
+        if v['exists'] and not v['newer']:
+            logger.debug("%s is a valid backup.", k)
+        else:
+            logger.debug("Adding %s to missing backups.", k)
+            missing[k] = v
     with open(missing_files, 'w') as fp:
         json.dump(missing, fp, indent=2, separators=(',', ': '))
     if nmissing > 0:
@@ -445,8 +485,9 @@ def scan_disk(disk_roots, disk_files_cache, overwrite=False):
         return True
     else:
         logger.info("No disk cache file, starting scan.")
-        with open(disk_files_cache, 'w') as t:
-            t.write('Name,Size\n')
+        with open(disk_files_cache, 'w', newline='') as t:
+            writer = csv.writer(t)
+            writer.writerow(['Name', 'Size', 'Mtime'])
             try:
                 for disk_root in disk_roots:
                     logger.debug("Starting os.walk at %s.", disk_root)
@@ -456,11 +497,14 @@ def scan_disk(disk_roots, disk_files_cache, overwrite=False):
                             fullname = os.path.join(root, f)
                             if not os.path.islink(fullname):
                                 cachename = fullname.replace(disk_root+'/', '')
-                                size = os.stat(fullname).st_size
-                                t.write("{0},{1:d}\n".format(cachename, size))
-            except OSError:
+                                s = os.stat(fullname)
+                                writer.writerow([cachename,
+                                                 s.st_size,
+                                                 int(s.st_mtime)])
+            except OSError as e:
                 logger.error('Exception encountered while creating ' +
                              'disk cache file!')
+                logger.error(e.strerror)
                 return False
     return True
 
@@ -479,33 +523,30 @@ def scan_hpss(hpss_root, hpss_files_cache, overwrite=False):
 
     Returns
     -------
-    :class:`frozenset`
-        The set of files found on HPSS.
+    :class:`dict`
+        The set of files found on HPSS, with size and modification time.
     """
     from .os import walk
     logger = logging.getLogger(__name__ + '.scan_hpss')
+    hpss_files = dict()
     if os.path.exists(hpss_files_cache) and not overwrite:
         logger.info("Found cache file %s.", hpss_files_cache)
-        with open(hpss_files_cache) as t:
-            hpss_files = [l.strip() for l in t.readlines()]
+        with open(hpss_files_cache, newline='') as t:
+            reader = csv.DictReader(t)
+            for row in reader:
+                hpss_files[row['Name']] = (int(row['Size']), int(row['Mtime']))
     else:
         logger.info("No HPSS cache file, starting scan at %s.", hpss_root)
-        hpss_files = list()
-        for root, dirs, files in walk(hpss_root):
-            # hpss_files += [f.path.replace(hpss_root+'/','')
-            #                for f in files if not f.ishtar]
-            logger.debug("Scanning HPSS directory %s.", root)
-            hpss_files += [f.path.replace(hpss_root+'/', '')
-                           for f in files if not f.path.endswith('.idx')]
-            # htar_files = [f for f in files if f.ishtar]
-            # for h in htar_files:
-            #     contents = h.htar_contents()
-            #     hpss_files += [join(root,c[9]).replace(hpss_root+'/','')
-            #                    for c in contents if c[0] == '-']
-            # links += [f for f in files if f.islink]
-        with open(hpss_files_cache, 'w') as t:
-            t.write('\n'.join(hpss_files)+'\n')
-    hpss_files = frozenset(hpss_files)
+        with open(hpss_files_cache, 'w', newline='') as t:
+            w = csv.writer(t)
+            w.writerow(['Name', 'Size', 'Mtime'])
+            for root, dirs, files in walk(hpss_root):
+                logger.debug("Scanning HPSS directory %s.", root)
+                for f in files:
+                    if not f.path.endswith('.idx'):
+                        ff = f.path.replace(hpss_root+'/', '')
+                        hpss_files[ff] = (f.st_size, f.st_mtime)
+                        w.writerow([ff, f.st_size, f.st_mtime])
     return hpss_files
 
 
@@ -617,12 +658,12 @@ def main():
     #
     if options.test:
         logger.info("Test mode. Pretending no files exist on HPSS.")
-        hpss_files = frozenset([])
+        hpss_files = dict()
     else:
         logger.debug("Cache files will be written to %s.", options.cache)
         hpss_files_cache = os.path.join(options.cache,
                                         ('hpss_files_' +
-                                         '{0}.txt').format(options.release))
+                                         '{0}.csv').format(options.release))
         logger.debug("hpss_files_cache = '%s'", hpss_files_cache)
         hpss_files = scan_hpss(hpss_release_root, hpss_files_cache,
                                overwrite=options.overwrite_hpss)
diff --git a/hpsspy/util.py b/hpsspy/util.py
index 4341502..a5f265d 100644
--- a/hpsspy/util.py
+++ b/hpsspy/util.py
@@ -139,9 +139,7 @@ class HpssFile(object):
     def st_mode(self):
         """File permission mode.
         """
-        if 'st_mode' in self._property_cache:
-            return self._property_cache['st_mode']
-        else:
+        if 'st_mode' not in self._property_cache:
             try:
                 mode = self._file_modes[self.raw_type]
             except KeyError:
@@ -178,15 +176,13 @@ class HpssFile(object):
             if self.raw_permission[8] == 't':
                 mode |= (stat.S_IXOTH | stat.S_ISVTX)
             self._property_cache['st_mode'] = mode
-            return mode
+        return self._property_cache['st_mode']
 
     @property
     def st_mtime(self):
         """File modification time.
         """
-        if 'st_mtime' in self._property_cache:
-            return self._property_cache['st_mtime']
-        else:
+        if 'st_mtime' not in self._property_cache:
             seconds = 0
             month = self._months.index(self.raw_month) + 1
             if self.raw_year.find(':') > 0:
@@ -201,7 +197,7 @@ class HpssFile(object):
             mtime = int(datetime(year, month, self.raw_day,
                                  hours, minutes, seconds).strftime('%s'))
             self._property_cache['st_mtime'] = mtime
-            return mtime
+        return self._property_cache['st_mtime']
 
     def htar_contents(self):
         """Return (and cache) the contents of an htar file.


Test patch:
diff --git a/hpsspy/test/t/test_scan.json b/hpsspy/test/t/test_scan.json
index b7562a8..e6ab245 100644
--- a/hpsspy/test/t/test_scan.json
+++ b/hpsspy/test/t/test_scan.json
@@ -1,11 +1,11 @@
 {
-    "config":{
+    "__config__":{
         "root":"/temporary",
         "hpss_root":"/nersc/projects/temporary",
         "physical_disks":["temporary"]
     },
     "data":{
-        "exclude":["README.html"],
+        "__exclude__":["README.html"],
         "d1":{
             "d1/batch/.*$":"d1/batch.tar",
             "d1/([^/]+\\.txt)$":"d1/\\1",
@@ -14,6 +14,7 @@
         "d2":{
             "d2/[^/]+$":"d2/d2_files.tar",
             "d2/(batch|fiberassign)/.*$":"d2/d2_\\1.tar",
+            "d2/spectro/redux/preproc/.*$":"EXCLUDE",
             "d2/spectro/redux/([0-9a-zA-Z_-]+)/[^/]+$":"d2/spectro/redux/\\1/\\1_files.tar",
             "d2/spectro/redux/([0-9a-zA-Z_-]+)/(calib2d|plan|run)/.*$":"d2/spectro/redux/\\1/\\1_\\2.tar",
             "d2/spectro/redux/([0-9a-zA-Z_-]+)/exposures/([0-9]{8})/.*$":"d2/spectro/redux/\\1/exposures/\\1_exposures_\\2.tar",
@@ -24,9 +25,9 @@
         }
     },
     "redux":{
-        "exclude":[]
+        "__exclude__":[]
     },
     "www":{
-        "exclude":[]
+        "__exclude__":[]
     }
 }
diff --git a/hpsspy/test/test_os.py b/hpsspy/test/test_os.py
index 03c0650..29fb7a0 100644
--- a/hpsspy/test/test_os.py
+++ b/hpsspy/test/test_os.py
@@ -216,7 +216,7 @@ class TestOs(unittest.TestCase):
                               'drwxrwxr-x    1 bweaver   bweaver           ' +
                               '21 Aug 22  2014 cosmo')]
             self.assertTrue(islink('cosmo'))
-            h.assert_has_calls([call('ls', '-ld', 'cosmo'),])
+            h.assert_has_calls([call('ls', '-ld', 'cosmo')])
 
     def test_walk(self):
         """Test the walk() function.
diff --git a/hpsspy/test/test_scan.py b/hpsspy/test/test_scan.py
index 2886725..82496ec 100644
--- a/hpsspy/test/test_scan.py
+++ b/hpsspy/test/test_scan.py
@@ -92,9 +92,9 @@ class TestScan(unittest.TestCase):
         """Test compiling regular expressions in the JSON configuration file.
         """
         new_map = compile_map(self.config, 'data')
-        self.assertEqual(new_map['exclude'], frozenset(['README.html']))
+        self.assertEqual(new_map['__exclude__'], frozenset(['README.html']))
         for k in self.config['data']:
-            if k != 'exclude':
+            if k != '__exclude__':
                 for l in new_map[k]:
                     self.assertIn(l[0].pattern, self.config['data'][k])
                     self.assertEqual(l[1],
@@ -177,36 +177,36 @@ class TestScan(unittest.TestCase):
         self.assertEqual(status, 0)
         # Valid file but missing some pieces
         c = self.config.copy()
-        del c['config']
+        del c['__config__']
         fn, tmp = tempfile.mkstemp(suffix='.json')
         with open(tmp, 'w') as fd:
             json.dump(c, fd)
         status = validate_configuration(tmp)
         self.assertEqual(status, 1)
-        self.assertLog(-1,
-                       "{0} does not contain a 'config' section.".format(tmp))
+        self.assertLog(-1, ("{0} does not contain a " +
+                            "'__config__' section.").format(tmp))
         os.close(fn)
         os.remove(tmp)
         c = self.config.copy()
-        del c['config']['physical_disks']
+        del c['__config__']['physical_disks']
         fn, tmp = tempfile.mkstemp(suffix='.json')
         with open(tmp, 'w') as fd:
             json.dump(c, fd)
         status = validate_configuration(tmp)
         self.assertEqual(status, 0)
-        self.assertLog(-1, ("{0} 'config' section does not contain an " +
+        self.assertLog(-1, ("{0} '__config__' section does not contain an " +
                             "entry for '{1}'.").format(tmp, 'physical_disks'))
         os.close(fn)
         os.remove(tmp)
         c = self.config.copy()
-        del c['redux']['exclude']
+        del c['redux']['__exclude__']
         fn, tmp = tempfile.mkstemp(suffix='.json')
         with open(tmp, 'w') as fd:
             json.dump(c, fd)
         status = validate_configuration(tmp)
         self.assertEqual(status, 0)
         self.assertLog(-1, ("Section '{0}' should at least have an " +
-                            "'exclude' entry.").format('redux'))
+                            "'__exclude__' entry.").format('redux'))
         os.close(fn)
         os.remove(tmp)
         c = self.config.copy()