Repository: glotzerlab/signac (python)

Issue:
_SyncedList dictionary children should be _SyncedDicts
### Description

I was editing a job document. When getting a list that is a child of a `_SyncedDict`, you get a `_SyncedList`. However, when getting a dict that is a child of a `_SyncedList`, you don't get a `_SyncedDict` (just a `dict`).

### To reproduce

```python
import signac

project = signac.init_project('test')
project.doc['key'] = [{'a': 1}, {'b': 2}]
print(type(project.doc.key))
print(type(project.doc.key[0]))

project.doc.key[0]['a'] = 3
assert project.doc.key[0]['a'] == 3  # still 1
```

Pull request:
Improve Sync Data Structures
<!-- Provide a general summary of your changes in the Title above -->
## Description
<!-- Describe your changes in detail -->
This PR is related to #249 . In this PR, we are implementing `SyncedCollection`, `SyncedAttrDict`, `SyncedList`, `JSONCollection`, `JSONDict`, `JSONList`.

## Motivation and Context
<!-- Why is this change required? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here. -->
This refractor is to provide support for the multiple backends and resolve #196.
## Types of Changes
<!-- Please select all items that apply either now or after creating the pull request: -->
- [ ] Documentation update
- [ ] Bug fix
- [x] New feature
- [x] Breaking change<sup>1</sup>

<sup>1</sup>The change breaks (or has the potential to break) existing functionality.

## Checklist:
<!-- Please select all items that apply either now or after creating the pull request. -->
<!-- If you are unsure about any of these items, do not hesitate to ask! -->
- [x] I am familiar with the [**Contributing Guidelines**](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md).
- [x] I agree with the terms of the [**Contributor Agreement**](https://github.com/glotzerlab/signac/blob/master/ContributorAgreement.md).
- [x] My name is on the [list of contributors](https://github.com/glotzerlab/signac/blob/master/contributors.yaml).
- [x] My code follows the [code style guideline](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md#code-style) of this project.
- [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:
- [ ] I have updated the API documentation as part of the package doc-strings.
- [ ] I have created a separate pull request to update the [framework documentation](https://docs.signac.io/) on [signac-docs](https://github.com/glotzerlab/signac-docs) and linked it here.
- [ ] I have updated the [changelog](https://github.com/glotzerlab/signac/blob/master/changelog.txt) and added all related issue and pull request numbers for future reference (if applicable). See example below.


Example for a changelog entry: `Fix issue with launching rockets to the moon (#101, #212).`


Patch:
diff --git a/.github/dependabot.yml b/.github/dependabot.yml
new file mode 100644
index 00000000..4cc29be6
--- /dev/null
+++ b/.github/dependabot.yml
@@ -0,0 +1,9 @@
+version: 2
+updates:
+- package-ecosystem: pip
+  directory: "/"
+  schedule:
+    interval: monthly
+  open-pull-requests-limit: 10
+  reviewers:
+    - glotzerlab/signac-committers
diff --git a/requirements-benchmark.txt b/requirements-benchmark.txt
index 42d3a8f1..43672f4f 100644
--- a/requirements-benchmark.txt
+++ b/requirements-benchmark.txt
@@ -1,7 +1,7 @@
 click>=7.0
-numpy==1.18
-gitdb2==3.0.2
-GitPython==3.0.8
-pandas==0.25.3; implementation_name!='cpython' --no-binary pandas
-pandas==0.25.3; implementation_name=='cpython' --no-binary :none:
-psutil==5.6.7
+numpy==1.19.0
+gitdb2==4.0.2
+GitPython==3.1.3
+pandas==1.0.5; implementation_name!='cpython' --no-binary pandas
+pandas==1.0.5; implementation_name=='cpython' --no-binary :none:
+psutil==5.7.0
diff --git a/requirements-dev.txt b/requirements-dev.txt
index b157db7f..1a3af8ef 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -1,13 +1,13 @@
-coverage==5.0.3
-numpy==1.18
-pandas==0.25.3; implementation_name!='cpython' --no-binary pandas
-pandas==0.25.3; implementation_name=='cpython' --no-binary :none:
+coverage==5.2
+numpy==1.19.0
+pandas==1.0.5; implementation_name!='cpython' --no-binary pandas
+pandas==1.0.5; implementation_name=='cpython' --no-binary :none:
 h5py==2.10; implementation_name=='cpython'
 tables==3.6.1; implementation_name=='cpython'
 click>=7.0
 ruamel.yaml>=0.15.89
 pytest==5.3.4
-pytest-subtests==0.3.0
+pytest-subtests==0.3.1
 pydocstyle==5.0.2
-pytest-cov==2.8.1
+pytest-cov==2.10.0
 pymongo==3.10.1
diff --git a/setup.cfg b/setup.cfg
index 86e0a463..75d15899 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -15,7 +15,7 @@ max-line-length = 100
 exclude = configobj,passlib,cite.py,conf.py
 
 [pydocstyle]
-match = jsondict.py
+match = jsondict.py | synced_collection.py | jsoncollection.py | syncedattrdict.py | synced_list.py 
 ignore = D105, D107, D203, D213
 
 [mypy]
diff --git a/signac/core/jsoncollection.py b/signac/core/jsoncollection.py
new file mode 100644
index 00000000..0a7e0212
--- /dev/null
+++ b/signac/core/jsoncollection.py
@@ -0,0 +1,151 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implements JSON-backend.
+
+This implements the JSON-backend for SyncedCollection API by
+implementing sync and load methods.
+"""
+
+import os
+import json
+import errno
+import uuid
+
+from .synced_collection import SyncedCollection
+from .syncedattrdict import SyncedAttrDict
+from .synced_list import SyncedList
+
+
+class JSONCollection(SyncedCollection):
+    """Implement sync and load using a JSON back end."""
+
+    backend = __name__  # type: ignore
+
+    def __init__(self, filename=None, write_concern=False, **kwargs):
+        self._filename = os.path.realpath(filename) if filename is not None else None
+        self._write_concern = write_concern
+        super().__init__(**kwargs)
+        if (filename is None) == (self._parent is None):
+            raise ValueError(
+                "Illegal argument combination, one of the two arguments, "
+                "parent or filename must be None, but not both.")
+
+    def _load(self):
+        """Load the data from a JSON-file."""
+        try:
+            with open(self._filename, 'rb') as file:
+                blob = file.read()
+                return json.loads(blob)
+        except IOError as error:
+            if error.errno == errno.ENOENT:
+                return None
+
+    def _sync(self):
+        """Write the data to json file."""
+        data = self.to_base()
+        # Serialize data:
+        blob = json.dumps(data).encode()
+        # When write_concern flag is set, we write the data into dummy file and then
+        # replace that file with original file.
+        if self._write_concern:
+            dirname, filename = os.path.split(self._filename)
+            fn_tmp = os.path.join(dirname, '._{uid}_{fn}'.format(
+                uid=uuid.uuid4(), fn=filename))
+            with open(fn_tmp, 'wb') as tmpfile:
+                tmpfile.write(blob)
+            os.replace(fn_tmp, self._filename)
+        else:
+            with open(self._filename, 'wb') as file:
+                file.write(blob)
+
+
+class JSONDict(JSONCollection, SyncedAttrDict):
+    """A dict-like mapping interface to a persistent JSON file.
+
+    The JSONDict inherits from :class:`~core.collection_api.SyncedCollection`
+    and :class:`~core.syncedattrdict.SyncedAttrDict`.
+
+    .. code-block:: python
+
+        doc = JSONDict('data.json', write_concern=True)
+        doc['foo'] = "bar"
+        assert doc.foo == doc['foo'] == "bar"
+        assert 'foo' in doc
+        del doc['foo']
+
+    .. code-block:: python
+
+        >>> doc['foo'] = dict(bar=True)
+        >>> doc
+        {'foo': {'bar': True}}
+        >>> doc.foo.bar = False
+        {'foo': {'bar': False}}
+
+    .. warning::
+
+        While the JSONDict object behaves like a dictionary, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying file, copying (even deep
+        copying) a JSONDict instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new JSONDict
+        instance: `new_dict = JSONDict(old_dict.to_base())`.
+
+    Parameters
+    ----------
+    filename: str, optional
+        The filename of the associated JSON file on disk (Default value = None).
+    data: mapping, optional
+        The intial data pass to JSONDict. Defaults to `list()`
+    parent: object, optional
+        A parent instance of JSONDict or None (Default value = None).
+    write_concern: bool, optional
+        Ensure file consistency by writing changes back to a temporary file
+        first, before replacing the original file (Default value = None).
+    """
+
+    pass
+
+
+class JSONList(JSONCollection, SyncedList):
+    """A non-string sequence interface to a persistent JSON file.
+
+    The JSONDict inherits from :class:`~core.collection_api.SyncedCollection`
+    and :class:`~core.syncedlist.SyncedList`.
+
+    .. code-block:: python
+
+        synced_list = JSONList('data.json', write_concern=True)
+        synced_list.append("bar")
+        assert synced_list[0] == "bar"
+        assert len(synced_list) == 1
+        del synced_list[0]
+
+    .. warning::
+
+        While the JSONList object behaves like a list, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying file, copying (even deep
+        copying) a JSONList instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new JSONList
+        instance: `new_list = JSONList(old_list.to_base())`.
+
+    Parameters
+    ----------
+    filename: str
+        The filename of the associated JSON file on disk (Default value = None).
+    data: non-str Sequence
+        The intial data pass to JSONDict
+    parent: object
+        A parent instance of JSONDict or None (Default value = None).
+    write_concern: bool
+        Ensure file consistency by writing changes back to a temporary file
+        first, before replacing the original file (Default value = None).
+    """
+
+    pass
+
+
+SyncedCollection.register(JSONDict, JSONList)
diff --git a/signac/core/synced_collection.py b/signac/core/synced_collection.py
new file mode 100644
index 00000000..f275238a
--- /dev/null
+++ b/signac/core/synced_collection.py
@@ -0,0 +1,165 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implement the SyncedCollection class.
+
+SyncedCollection encapsulates the synchronization of different data-structures.
+These features are implemented in different subclasses which enable us to use a
+backend with different data-structures or vice-versa. It declares as abstract
+methods the methods that must be implemented by any subclass to match the API.
+"""
+
+from contextlib import contextmanager
+from abc import abstractmethod
+from collections import defaultdict
+from collections.abc import Collection
+
+try:
+    import numpy
+    NUMPY = True
+except ImportError:
+    NUMPY = False
+
+
+class SyncedCollection(Collection):
+    """The base synced collection represents a collection that is synced with a backend.
+
+    The class is intended for use as an ABC. The SyncedCollection is a
+    :class:`~collections.abc.Collection` where all data is stored persistently
+    in the underlying backend. The backend name wil be same as the module name.
+    """
+
+    backend = None
+
+    def __init__(self, parent=None):
+        self._data = None
+        self._parent = parent
+        self._suspend_sync_ = 0
+
+    @classmethod
+    def register(cls, *args):
+        """Register the synced data structures.
+
+        Registry is used when recursively converting synced data structures to determine
+        what to convert their children into.
+
+        Parameters
+        ----------
+        *args
+            Classes to register
+        """
+        if not hasattr(cls, 'registry'):
+            cls.registry = defaultdict(list)
+        for _cls in args:
+            cls.registry[_cls.backend].append(_cls)
+
+    @classmethod
+    def from_base(cls, data, backend=None, **kwargs):
+        """Dynamically resolve the type of object to the corresponding synced collection.
+
+        Parameters
+        ----------
+        data : any
+            Data to be converted from base class.
+        backend: str
+            Name of backend for synchronization. Default to backend of class.
+        **kwargs:
+            Kwargs passed to instance of synced collection.
+
+        Returns
+        -------
+        data : object
+            Synced object of corresponding base type.
+        """
+        backend = cls.backend if backend is None else backend
+        if backend is None:
+            raise ValueError("No backend found.")
+        for _cls in cls.registry[backend]:
+            if _cls.is_base_type(data):
+                return _cls(data=data, **kwargs)
+        if NUMPY:
+            if isinstance(data, numpy.number):
+                return data.item()
+        return data
+
+    @abstractmethod
+    def to_base(self):
+        """Dynamically resolve the synced collection to the corresponding base type."""
+        pass
+
+    @contextmanager
+    def _suspend_sync(self):
+        """Prepare context where load and sync are suspended."""
+        self._suspend_sync_ += 1
+        yield
+        self._suspend_sync_ -= 1
+
+    @classmethod
+    @abstractmethod
+    def is_base_type(cls, data):
+        """Check whether data is of the same base type (such as list or dict) as this class."""
+        pass
+
+    @abstractmethod
+    def _load(self):
+        """Load data from underlying backend."""
+        pass
+
+    @abstractmethod
+    def _sync(self):
+        """Write data to underlying backend."""
+        pass
+
+    def sync(self):
+        """Synchronize the data with the underlying backend."""
+        if self._suspend_sync_ <= 0:
+            if self._parent is None:
+                self._sync()
+            else:
+                self._parent.sync()
+
+    def load(self):
+        """Load the data from the underlying backend."""
+        if self._suspend_sync_ <= 0:
+            if self._parent is None:
+                data = self._load()
+                with self._suspend_sync():
+                    self._update(data)
+            else:
+                self._parent.load()
+
+    # The following methods share a common implementation for
+    # all data structures and regardless of backend.
+    def __getitem__(self, key):
+        self.load()
+        return self._data[key]
+
+    def __delitem__(self, item):
+        del self._data[item]
+        self.sync()
+
+    def __iter__(self):
+        self.load()
+        return iter(self._data)
+
+    def __len__(self):
+        self.load()
+        return len(self._data)
+
+    def __call__(self):
+        self.load()
+        return self.to_base()
+
+    def __eq__(self, other):
+        if isinstance(other, type(self)):
+            return self() == other()
+        else:
+            return self() == other
+
+    def __repr__(self):
+        self.load()
+        return repr(self._data)
+
+    def __str__(self):
+        self.load()
+        return str(self._data)
diff --git a/signac/core/synced_list.py b/signac/core/synced_list.py
new file mode 100644
index 00000000..771330ae
--- /dev/null
+++ b/signac/core/synced_list.py
@@ -0,0 +1,180 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implements the SyncedList class.
+
+This implements the list data-structure for SyncedCollection API by
+implementing the convert methods (`to_base`, `from_base`) for lists.
+"""
+
+from collections.abc import Sequence
+from collections.abc import MutableSequence
+
+from .synced_collection import SyncedCollection
+from .synced_collection import NUMPY
+
+if NUMPY:
+    import numpy
+
+
+class SyncedList(SyncedCollection, MutableSequence):
+    """Implementation of list data structure.
+
+    The SyncedList inherits from :class:`~core.collection_api.SyncedCollection`
+    and :class:`~collections.abc.MutableSequence`. Therefore, it behaves similar
+    to a :class:`list`.
+
+    .. warning::
+
+        While the SyncedList object behaves like a dictionary, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying backend, copying (even deep
+        copying) a SyncedList instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new SyncedList.
+    """
+
+    def __init__(self, data=None, **kwargs):
+        super().__init__(**kwargs)
+        if data is None:
+            self._data = []
+        else:
+            if NUMPY and isinstance(data, numpy.ndarray):
+                data = data.tolist()
+            with self._suspend_sync():
+                self._data = [self.from_base(data=value, parent=self) for value in data]
+            self.sync()
+
+    @classmethod
+    def is_base_type(cls, data):
+        """Check whether the data is an non-string Sequence.
+
+        Parameters
+        ----------
+        data: any
+            Data to be checked
+
+        Returns
+        -------
+        bool
+        """
+        if isinstance(data, Sequence) and not isinstance(data, str):
+            return True
+        elif NUMPY:
+            if isinstance(data, numpy.ndarray):
+                return True
+        return False
+
+    def to_base(self):
+        """Convert the SyncedList object to list.
+
+        Returns
+        -------
+        converted: list
+            List containing the conveted SyncedList object.
+        """
+        converted = list()
+        for value in self._data:
+            if isinstance(value, SyncedCollection):
+                converted.append(value.to_base())
+            else:
+                converted.append(value)
+        return converted
+
+    def _update(self, data=None):
+        """Update the instance of SyncedList with data using depth-first traversal."""
+        if data is None:
+            data = []
+        if isinstance(data, Sequence) and not isinstance(data, str):
+            with self._suspend_sync():
+                # This loop avoids rebuilding existing synced collections for performance.
+                # TODO: Potential improvements to this code: Remove order constraints.
+                for i in range(min(len(self), len(data))):
+                    if data[i] == self._data[i]:
+                        continue
+                    if isinstance(self._data[i], SyncedCollection):
+                        try:
+                            self._data[i]._update(data[i])
+                            continue
+                        except ValueError:
+                            pass
+                    self._data[i] = self.from_base(data=data[i], parent=self)
+                if len(self._data) > len(data):
+                    self._data = self._data[:len(data)]
+                else:
+                    self.extend(data[len(self):])
+        else:
+            raise ValueError(
+                "Unsupported type: {}. The data must be a non-string sequence or None."
+                .format(type(data)))
+
+    def reset(self, data=None):
+        """Update the instance with new data.
+
+        Parameters
+        ----------
+        data: non-string Sequence, optional
+            Data to update the instance (Default value = None).
+
+        Raises
+        ------
+        ValueError
+            If the data is not instance of non-string seqeuence
+        """
+        if data is None:
+            data = []
+        if NUMPY and isinstance(data, numpy.ndarray):
+            data = data.tolist()
+        if isinstance(data, Sequence) and not isinstance(data, str):
+            with self._suspend_sync():
+                self._data = [self.from_base(data=value, parent=self) for value in data]
+            self.sync()
+        else:
+            raise ValueError(
+                "Unsupported type: {}. The data must be a non-string sequence or None."
+                .format(type(data)))
+
+    def __setitem__(self, key, value):
+        self.load()
+        with self._suspend_sync():
+            self._data[key] = self.from_base(data=value, parent=self)
+        self.sync()
+
+    def __reversed__(self):
+        self.load()
+        return reversed(self._data)
+
+    def __iadd__(self, iterable):
+        self.load()
+        with self._suspend_sync():
+            self._data += [self.from_base(data=value, parent=self) for value in iterable]
+        self.sync()
+        return self
+
+    def insert(self, index, item):
+        self.load()
+        with self._suspend_sync():
+            self._data.insert(index, self.from_base(data=item, parent=self))
+        self.sync()
+
+    def append(self, item):
+        self.load()
+        with self._suspend_sync():
+            self._data.append(self.from_base(data=item, parent=self))
+        self.sync()
+
+    def extend(self, iterable):
+        self.load()
+        with self._suspend_sync():
+            self._data.extend([self.from_base(data=value, parent=self) for value in iterable])
+        self.sync()
+
+    def remove(self, item):
+        self.load()
+        with self._suspend_sync():
+            self._data.remove(self.from_base(data=item, parent=self))
+        self.sync()
+
+    def clear(self):
+        self._data = []
+        self.sync()
diff --git a/signac/core/syncedattrdict.py b/signac/core/syncedattrdict.py
new file mode 100644
index 00000000..ffc592eb
--- /dev/null
+++ b/signac/core/syncedattrdict.py
@@ -0,0 +1,234 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implements the SyncedAttrDict class.
+
+This implements the dict data-structure for SyncedCollection API by
+implementing the convert methods (`to_base`, `from_base`) for dictionaries.
+This class also allows access to values through key indexing or attributes
+named by keys, including nested keys.
+"""
+
+from collections.abc import Mapping
+from collections.abc import MutableMapping
+
+from .synced_collection import SyncedCollection
+from ..errors import InvalidKeyError
+from ..errors import KeyTypeError
+
+
+class SyncedAttrDict(SyncedCollection, MutableMapping):
+    """Implement the dict data structure along with values access through attributes named as keys.
+
+    The SyncedAttrDict inherits from :class:`~core.collection_api.SyncedCollection`
+    and :class:`~collections.abc.MutableMapping`. Therefore, it behaves similar to
+    a :class:`dict`. This class also allows access to values through key indexing or
+    attributes named by keys, including nested keys.
+
+    .. warning::
+
+        While the SyncedAttrDict object behaves like a dictionary, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying backend, copying (even deep
+        copying) a SyncedAttrDict instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new SyncedAttrDict.
+    """
+
+    _PROTECTED_KEYS = ('_data', '_suspend_sync_', '_load', '_sync', '_parent')
+
+    VALID_KEY_TYPES = (str, int, bool, type(None))
+
+    def __init__(self, data=None, **kwargs):
+        super().__init__(**kwargs)
+        if data is None:
+            self._data = {}
+        else:
+            with self._suspend_sync():
+                self._data = {
+                    self._validate_key(key): self.from_base(data=value, parent=self)
+                    for key, value in data.items()
+                }
+            self.sync()
+
+    def to_base(self):
+        """Convert the SyncedDict object to Dictionary.
+
+        Returns:
+        --------
+        converted: dict
+            Dictionary containing the converted synced dict object.
+        """
+        converted = {}
+        for key, value in self._data.items():
+            if isinstance(value, SyncedCollection):
+                converted[key] = value.to_base()
+            else:
+                converted[key] = value
+        return converted
+
+    @classmethod
+    def is_base_type(cls, data):
+        """Check whether the data is an instance of mapping.
+
+        Parameters
+        ----------
+        data: any
+            Data to be checked.
+
+        Returns
+        -------
+        bool
+        """
+        if isinstance(data, Mapping):
+            return True
+        return False
+
+    def _update(self, data=None):
+        """Update the SyncedDict instance with data using depth-first traversal."""
+        if data is None:
+            data = {}
+        if isinstance(data, Mapping):
+            with self._suspend_sync():
+                # This loop avoids rebuilding existing synced collections for performance.
+                for key in data:
+                    if key in self._data:
+                        if data[key] == self._data[key]:
+                            continue
+                        if isinstance(self._data[key], SyncedCollection):
+                            try:
+                                self._data[key]._update(data[key])
+                                continue
+                            except ValueError:
+                                pass
+                    self._data[self._validate_key(key)] = self.from_base(data[key], parent=self)
+                remove = set()
+                for key in self._data:
+                    if key not in data:
+                        remove.add(key)
+                for key in remove:
+                    del self._data[key]
+        else:
+            raise ValueError(
+                "Unsupported type: {}. The data must be a mapping or None.".format(type(data)))
+
+    @staticmethod
+    def _validate_key(key):
+        """Raise an exception if key is invalid. Returns key."""
+        if isinstance(key, SyncedAttrDict.VALID_KEY_TYPES):
+            key = str(key)
+            if '.' in key:
+                raise InvalidKeyError(
+                    "SyncedDict keys may not contain dots ('.'): {}".format(key))
+            else:
+                return key
+        else:
+            raise KeyTypeError(
+                "SyncedDict keys must be str, int, bool or None, not {}".format(type(key).__name__))
+
+    def __setitem__(self, key, value):
+        self.load()
+        with self._suspend_sync():
+            self._data[self._validate_key(key)] = self.from_base(data=value, parent=self)
+        self.sync()
+
+    def reset(self, data=None):
+        """Update the instance with new data.
+
+        Parameters
+        ----------
+        data: mapping
+            Data to update the instance (Default value = None).
+
+        Raises
+        ------
+        ValueError
+            If the data is not instance of mapping
+        """
+        if data is None:
+            data = {}
+
+        if isinstance(data, Mapping):
+            self.load()
+            with self._suspend_sync():
+                self._data = {
+                    self._validate_key(key): self.from_base(data=value, parent=self)
+                    for key, value in data.items()
+                }
+            self.sync()
+        else:
+            raise ValueError(
+                "Unsupported type: {}. The data must be a mapping or None.".format(type(data)))
+
+    def keys(self):
+        self.load()
+        return self._data.keys()
+
+    def values(self):
+        self.load()
+        return self.to_base().values()
+
+    def items(self):
+        self.load()
+        return self.to_base().items()
+
+    def get(self, key, default=None):
+        self.load()
+        return self._data.get(key, default)
+
+    def pop(self, key, default=None):
+        self.load()
+        ret = self._data.pop(key, default)
+        self.sync()
+        return ret
+
+    def popitem(self):
+        self.load()
+        ret = self._data.popitem()
+        self.sync()
+        return ret
+
+    def clear(self):
+        self.load()
+        self._data = {}
+        self.sync()
+
+    def update(self, mapping):
+        self.load()
+        with self._suspend_sync():
+            for key, value in mapping.items():
+                self._data[self._validate_key(key)] = self.from_base(data=value, parent=self)
+        self.sync()
+
+    def setdefault(self, key, default=None):
+        self.load()
+        with self._suspend_sync():
+            ret = self._data.setdefault(self._validate_key(key),
+                                        self.from_base(data=default, parent=self))
+        self.sync()
+        return ret
+
+    def __getattr__(self, name):
+        if name.startswith('__'):
+            raise AttributeError("'SyncedAttrDict' object has no attribute '{}'".format(name))
+        try:
+            return self.__getitem__(name)
+        except KeyError as e:
+            raise AttributeError(e)
+
+    def __setattr__(self, key, value):
+        try:
+            self.__getattribute__('_data')
+        except AttributeError:
+            super().__setattr__(key, value)
+        else:
+            if key.startswith('__') or key in self._PROTECTED_KEYS:
+                super().__setattr__(key, value)
+            else:
+                self.__setitem__(key, value)
+
+    def __delattr__(self, key):
+        if key.startswith('__') or key in self._PROTECTED_KEYS:
+            super().__delattr__(key)
+        else:
+            self.__delitem__(key)


Test patch:
diff --git a/tests/test_synced_collection.py b/tests/test_synced_collection.py
new file mode 100644
index 00000000..0792be92
--- /dev/null
+++ b/tests/test_synced_collection.py
@@ -0,0 +1,604 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+import pytest
+import uuid
+import os
+import json
+from tempfile import TemporaryDirectory
+from collections.abc import MutableMapping
+from collections.abc import MutableSequence
+from copy import deepcopy
+
+from signac.core.synced_list import SyncedCollection
+from signac.core.jsoncollection import JSONDict
+from signac.core.jsoncollection import JSONList
+from signac.errors import InvalidKeyError
+from signac.errors import KeyTypeError
+
+try:
+    import numpy
+    NUMPY = True
+except ImportError:
+    NUMPY = False
+
+FN_JSON = 'test.json'
+
+
+@pytest.fixture
+def testdata():
+    return str(uuid.uuid4())
+
+
+class TestSyncedCollectionBase:
+
+    # this fixture sets temprary directory for tests
+    @pytest.fixture(autouse=True)
+    def synced_collection(self):
+        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        yield
+        self._tmp_dir.cleanup()
+
+    def test_from_base(self):
+        sd = SyncedCollection.from_base(filename=self._fn_,
+                                        data={'a': 0}, backend='signac.core.jsoncollection')
+        assert isinstance(sd, JSONDict)
+        assert 'a' in sd
+        assert sd['a'] == 0
+
+        # invalid input
+        with pytest.raises(ValueError):
+            SyncedCollection.from_base(data={'a': 0}, filename=self._fn_)
+
+
+class TestJSONDict:
+
+    @pytest.fixture
+    def synced_dict(self):
+        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        self._backend_kwargs = {'filename': self._fn_, 'write_concern': False}
+        yield JSONDict(**self._backend_kwargs)
+        self._tmp_dir.cleanup()
+
+    def store(self, data):
+        with open(self._fn_, 'wb') as file:
+            file.write(json.dumps(data).encode())
+
+    def test_init(self, synced_dict):
+        assert len(synced_dict) == 0
+
+    def test_invalid_kwargs(self, synced_dict):
+        # JSONDict raise an error when neither filename nor parent is passed.
+        with pytest.raises(ValueError):
+            return type(synced_dict)()
+
+    def test_isinstance(self, synced_dict):
+        assert isinstance(synced_dict, SyncedCollection)
+        assert isinstance(synced_dict, MutableMapping)
+
+    def test_set_get(self, synced_dict, testdata):
+        key = 'setget'
+        synced_dict.clear()
+        assert not bool(synced_dict)
+        assert len(synced_dict) == 0
+        assert key not in synced_dict
+        synced_dict[key] = testdata
+        assert bool(synced_dict)
+        assert len(synced_dict) == 1
+        assert key in synced_dict
+        assert synced_dict[key] == testdata
+        assert synced_dict.get(key) == testdata
+
+    def test_set_get_explicit_nested(self, synced_dict, testdata):
+        key = 'setgetexplicitnested'
+        synced_dict.setdefault('a', dict())
+        child1 = synced_dict['a']
+        child2 = synced_dict['a']
+        assert child1 == child2
+        assert isinstance(child1, type(child2))
+        assert id(child1) == id(child2)
+        assert not child1
+        assert not child2
+        child1[key] = testdata
+        assert child1
+        assert child2
+        assert key in child1
+        assert key in child2
+        assert child1 == child2
+        assert child1[key] == testdata
+        assert child2[key] == testdata
+
+    def test_copy_value(self, synced_dict, testdata):
+        key = 'copy_value'
+        key2 = 'copy_value2'
+        assert key not in synced_dict
+        assert key2 not in synced_dict
+        synced_dict[key] = testdata
+        assert key in synced_dict
+        assert synced_dict[key] == testdata
+        assert key2 not in synced_dict
+        synced_dict[key2] = synced_dict[key]
+        assert key in synced_dict
+        assert synced_dict[key] == testdata
+        assert key2 in synced_dict
+        assert synced_dict[key2] == testdata
+
+    def test_iter(self, synced_dict, testdata):
+        key1 = 'iter1'
+        key2 = 'iter2'
+        d = {key1: testdata, key2: testdata}
+        synced_dict.update(d)
+        assert key1 in synced_dict
+        assert key2 in synced_dict
+        for i, key in enumerate(synced_dict):
+            assert key in d
+            assert d[key] == synced_dict[key]
+        assert i == 1
+
+    def test_delete(self, synced_dict, testdata):
+        key = 'delete'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        del synced_dict[key]
+        assert len(synced_dict) == 0
+        with pytest.raises(KeyError):
+            synced_dict[key]
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        del synced_dict['delete']
+        assert len(synced_dict) == 0
+        with pytest.raises(KeyError):
+            synced_dict[key]
+
+    def test_update(self, synced_dict, testdata):
+        key = 'update'
+        d = {key: testdata}
+        synced_dict.update(d)
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == d[key]
+
+    def test_pop(self, synced_dict, testdata):
+        key = 'pop'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        d1 = synced_dict.pop(key)
+        assert len(synced_dict) == 0
+        assert testdata == d1
+        with pytest.raises(KeyError):
+            synced_dict[key]
+        d2 = synced_dict.pop(key, 'default')
+        assert len(synced_dict) == 0
+        assert d2 == 'default'
+
+    def test_popitem(self, synced_dict, testdata):
+        key = 'pop'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        key1, d1 = synced_dict.popitem()
+        assert len(synced_dict) == 0
+        assert key == key1
+        assert testdata == d1
+        with pytest.raises(KeyError):
+            synced_dict[key]
+
+    def test_values(self, synced_dict, testdata):
+        data = {'value1': testdata, 'value_nested': {'value2': testdata}}
+        synced_dict.reset(data)
+        assert 'value1' in synced_dict
+        assert 'value_nested' in synced_dict
+        for val in synced_dict.values():
+            assert not isinstance(val, SyncedCollection)
+            assert val in data.values()
+
+    def test_items(self, synced_dict, testdata):
+        data = {'item1': testdata, 'item_nested': {'item2': testdata}}
+        synced_dict.reset(data)
+        assert 'item1' in synced_dict
+        assert 'item_nested' in synced_dict
+        for key, val in synced_dict.items():
+            assert synced_dict[key] == data[key]
+            assert not isinstance(val, type(synced_dict))
+            assert (key, val) in data.items()
+
+    def test_reset(self, synced_dict, testdata):
+        key = 'reset'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        synced_dict.reset()
+        assert len(synced_dict) == 0
+        synced_dict.reset({'reset': 'abc'})
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == 'abc'
+
+        # invalid input
+        with pytest.raises(ValueError):
+            synced_dict.reset([0, 1])
+
+    def test_attr_dict(self, synced_dict, testdata):
+        key = 'test'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert key in synced_dict
+        assert synced_dict[key] == testdata
+        assert synced_dict.get(key) == testdata
+        assert synced_dict.test == testdata
+        del synced_dict.test
+        assert len(synced_dict) == 0
+        assert key not in synced_dict
+        key = 'test2'
+        synced_dict.test2 = testdata
+        assert len(synced_dict) == 1
+        assert key in synced_dict
+        assert synced_dict[key] == testdata
+        assert synced_dict.get(key) == testdata
+        assert synced_dict.test2 == testdata
+        with pytest.raises(AttributeError):
+            synced_dict.not_exist
+
+        # deleting a protected attribute
+        synced_dict.load()
+        del synced_dict._parent
+        # deleting _parent will lead to recursion as _parent is treated as key
+        # load() will check for _parent and __getattr__ will call __getitem__ which calls load()
+        with pytest.raises(RecursionError):
+            synced_dict.load()
+
+    def test_clear(self, synced_dict, testdata):
+        key = 'clear'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        synced_dict.clear()
+        assert len(synced_dict) == 0
+
+    def test_repr(self, synced_dict):
+        repr(synced_dict)
+        p = eval(repr(synced_dict))
+        assert repr(p) == repr(synced_dict)
+        assert p == synced_dict
+
+    def test_str(self, synced_dict):
+        str(synced_dict) == str(synced_dict.to_base())
+
+    def test_call(self, synced_dict, testdata):
+        key = 'call'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        assert isinstance(synced_dict(), dict)
+        assert not isinstance(synced_dict(), SyncedCollection)
+        assert synced_dict() == synced_dict.to_base()
+
+    def test_reopen(self, synced_dict, testdata):
+        key = 'reopen'
+        synced_dict[key] = testdata
+        synced_dict2 = deepcopy(synced_dict)
+        synced_dict.sync()
+        del synced_dict  # possibly unsafe
+        synced_dict2.load()
+        assert len(synced_dict2) == 1
+        assert synced_dict2[key] == testdata
+
+    def test_update_recursive(self, synced_dict, testdata):
+        synced_dict.a = {'a': 1}
+        synced_dict.b = 'test'
+        synced_dict.c = [0, 1, 2]
+        assert 'a' in synced_dict
+        assert 'b' in synced_dict
+        assert 'c' in synced_dict
+        data = {'a': 1, 'c': [0, 1, 3], 'd': 1}
+        self.store(data)
+        assert synced_dict == data
+
+        # invalid data
+        data = [1, 2, 3]
+        self.store(data)
+        with pytest.raises(ValueError):
+            synced_dict.load()
+
+    def test_copy_as_dict(self, synced_dict, testdata):
+        key = 'copy'
+        synced_dict[key] = testdata
+        copy = dict(synced_dict)
+        del synced_dict
+        assert key in copy
+        assert copy[key] == testdata
+
+    def test_nested_dict(self, synced_dict):
+        synced_dict['a'] = dict(a=dict())
+        child1 = synced_dict['a']
+        child2 = synced_dict['a']['a']
+        assert isinstance(child1, type(synced_dict))
+        assert isinstance(child1, type(child2))
+
+    def test_nested_dict_with_list(self, synced_dict):
+        synced_dict['a'] = [1, 2, 3]
+        child1 = synced_dict['a']
+        synced_dict['a'].append(dict(a=[1, 2, 3]))
+        child2 = synced_dict['a'][3]
+        child3 = synced_dict['a'][3]['a']
+        assert isinstance(child2, type(synced_dict))
+        assert isinstance(child1, type(child3))
+        assert isinstance(child1, SyncedCollection)
+        assert isinstance(child3, SyncedCollection)
+
+    def test_write_invalid_type(self, synced_dict, testdata):
+        class Foo(object):
+            pass
+
+        key = 'write_invalid_type'
+        synced_dict[key] = testdata
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+        d2 = Foo()
+        with pytest.raises(TypeError):
+            synced_dict[key + '2'] = d2
+        assert len(synced_dict) == 1
+        assert synced_dict[key] == testdata
+
+    def test_keys_with_dots(self, synced_dict):
+        with pytest.raises(InvalidKeyError):
+            synced_dict['a.b'] = None
+
+    def test_keys_valid_type(self, synced_dict, testdata):
+
+        class MyStr(str):
+            pass
+        for key in ('key', MyStr('key'), 0, None, True):
+            synced_dict[key] = testdata
+            assert str(key) in synced_dict
+            assert synced_dict[str(key)] == testdata
+
+    def test_keys_invalid_type(self, synced_dict, testdata):
+
+        class A:
+            pass
+        for key in (0.0, A(), (1, 2, 3)):
+            with pytest.raises(KeyTypeError):
+                synced_dict[key] = testdata
+        for key in ([], {}, dict()):
+            with pytest.raises(TypeError):
+                synced_dict[key] = testdata
+
+
+class TestJSONList:
+
+    _write_concern = False
+
+    @pytest.fixture
+    def synced_list(self):
+        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        self._backend_kwargs = {'filename': self._fn_, 'write_concern': self._write_concern}
+        yield JSONList(**self._backend_kwargs)
+        self._tmp_dir.cleanup()
+
+    def store(self, data):
+        with open(self._fn_, 'wb') as file:
+            file.write(json.dumps(data).encode())
+
+    def test_init(self, synced_list):
+        assert len(synced_list) == 0
+
+    def test_invalid_kwargs(self, synced_list):
+        # JSONList raise an error when neither filename nor parent is passed.
+        with pytest.raises(ValueError):
+            return type(synced_list)()
+
+    def test_isinstance(self, synced_list):
+        assert isinstance(synced_list, MutableSequence)
+        assert isinstance(synced_list, SyncedCollection)
+
+    def test_set_get(self, synced_list, testdata):
+        synced_list.clear()
+        assert not bool(synced_list)
+        assert len(synced_list) == 0
+        synced_list.append(testdata)
+        assert bool(synced_list)
+        assert len(synced_list) == 1
+        assert synced_list[0] == testdata
+        synced_list[0] = 1
+        assert bool(synced_list)
+        assert len(synced_list) == 1
+        assert synced_list[0] == 1
+
+    @pytest.mark.skipif(not NUMPY, reason='test requires the numpy package')
+    def test_set_get_numpy_data(self, synced_list):
+        data = numpy.random.rand(3, 4)
+        data_as_list = data.tolist()
+        synced_list.reset(data)
+        assert len(synced_list) == len(data_as_list)
+        assert synced_list == data_as_list
+        data2 = numpy.random.rand(3, 4)
+        synced_list.append(data2)
+        assert len(synced_list) == len(data_as_list) + 1
+        assert synced_list[len(data_as_list)] == data2.tolist()
+        data3 = numpy.float_(3.14)
+        synced_list.append(data3)
+        assert len(synced_list) == len(data_as_list) + 2
+        assert synced_list[len(data_as_list) + 1] == data3
+
+    def test_iter(self, synced_list, testdata):
+        d = [testdata, 43]
+        synced_list.extend(d)
+        for i in range(len(synced_list)):
+            assert d[i] == synced_list[i]
+        assert i == 1
+
+    def test_delete(self, synced_list, testdata):
+        synced_list.append(testdata)
+        assert len(synced_list) == 1
+        assert synced_list[0] == testdata
+        del synced_list[0]
+        assert len(synced_list) == 0
+        with pytest.raises(IndexError):
+            synced_list[0]
+
+    def test_extend(self, synced_list, testdata):
+        d = [testdata]
+        synced_list.extend(d)
+        assert len(synced_list) == 1
+        assert synced_list[0] == d[0]
+        d1 = testdata
+        synced_list += [d1]
+        assert len(synced_list) == 2
+        assert synced_list[0] == d[0]
+        assert synced_list[1] == d1
+
+    def test_clear(self, synced_list, testdata):
+        synced_list.append(testdata)
+        assert len(synced_list) == 1
+        assert synced_list[0] == testdata
+        synced_list.clear()
+        assert len(synced_list) == 0
+
+    def test_reset(self, synced_list):
+        synced_list.reset([1, 2, 3])
+        assert len(synced_list) == 3
+        assert synced_list == [1, 2, 3]
+        synced_list.reset()
+        assert len(synced_list) == 0
+        synced_list.reset([3, 4])
+        assert len(synced_list) == 2
+        assert synced_list == [3, 4]
+
+        # invalid inputs
+        with pytest.raises(ValueError):
+            synced_list.reset({'a': 1})
+
+        with pytest.raises(ValueError):
+            synced_list.reset(1)
+
+    def test_insert(self, synced_list, testdata):
+        synced_list.reset([1, 2])
+        assert len(synced_list) == 2
+        synced_list.insert(1, testdata)
+        assert len(synced_list) == 3
+        assert synced_list[1] == testdata
+
+    def test_reversed(self,  synced_list):
+        data = [1, 2, 3]
+        synced_list.reset([1, 2, 3])
+        assert len(synced_list) == 3
+        assert synced_list == data
+        for i, j in zip(reversed(synced_list), reversed(data)):
+            assert i == j
+
+    def test_remove(self, synced_list):
+        synced_list.reset([1, 2])
+        assert len(synced_list) == 2
+        synced_list.remove(1)
+        assert len(synced_list) == 1
+        assert synced_list[0] == 2
+        synced_list.reset([1, 2, 1])
+        synced_list.remove(1)
+        assert len(synced_list) == 2
+        assert synced_list[0] == 2
+        assert synced_list[1] == 1
+
+    def test_call(self, synced_list):
+        synced_list.reset([1, 2])
+        assert len(synced_list) == 2
+        assert isinstance(synced_list(), list)
+        assert not isinstance(synced_list(), SyncedCollection)
+        assert synced_list() == [1, 2]
+
+    def test_update_recursive(self, synced_list):
+        synced_list.reset([{'a': 1}, 'b', [1, 2, 3]])
+        assert synced_list == [{'a': 1}, 'b', [1, 2, 3]]
+        data = ['a', 'b', [1, 2, 4], 'd']
+        self.store(data)
+        assert synced_list == data
+        data1 = ['a', 'b']
+        self.store(data1)
+        assert synced_list == data1
+
+        # inavlid data in file
+        data2 = {'a': 1}
+        self.store(data2)
+        with pytest.raises(ValueError):
+            synced_list.load()
+
+    def test_reopen(self, synced_list, testdata):
+        synced_list2 = deepcopy(synced_list)
+        synced_list.append(testdata)
+        synced_list.sync()
+        del synced_list  # possibly unsafe
+        synced_list2.load()
+        assert len(synced_list2) == 1
+        assert synced_list2[0] == testdata
+
+    def test_copy_as_list(self, synced_list, testdata):
+        synced_list.append(testdata)
+        assert synced_list[0] == testdata
+        copy = list(synced_list)
+        del synced_list
+        assert copy[0] == testdata
+
+    def test_repr(self, synced_list):
+        repr(synced_list)
+        p = eval(repr(synced_list))
+        assert repr(p) == repr(synced_list)
+        assert p == synced_list
+
+    def test_str(self, synced_list):
+        str(synced_list) == str(synced_list.to_base())
+
+    def test_nested_list(self, synced_list):
+        synced_list.reset([1, 2, 3])
+        synced_list.append([2, 4])
+        child1 = synced_list[3]
+        child2 = synced_list[3]
+        assert child1 == child2
+        assert isinstance(child1, type(child2))
+        assert isinstance(child1, type(synced_list))
+        assert id(child1) == id(child2)
+        child1.append(1)
+        assert child2[2] == child1[2]
+        assert child1 == child2
+        assert len(synced_list) == 4
+        assert isinstance(child1, type(child2))
+        assert isinstance(child1, type(synced_list))
+        assert id(child1) == id(child2)
+        del child1[0]
+        assert child1 == child2
+        assert len(synced_list) == 4
+        assert isinstance(child1, type(child2))
+        assert isinstance(child1, type(synced_list))
+        assert id(child1) == id(child2)
+
+    def test_nested_list_with_dict(self, synced_list):
+        synced_list.reset([{'a': [1, 2, 3, 4]}])
+        child1 = synced_list[0]
+        child2 = synced_list[0]['a']
+        assert isinstance(child2, SyncedCollection)
+        assert isinstance(child1, SyncedCollection)
+
+
+class TestJSONDictWriteConcern(TestJSONDict):
+
+    @pytest.fixture
+    def synced_dict(self):
+        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        self._backend_kwargs = {'filename': self._fn_, 'write_concern': True}
+        yield JSONDict(**self._backend_kwargs)
+        self._tmp_dir.cleanup()
+
+
+class TestJSONListWriteConcern(TestJSONList):
+
+    @pytest.fixture
+    def synced_list(self):
+        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        self._backend_kwargs = {'filename': self._fn_, 'write_concern': True}
+        yield JSONList(**self._backend_kwargs)
+        self._tmp_dir.cleanup()