Repository: glotzerlab/signac
Pull request #364 (2020-07-31 20:22:40)

Added backends to synced_collection
<!-- Provide a general summary of your changes in the Title above -->

## Description
<!-- Describe your changes in detail -->
Implemented Redis, Zarr, and MongoDB backends for SyncedCollection API.
## Motivation and Context
<!-- Why is this change required? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here. -->
Related to #249.
## Types of Changes
<!-- Please select all items that apply either now or after creating the pull request: -->
- [ ] Documentation update
- [ ] Bug fix
- [x] New feature
- [ ] Breaking change<sup>1</sup>

<sup>1</sup>The change breaks (or has the potential to break) existing functionality.

## Checklist:
<!-- Please select all items that apply either now or after creating the pull request. -->
<!-- If you are unsure about any of these items, do not hesitate to ask! -->
- [x] I am familiar with the [**Contributing Guidelines**](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md).
- [x] I agree with the terms of the [**Contributor Agreement**](https://github.com/glotzerlab/signac/blob/master/ContributorAgreement.md).
- [x] My name is on the [list of contributors](https://github.com/glotzerlab/signac/blob/master/contributors.yaml).
- [x] My code follows the [code style guideline](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md#code-style) of this project.
- [x] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:
- [ ] I have updated the API documentation as part of the package doc-strings.
- [ ] I have created a separate pull request to update the [framework documentation](https://docs.signac.io/) on [signac-docs](https://github.com/glotzerlab/signac-docs) and linked it here.
- [ ] I have updated the [changelog](https://github.com/glotzerlab/signac/blob/master/changelog.txt) and added all related issue and pull request numbers for future reference (if applicable). See example below.


Example for a changelog entry: `Fix issue with launching rockets to the moon (#101, #212).`


codecov[bot]: # [Codecov](https://codecov.io/gh/glotzerlab/signac/pull/364?src=pr&el=h1) Report
> Merging [#364](https://codecov.io/gh/glotzerlab/signac/pull/364?src=pr&el=desc) into [feature/synced_collections](https://codecov.io/gh/glotzerlab/signac/commit/7605b6b1def69d28ac97498e5a0382de370cd877&el=desc) will **decrease** coverage by `0.52%`.
> The diff coverage is `33.33%`.

[![Impacted file tree graph](https://codecov.io/gh/glotzerlab/signac/pull/364/graphs/tree.svg?width=650&height=150&src=pr&token=FghbjTyzgM)](https://codecov.io/gh/glotzerlab/signac/pull/364?src=pr&el=tree)

```diff
@@                      Coverage Diff                       @@
##           feature/synced_collections     #364      +/-   ##
==============================================================
- Coverage                       77.28%   76.75%   -0.53%     
==============================================================
  Files                              47       50       +3     
  Lines                            7453     7543      +90     
==============================================================
+ Hits                             5760     5790      +30     
- Misses                           1693     1753      +60     
```


| [Impacted Files](https://codecov.io/gh/glotzerlab/signac/pull/364?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [signac/core/mongodbcollection.py](https://codecov.io/gh/glotzerlab/signac/pull/364/diff?src=pr&el=tree#diff-c2lnbmFjL2NvcmUvbW9uZ29kYmNvbGxlY3Rpb24ucHk=) | `0.00% <0.00%> (ø)` | |
| [signac/core/rediscollection.py](https://codecov.io/gh/glotzerlab/signac/pull/364/diff?src=pr&el=tree#diff-c2lnbmFjL2NvcmUvcmVkaXNjb2xsZWN0aW9uLnB5) | `0.00% <0.00%> (ø)` | |
| [signac/core/zarrcollection.py](https://codecov.io/gh/glotzerlab/signac/pull/364/diff?src=pr&el=tree#diff-c2lnbmFjL2NvcmUvemFycmNvbGxlY3Rpb24ucHk=) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/glotzerlab/signac/pull/364?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/glotzerlab/signac/pull/364?src=pr&el=footer). Last update [7605b6b...113b274](https://codecov.io/gh/glotzerlab/signac/pull/364?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

csadorf: @bdice @tommy-waltmann This appears almost ready to merge.
csadorf: @bdice Do you still want to review?
bdice: > @bdice Do you still want to review?

@csadorf I’m not able to review for a few days. I am confident that the approvals here are enough to merge.

Patch:
diff --git a/requirements-dev.txt b/requirements-dev.txt
index 1a3af8ef..affb371f 100644
--- a/requirements-dev.txt
+++ b/requirements-dev.txt
@@ -11,3 +11,5 @@ pytest-subtests==0.3.1
 pydocstyle==5.0.2
 pytest-cov==2.10.0
 pymongo==3.10.1
+zarr==2.4.0
+redis==3.5.3
diff --git a/signac/core/jsoncollection.py b/signac/core/collection_json.py
similarity index 88%
rename from signac/core/jsoncollection.py
rename to signac/core/collection_json.py
index 0a7e0212..18bcd7de 100644
--- a/signac/core/jsoncollection.py
+++ b/signac/core/collection_json.py
@@ -23,13 +23,10 @@ class JSONCollection(SyncedCollection):
     backend = __name__  # type: ignore
 
     def __init__(self, filename=None, write_concern=False, **kwargs):
-        self._filename = os.path.realpath(filename) if filename is not None else None
+        self._filename = None if filename is None else os.path.realpath(filename)
         self._write_concern = write_concern
+        kwargs['name'] = filename
         super().__init__(**kwargs)
-        if (filename is None) == (self._parent is None):
-            raise ValueError(
-                "Illegal argument combination, one of the two arguments, "
-                "parent or filename must be None, but not both.")
 
     def _load(self):
         """Load the data from a JSON-file."""
@@ -96,22 +93,20 @@ class JSONDict(JSONCollection, SyncedAttrDict):
     ----------
     filename: str, optional
         The filename of the associated JSON file on disk (Default value = None).
+    write_concern: bool, optional
+        Ensure file consistency by writing changes back to a temporary file
+        first, before replacing the original file (Default value = None).
     data: mapping, optional
         The intial data pass to JSONDict. Defaults to `list()`
     parent: object, optional
         A parent instance of JSONDict or None (Default value = None).
-    write_concern: bool, optional
-        Ensure file consistency by writing changes back to a temporary file
-        first, before replacing the original file (Default value = None).
     """
 
-    pass
-
 
 class JSONList(JSONCollection, SyncedList):
     """A non-string sequence interface to a persistent JSON file.
 
-    The JSONDict inherits from :class:`~core.collection_api.SyncedCollection`
+    The JSONList inherits from :class:`~core.collection_api.SyncedCollection`
     and :class:`~core.syncedlist.SyncedList`.
 
     .. code-block:: python
@@ -134,18 +129,16 @@ class JSONList(JSONCollection, SyncedList):
 
     Parameters
     ----------
-    filename: str
+    filename: str, optional
         The filename of the associated JSON file on disk (Default value = None).
-    data: non-str Sequence
-        The intial data pass to JSONDict
-    parent: object
-        A parent instance of JSONDict or None (Default value = None).
-    write_concern: bool
+    write_concern: bool, optional
         Ensure file consistency by writing changes back to a temporary file
         first, before replacing the original file (Default value = None).
+    data: non-str Sequence, optional
+        The intial data pass to JSONList
+    parent: object, optional
+        A parent instance of JSONList or None (Default value = None).
     """
 
-    pass
-
 
 SyncedCollection.register(JSONDict, JSONList)
diff --git a/signac/core/collection_mongodb.py b/signac/core/collection_mongodb.py
new file mode 100644
index 00000000..83897e71
--- /dev/null
+++ b/signac/core/collection_mongodb.py
@@ -0,0 +1,135 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implements MongoDB-backend.
+
+This implements the MongoDB-backend for SyncedCollection API by
+implementing sync and load methods.
+"""
+from copy import deepcopy
+
+from .synced_collection import SyncedCollection
+from .syncedattrdict import SyncedAttrDict
+from .synced_list import SyncedList
+
+
+class MongoDBCollection(SyncedCollection):
+    """Implement sync and load using a MongoDB backend."""
+
+    backend = __name__  # type: ignore
+
+    def __init__(self, collection=None, **kwargs):
+        import bson  # for InvalidDocument
+
+        self._collection = collection
+        self._errors = bson.errors
+        self._key = type(self).__name__ + '::name'
+        super().__init__(**kwargs)
+
+    def _load(self):
+        """Load the data from a MongoDB."""
+        blob = self._collection.find_one({self._key: self._name})
+        return blob['data'] if blob is not None else None
+
+    def _sync(self):
+        """Write the data from MongoDB."""
+        data = self.to_base()
+        data_to_insert = {self._key: self._name, 'data': data}
+        try:
+            self._collection.replace_one({self._key: self._name}, data_to_insert, True)
+        except self._errors.InvalidDocument as err:
+            raise TypeError(str(err))
+
+    def _pseudo_deepcopy(self):
+        """Return a copy of instance.
+
+        It is a psuedo implementation for `deepcopy` because
+        `pymongo.Collection` does not support `deepcopy` method.
+        """
+        return type(self)(collection=self._collection, name=self._name, data=self.to_base(),
+                          parent=deepcopy(self._parent))
+
+
+class MongoDBDict(MongoDBCollection, SyncedAttrDict):
+    """A dict-like mapping interface to a persistent Mongo-database.
+
+    The MongoDBDict inherits from :class:`~core.collection_api.MongoCollection`
+    and :class:`~core.syncedattrdict.SyncedAttrDict`.
+
+    .. code-block:: python
+
+        doc = MongoDBDict('data')
+        doc['foo'] = "bar"
+        assert doc.foo == doc['foo'] == "bar"
+        assert 'foo' in doc
+        del doc['foo']
+
+    .. code-block:: python
+
+        >>> doc['foo'] = dict(bar=True)
+        >>> doc
+        {'foo': {'bar': True}}
+        >>> doc.foo.bar = False
+        {'foo': {'bar': False}}
+
+    .. warning::
+
+        While the MongoDBDict object behaves like a dictionary, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying database, copying (even deep
+        copying) a MongoDBDict instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new MongoDBDict
+        instance: `new_dict = MongoDBDict(old_dict.to_base())`.
+
+    Parameters
+    ----------
+    collection : object, optional
+        A pymongo.Collection instance.
+    data: mapping, optional
+        The intial data pass to MongoDBDict. Defaults to `dict()`.
+    name: str, optional
+        The name of the  collection (Default value = None).
+    parent: object, optional
+        A parent instance of MongoDBDict (Default value = None).
+    """
+
+
+class MongoDBList(MongoDBCollection, SyncedList):
+    """A non-string sequence interface to a persistent Mongo file.
+
+    The MongoDBList inherits from :class:`~core.synced_collection.SyncedCollection`
+    and :class:`~core.syncedlist.SyncedList`.
+
+    .. code-block:: python
+
+        synced_list = MongoDBList('data')
+        synced_list.append("bar")
+        assert synced_list[0] == "bar"
+        assert len(synced_list) == 1
+        del synced_list[0]
+
+    .. warning::
+
+        While the MongoDBList object behaves like a list, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying database, copying (even deep
+        copying) a MongoDBList instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new MongoDBList
+        instance: `new_list = MongoDBList(old_list.to_base())`.
+
+    Parameters
+    ----------
+    collection : object, optional
+        A pymongo.Collection instance (Default value = None).
+    data: non-str Sequence, optional
+        The intial data pass to MongoDBList. Defaults to `list()`.
+    name: str, optional
+        The name of the  collection (Default value = None).
+    parent: object, optional
+        A parent instance of MongoDBList (Default value = None).
+    """
+
+
+SyncedCollection.register(MongoDBDict, MongoDBList)
diff --git a/signac/core/collection_redis.py b/signac/core/collection_redis.py
new file mode 100644
index 00000000..1dc9e504
--- /dev/null
+++ b/signac/core/collection_redis.py
@@ -0,0 +1,127 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implements Redis-backend.
+
+This implements the Redis-backend for SyncedCollection API by
+implementing sync and load methods.
+"""
+import json
+from copy import deepcopy
+
+from .synced_collection import SyncedCollection
+from .syncedattrdict import SyncedAttrDict
+from .synced_list import SyncedList
+
+
+class RedisCollection(SyncedCollection):
+    """Implement sync and load using a Redis backend."""
+
+    backend = __name__  # type: ignore
+
+    def __init__(self, client=None, **kwargs):
+        self._client = client
+        super().__init__(**kwargs)
+
+    def _load(self):
+        """Load the data from a Redis-database."""
+        blob = self._client.get(self._name)
+        return None if blob is None else json.loads(blob)
+
+    def _sync(self):
+        """Write the data from Redis-database."""
+        self._client.set(self._name, json.dumps(self.to_base()).encode())
+
+    def _pseudo_deepcopy(self):
+        """Return a copy of instance.
+
+        It is a psuedo implementation for `deepcopy` because
+        `redis.Redis` does not support `deepcopy` method.
+        """
+        return type(self)(client=self._client, name=self._name, data=self.to_base(),
+                          parent=deepcopy(self._parent))
+
+
+class RedisDict(RedisCollection, SyncedAttrDict):
+    """A dict-like mapping interface to a persistent Redis-database.
+
+    The RedisDict inherits from :class:`~core.rediscollection.RedisCollection`
+    and :class:`~core.syncedattrdict.SyncedAttrDict`.
+
+    .. code-block:: python
+
+        doc = RedisDict('data')
+        doc['foo'] = "bar"
+        assert doc.foo == doc['foo'] == "bar"
+        assert 'foo' in doc
+        del doc['foo']
+
+    .. code-block:: python
+
+        >>> doc['foo'] = dict(bar=True)
+        >>> doc
+        {'foo': {'bar': True}}
+        >>> doc.foo.bar = False
+        {'foo': {'bar': False}}
+
+    .. warning::
+
+        While the RedisDict object behaves like a dictionary, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying database, copying (even deep
+        copying) a RedisDict instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new RedisDict
+        instance: `new_dict = RedisDict(old_dict.to_base())`.
+
+    Parameters
+    ----------
+    client: object, optional
+        A redis client (Default value = None).
+    data: mapping, optional
+        The intial data pass to RedisDict. Defaults to `dict()`
+    name: str, optional
+        The name of the  collection (Default value = None).
+    parent: object, optional
+        A parent instance of RedisDict (Default value = None).
+    """
+
+
+class RedisList(RedisCollection, SyncedList):
+    """A non-string sequence interface to a persistent Redis file.
+
+    The RedisList inherits from :class:`~core.collection_api.SyncedCollection`
+    and :class:`~core.syncedlist.SyncedList`.
+
+    .. code-block:: python
+
+        synced_list = RedisList('data')
+        synced_list.append("bar")
+        assert synced_list[0] == "bar"
+        assert len(synced_list) == 1
+        del synced_list[0]
+
+    .. warning::
+
+        While the RedisList object behaves like a list, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying database, copying (even deep
+        copying) a RedisList instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new RedisList
+        instance: `new_list = RedisList(old_list.to_base())`.
+
+    Parameters
+    ----------
+    client: object, optional
+        A redis client (Default value = None).
+    data: non-str Sequence, optional
+        The intial data pass to RedisList. Defaults to `list()`
+    name: str, optional
+        The name of the  collection (Default value = None).
+    parent: object, optional
+        A parent instance of RedisList (Default value = None).
+    """
+
+
+SyncedCollection.register(RedisDict, RedisList)
diff --git a/signac/core/collection_zarr.py b/signac/core/collection_zarr.py
new file mode 100644
index 00000000..3cfef408
--- /dev/null
+++ b/signac/core/collection_zarr.py
@@ -0,0 +1,130 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implements Zarr-backend.
+
+This implements the Zarr-backend for SyncedCollection API by
+implementing sync and load methods.
+"""
+from copy import deepcopy
+
+from .synced_collection import SyncedCollection
+from .syncedattrdict import SyncedAttrDict
+from .synced_list import SyncedList
+
+
+class ZarrCollection(SyncedCollection):
+    """Implement sync and load using a Zarr backend."""
+
+    backend = __name__  # type: ignore
+
+    def __init__(self, group=None, **kwargs):
+        import numcodecs  # zarr depends on numcodecs
+
+        self._root = group
+        self._object_codec = numcodecs.JSON()
+        super().__init__(**kwargs)
+
+    def _load(self):
+        """Load the data from zarr-store."""
+        try:
+            return self._root[self._name][0]
+        except KeyError:
+            return None
+
+    def _sync(self):
+        """Write the data to zarr-store."""
+        data = self.to_base()
+        dataset = self._root.require_dataset(
+            self._name, overwrite=True, shape=1, dtype='object', object_codec=self._object_codec)
+        dataset[0] = data
+
+    def __deepcopy__(self, memo):
+        return type(self)(group=deepcopy(self._root, memo), name=self._name, data=self.to_base(),
+                          parent=deepcopy(self._parent, memo))
+
+
+class ZarrDict(ZarrCollection, SyncedAttrDict):
+    """A dict-like mapping interface to a persistent Zarr-database.
+
+    The ZarrDict inherits from :class:`~core.collection_api.ZarrCollection`
+    and :class:`~core.syncedattrdict.SyncedAttrDict`.
+
+    .. code-block:: python
+
+        doc = ZarrDict('data')
+        doc['foo'] = "bar"
+        assert doc.foo == doc['foo'] == "bar"
+        assert 'foo' in doc
+        del doc['foo']
+
+    .. code-block:: python
+
+        >>> doc['foo'] = dict(bar=True)
+        >>> doc
+        {'foo': {'bar': True}}
+        >>> doc.foo.bar = False
+        {'foo': {'bar': False}}
+
+    .. warning::
+
+        While the ZarrDict object behaves like a dictionary, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying database, copying (even deep
+        copying) a ZarrDict instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new ZarrDict
+        instance: `new_dict = ZarrDict(old_dict.to_base())`.
+
+    Parameters
+    ----------
+    group: object, optional
+        A zarr.hierarchy.Group instance (Default value = None).
+    data: mapping, optional
+        The intial data pass to ZarrDict. Defaults to `dict()`.
+    name: str, optional
+        The name of the  collection (Default value = None).
+    parent: object, optional
+        A parent instance of ZarrDict or None (Default value = None).
+    """
+
+
+class ZarrList(ZarrCollection, SyncedList):
+    """A non-string sequence interface to a persistent Zarr file.
+
+    The ZarrList inherits from :class:`~core.collection_api.ZarrCollection`
+    and :class:`~core.syncedlist.SyncedList`.
+
+    .. code-block:: python
+
+        synced_list = ZarrList('data')
+        synced_list.append("bar")
+        assert synced_list[0] == "bar"
+        assert len(synced_list) == 1
+        del synced_list[0]
+
+    .. warning::
+
+        While the ZarrList object behaves like a list, there are
+        important distinctions to remember. In particular, because operations
+        are reflected as changes to an underlying database, copying (even deep
+        copying) a ZarrList instance may exhibit unexpected behavior. If a
+        true copy is required, you should use the `to_base()` method to get a
+        dictionary representation, and if necessary construct a new ZarrList
+        instance: `new_list = ZarrList(old_list.to_base())`.
+
+    Parameters
+    ----------
+
+    group: object, optional
+        A zarr.hierarchy.Group instance (Default value = None).
+    data: non-str Sequence, optional
+        The intial data pass to ZarrList. Defaults to `list()`.
+    name: str, optional
+        The name of the  collection (Default value = None).
+    parent: object, optional
+        A parent instance of ZarrList or None (Default value = None).
+    """
+
+
+SyncedCollection.register(ZarrDict, ZarrList)
diff --git a/signac/core/synced_collection.py b/signac/core/synced_collection.py
index f275238a..52eb7c81 100644
--- a/signac/core/synced_collection.py
+++ b/signac/core/synced_collection.py
@@ -31,10 +31,15 @@ class SyncedCollection(Collection):
 
     backend = None
 
-    def __init__(self, parent=None):
+    def __init__(self, name=None, parent=None):
         self._data = None
         self._parent = parent
+        self._name = name
         self._suspend_sync_ = 0
+        if (name is None) == (parent is None):
+            raise ValueError(
+                "Illegal argument combination, one of the two arguments, "
+                "parent or name must be None, but not both.")
 
     @classmethod
     def register(cls, *args):
diff --git a/signac/core/syncedattrdict.py b/signac/core/syncedattrdict.py
index ffc592eb..20665045 100644
--- a/signac/core/syncedattrdict.py
+++ b/signac/core/syncedattrdict.py
@@ -35,7 +35,7 @@ class SyncedAttrDict(SyncedCollection, MutableMapping):
         dictionary representation, and if necessary construct a new SyncedAttrDict.
     """
 
-    _PROTECTED_KEYS = ('_data', '_suspend_sync_', '_load', '_sync', '_parent')
+    _PROTECTED_KEYS = ('_data', '_name', '_suspend_sync_', '_load', '_sync', '_parent')
 
     VALID_KEY_TYPES = (str, int, bool, type(None))
 


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index e94b7439..fe2f608e 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -2,6 +2,7 @@ from contextlib import contextmanager
 from packaging import version
 import signac
 import pytest
+import uuid
 
 
 @contextmanager
@@ -11,3 +12,8 @@ def deprecated_in_version(version_string):
             yield
     else:
         yield
+
+
+@pytest.fixture
+def testdata():
+    return str(uuid.uuid4())
diff --git a/tests/test_mongodb_collection.py b/tests/test_mongodb_collection.py
new file mode 100644
index 00000000..328ebc06
--- /dev/null
+++ b/tests/test_mongodb_collection.py
@@ -0,0 +1,57 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+import pytest
+
+from signac.core.collection_mongodb import MongoDBDict
+from signac.core.collection_mongodb import MongoDBList
+from test_synced_collection import TestJSONDict
+from test_synced_collection import TestJSONList
+
+try:
+    import pymongo
+    try:
+        # test the mongodb server
+        MongoClient = pymongo.MongoClient()
+        tmp_collection = MongoClient['test_db']['test']
+        tmp_collection.insert_one({'test': '0'})
+        ret = tmp_collection.find_one({'test': '0'})
+        assert ret['test'] == '0'
+        tmp_collection.drop()
+        PYMONGO = True
+    except (pymongo.errors.ServerSelectionTimeoutError, AssertionError):
+        PYMONGO = False
+except ImportError:
+    PYMONGO = False
+
+
+@pytest.mark.skipif(not PYMONGO, reason='test requires the pymongo package and mongodb server')
+class TestMongoDBDict(TestJSONDict):
+
+    @pytest.fixture
+    def synced_dict(self, request):
+        self._client = MongoClient
+        self._name = 'test'
+        self._collection = self._client.test_db.test_dict
+        yield MongoDBDict(name=self._name, collection=self._collection)
+        self._collection.drop()
+
+    def store(self, data):
+        data_to_insert = {'MongoDBDict::name': self._name, 'data': data}
+        self._collection.replace_one({'MongoDBDict::name': self._name}, data_to_insert)
+
+
+@pytest.mark.skipif(not PYMONGO, reason='test requires the pymongo package and mongodb server')
+class TestMongoDBList(TestJSONList):
+
+    @pytest.fixture
+    def synced_list(self, request):
+        self._client = MongoClient
+        self._name = 'test'
+        self._collection = self._client.test_db.test_list
+        yield MongoDBList(name=self._name, collection=self._collection)
+        self._collection.drop()
+
+    def store(self, data):
+        data_to_insert = {'MongoDBList::name': self._name, 'data': data}
+        self._collection.replace_one({'MongoDBList::name': self._name}, data_to_insert)
diff --git a/tests/test_redis_collection.py b/tests/test_redis_collection.py
new file mode 100644
index 00000000..cc1cca1d
--- /dev/null
+++ b/tests/test_redis_collection.py
@@ -0,0 +1,54 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+import pytest
+import json
+import uuid
+
+from signac.core.collection_redis import RedisDict
+from signac.core.collection_redis import RedisList
+from test_synced_collection import TestJSONDict
+from test_synced_collection import TestJSONList
+
+try:
+    import redis
+    try:
+        # try to connect to server
+        RedisClient = redis.Redis()
+        test_key = str(uuid.uuid4())
+        RedisClient.set(test_key, 0)
+        assert RedisClient.get(test_key) == b'0'  # redis store data as bytes
+        RedisClient.delete(test_key)
+        REDIS = True
+    except (redis.exceptions.ConnectionError, AssertionError):
+        REDIS = False
+except ImportError:
+    REDIS = False
+
+
+@pytest.mark.skipif(not REDIS, reason='test requires the redis package and running redis-server')
+class TestRedisDict(TestJSONDict):
+
+    @pytest.fixture
+    def synced_dict(self, request):
+        self._client = RedisClient
+        request.addfinalizer(self._client.flushall)
+        self._name = 'test'
+        yield RedisDict(name=self._name, client=self._client)
+
+    def store(self, data):
+        self._client.set(self._name, json.dumps(data).encode())
+
+
+@pytest.mark.skipif(not REDIS, reason='test requires the redis package and running redis-server')
+class TestRedisList(TestJSONList):
+
+    @pytest.fixture
+    def synced_list(self, request):
+        self._client = RedisClient
+        request.addfinalizer(self._client.flushall)
+        self._name = 'test'
+        yield RedisList(name=self._name, client=self._client)
+
+    def store(self, data):
+        self._client.set(self._name, json.dumps(data).encode())
diff --git a/tests/test_synced_collection.py b/tests/test_synced_collection.py
index 0792be92..0ba22c43 100644
--- a/tests/test_synced_collection.py
+++ b/tests/test_synced_collection.py
@@ -2,7 +2,6 @@
 # All rights reserved.
 # This software is licensed under the BSD 3-Clause License.
 import pytest
-import uuid
 import os
 import json
 from tempfile import TemporaryDirectory
@@ -11,8 +10,8 @@ from collections.abc import MutableSequence
 from copy import deepcopy
 
 from signac.core.synced_list import SyncedCollection
-from signac.core.jsoncollection import JSONDict
-from signac.core.jsoncollection import JSONList
+from signac.core.collection_json import JSONDict
+from signac.core.collection_json import JSONList
 from signac.errors import InvalidKeyError
 from signac.errors import KeyTypeError
 
@@ -25,31 +24,26 @@ except ImportError:
 FN_JSON = 'test.json'
 
 
-@pytest.fixture
-def testdata():
-    return str(uuid.uuid4())
+class TestJSONCollectionBase:
 
-
-class TestSyncedCollectionBase:
-
-    # this fixture sets temprary directory for tests
+    # this fixture sets temporary directory for tests
     @pytest.fixture(autouse=True)
     def synced_collection(self):
-        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._tmp_dir = TemporaryDirectory(prefix='synced_collection_')
         self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
         yield
         self._tmp_dir.cleanup()
 
-    def test_from_base(self):
+    def test_from_base_json(self):
         sd = SyncedCollection.from_base(filename=self._fn_,
-                                        data={'a': 0}, backend='signac.core.jsoncollection')
+                                        data={'a': 0}, backend='signac.core.collection_json')
         assert isinstance(sd, JSONDict)
         assert 'a' in sd
         assert sd['a'] == 0
 
-        # invalid input
+    def test_from_base_no_backend(self):
         with pytest.raises(ValueError):
-            SyncedCollection.from_base(data={'a': 0}, filename=self._fn_)
+            SyncedCollection.from_base(data={'a': 0})
 
 
 class TestJSONDict:
@@ -279,7 +273,11 @@ class TestJSONDict:
     def test_reopen(self, synced_dict, testdata):
         key = 'reopen'
         synced_dict[key] = testdata
-        synced_dict2 = deepcopy(synced_dict)
+        try:
+            synced_dict2 = deepcopy(synced_dict)
+        except TypeError:
+            # Use fallback implementation, deepcopy not supported by backend.
+            synced_dict2 = synced_dict._pseudo_deepcopy()
         synced_dict.sync()
         del synced_dict  # possibly unsafe
         synced_dict2.load()
@@ -374,7 +372,7 @@ class TestJSONList:
 
     @pytest.fixture
     def synced_list(self):
-        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._tmp_dir = TemporaryDirectory(prefix='jsonlist_')
         self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
         self._backend_kwargs = {'filename': self._fn_, 'write_concern': self._write_concern}
         yield JSONList(**self._backend_kwargs)
@@ -520,14 +518,18 @@ class TestJSONList:
         self.store(data1)
         assert synced_list == data1
 
-        # inavlid data in file
+        # invalid data in file
         data2 = {'a': 1}
         self.store(data2)
         with pytest.raises(ValueError):
             synced_list.load()
 
     def test_reopen(self, synced_list, testdata):
-        synced_list2 = deepcopy(synced_list)
+        try:
+            synced_list2 = deepcopy(synced_list)
+        except TypeError:
+            # Use fallback implementation, deepcopy not supported by backend.
+            synced_list2 = synced_list._pseudo_deepcopy()
         synced_list.append(testdata)
         synced_list.sync()
         del synced_list  # possibly unsafe
diff --git a/tests/test_zarr_collection.py b/tests/test_zarr_collection.py
new file mode 100644
index 00000000..db109b03
--- /dev/null
+++ b/tests/test_zarr_collection.py
@@ -0,0 +1,51 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+import pytest
+from tempfile import TemporaryDirectory
+
+from signac.core.collection_zarr import ZarrDict
+from signac.core.collection_zarr import ZarrList
+from test_synced_collection import TestJSONDict
+from test_synced_collection import TestJSONList
+
+try:
+    import zarr
+    import numcodecs  # zarr depends on numcodecs
+    ZARR = True
+except ImportError:
+    ZARR = False
+
+
+@pytest.mark.skipif(not ZARR, reason='test requires the zarr package')
+class TestZarrDict(TestJSONDict):
+
+    @pytest.fixture(autouse=True)
+    def synced_dict(self):
+        self._tmp_dir = TemporaryDirectory(prefix='zarrdict_')
+        self._group = zarr.group(zarr.DirectoryStore(self._tmp_dir.name))
+        self._name = 'test'
+        yield ZarrDict(name=self._name, group=self._group)
+        self._tmp_dir.cleanup()
+
+    def store(self, data):
+        dataset = self._group.require_dataset(
+            'test', overwrite=True, shape=1, dtype='object', object_codec=numcodecs.JSON())
+        dataset[0] = data
+
+
+@pytest.mark.skipif(not ZARR, reason='test requires the zarr package')
+class TestZarrList(TestJSONList):
+
+    @pytest.fixture(autouse=True)
+    def synced_list(self):
+        self._tmp_dir = TemporaryDirectory(prefix='zarrlist_')
+        self._group = zarr.group(zarr.DirectoryStore(self._tmp_dir.name))
+        self._name = 'test'
+        yield ZarrList(name=self._name, group=self._group)
+        self._tmp_dir.cleanup()
+
+    def store(self, data):
+        dataset = self._group.require_dataset(
+            'test', overwrite=True, shape=1, dtype='object', object_codec=numcodecs.JSON())
+        dataset[0] = data