Repository: glotzerlab/signac
Pull request #363 (2020-07-31 18:57:10)

Added buffering to SyncedCollection
<!-- Provide a general summary of your changes in the Title above -->

## Description
<!-- Describe your changes in detail -->
 Added buffering feature for `SyncedCollection`.
The buffering will be provided by `signac.buffered` and `SyncedCollection.buffered`.
## Motivation and Context
<!-- Why is this change required? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here. -->
Related to #249. This is continuation of work in [PR](https://github.com/vishav1771/signac/pull/2)
## Types of Changes
<!-- Please select all items that apply either now or after creating the pull request: -->
- [ ] Documentation update
- [ ] Bug fix
- [x] New feature
- [ ] Breaking change<sup>1</sup>

<sup>1</sup>The change breaks (or has the potential to break) existing functionality.

## Checklist:
<!-- Please select all items that apply either now or after creating the pull request. -->
<!-- If you are unsure about any of these items, do not hesitate to ask! -->
- [x] I am familiar with the [**Contributing Guidelines**](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md).
- [x] I agree with the terms of the [**Contributor Agreement**](https://github.com/glotzerlab/signac/blob/master/ContributorAgreement.md).
- [x] My name is on the [list of contributors](https://github.com/glotzerlab/signac/blob/master/contributors.yaml).
- [x] My code follows the [code style guideline](https://github.com/glotzerlab/signac/blob/master/CONTRIBUTING.md#code-style) of this project.
- [ ] The changes introduced by this pull request are covered by existing or newly introduced tests.

If necessary:
- [ ] I have updated the API documentation as part of the package doc-strings.
- [ ] I have created a separate pull request to update the [framework documentation](https://docs.signac.io/) on [signac-docs](https://github.com/glotzerlab/signac-docs) and linked it here.
- [ ] I have updated the [changelog](https://github.com/glotzerlab/signac/blob/master/changelog.txt) and added all related issue and pull request numbers for future reference (if applicable). See example below.


Example for a changelog entry: `Fix issue with launching rockets to the moon (#101, #212).`


vishav1771: I have implemented buffering by storing dict to the BUFFER. The current implementation is independent of backend and data-type. The remaining work is to add tests for global buffer.
codecov[bot]: # [Codecov](https://codecov.io/gh/glotzerlab/signac/pull/363?src=pr&el=h1) Report
> Merging [#363](https://codecov.io/gh/glotzerlab/signac/pull/363?src=pr&el=desc) into [feature/synced_collections](https://codecov.io/gh/glotzerlab/signac/commit/7605b6b1def69d28ac97498e5a0382de370cd877&el=desc) will **decrease** coverage by `0.63%`.
> The diff coverage is `56.02%`.

[![Impacted file tree graph](https://codecov.io/gh/glotzerlab/signac/pull/363/graphs/tree.svg?width=650&height=150&src=pr&token=FghbjTyzgM)](https://codecov.io/gh/glotzerlab/signac/pull/363?src=pr&el=tree)

```diff
@@                      Coverage Diff                       @@
##           feature/synced_collections     #363      +/-   ##
==============================================================
- Coverage                       77.28%   76.65%   -0.64%     
==============================================================
  Files                              47       48       +1     
  Lines                            7453     7606     +153     
==============================================================
+ Hits                             5760     5830      +70     
- Misses                           1693     1776      +83     
```


| [Impacted Files](https://codecov.io/gh/glotzerlab/signac/pull/363?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [signac/core/buffered\_collection.py](https://codecov.io/gh/glotzerlab/signac/pull/363/diff?src=pr&el=tree#diff-c2lnbmFjL2NvcmUvYnVmZmVyZWRfY29sbGVjdGlvbi5weQ==) | `47.01% <47.01%> (ø)` | |
| [signac/core/jsoncollection.py](https://codecov.io/gh/glotzerlab/signac/pull/363/diff?src=pr&el=tree#diff-c2lnbmFjL2NvcmUvanNvbmNvbGxlY3Rpb24ucHk=) | `96.22% <91.66%> (-3.78%)` | :arrow_down: |
| [signac/core/synced\_collection.py](https://codecov.io/gh/glotzerlab/signac/pull/363/diff?src=pr&el=tree#diff-c2lnbmFjL2NvcmUvc3luY2VkX2NvbGxlY3Rpb24ucHk=) | `83.15% <100.00%> (-10.11%)` | :arrow_down: |
| [signac/core/syncedattrdict.py](https://codecov.io/gh/glotzerlab/signac/pull/363/diff?src=pr&el=tree#diff-c2lnbmFjL2NvcmUvc3luY2VkYXR0cmRpY3QucHk=) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/glotzerlab/signac/pull/363?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/glotzerlab/signac/pull/363?src=pr&el=footer). Last update [7605b6b...7060709](https://codecov.io/gh/glotzerlab/signac/pull/363?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

csadorf: @vishav1771 Could you provide a brief summary of the changes introduced in this PR? I will do a thorough code review, but for that I need a bit more context. Also, I remember that we had discussed that caching and buffering should happen exclusively on the backend, but while skimming the change set I saw some code that suggest that we are caching individual objects after all, please advise.
vishav1771: @csadorf 
Every object have a pointer to global cache (currently local but will be changed in next commit) as `_cache` and every backend provide `read_from_cache` and `write_from_cache` for caching.
The global buffering logic is implemented in `BufferedSyncedCollection`. It changes the behaviour of `sync` by overwriting `sync_to_backend`. When in the buffer mode it uses the cache to store and load the data and store the backend_kwargs and reference to cache to the buffer that are used while flushing the data.

If a backend inherits from `BufferedSyncedCollection` from it will have buffering feature and if it inherits `SyncedColection` it does not have the buffering feature (I am not completely sure that this is the correct approach).
```
class backend_a(BufferedSyncedCollection):
pass

class backend_b(SyncedCollection):
pass
```
`backend_a` will have buffering feature and `backend_b` will not. 

I try to do the implementation as discussed in the meeting, I think I must have misinterpreted something. Kindly correct me if I am wrong!
csadorf: @vishav1771 I think the design of the buffering layer is good and in line with what he had discussed. We will need to experiment with this to see how that pans out in practice.

Concerning the cache, I know this is roughly in line with the structure that I proposed with the signac2-prototype, but I have since moved away from this idea and would propose to have all caching mechanisms strictly confined to the backend scope. That means a backend may implement caching mechanisms internally or we might provide a caching layer similar to the buffering layer, but there should be no explicit handling of caches within the signac namespace.
vishav1771: @csadorf Since I am using cache to store data while buffered mode. So should be provide buffering to the backend with no caching mechanisms?
csadorf: > @csadorf Since I am using cache to store data while buffered mode. So should be provide buffering to the backend with no caching mechanisms?

I don't think I fully understand your question, but you can use a cache in your buffering implementation, but there should be no explicit caching within the signac namespace outside of the backend implementation.
vishav1771: @csadorf  I have updated the implementation. I have added the `CachedSyncedCollection` class in `caching.py` which adds the caching logic to the `SyncedCollection` and defines `_write_to_cache` and `_read_to_cache` as `abstractmethods` to be implemented by backend. Please review.
csadorf: @vishav1771 I saw you added a few commits, is this still ready for review?
vishav1771: @csadorf Yes, The implementation can be reviewed. I want to be sure it meets the requirements before adding tests.
vyasr: @vishav1771 thanks for the class diagram, this is an easier starting point for discussion before you start modifying the code. With this structure, though, I am confused about two things:
1. Why is there both a `_JSONCache` and a `_JSONBuffer`? How do they differ, and are they both really necessary?
2. Why does the `JSONCollection` have a cache at all? The whole point of introducing the extra `BufferedSyncedCollection` layer into the hierarchy is so that the `JSONCollection` wouldn't have any such logic. I think that the `JSONCollection` in this PR should be modified as little as possible from the upstream, i.e. the `JSONCollection` should look almost the same after this PR. The only changes should be things that are _absolutely required_ to enable buffering in children, and such changes should really be at the `SyncedCollection` level from a conceptual standpoint unless there are implementation specifics that need to be handled by different backend subclasses.
vishav1771: >Why is there both a `_JSONCache` and a `_JSONBuffer`? How do they differ, and are they both really necessary?

`_JSONCache`  stores serialized data of files which is used in caching and buffering and `_JSON_Buffer` store metadata of files which is used while flushing the buffer (maybe name can be changed ).

>Why does the JSONCollection have a cache at all?The whole point of introducing the extra BufferedSyncedCollection layer into the hierarchy is so that the JSONCollection wouldn't have any such logic. I think that the JSONCollection in this PR should be modified as little as possible from the upstream, i.e. the JSONCollection should look almost the same after this PR. The only changes should be things that are absolutely required to enable buffering in children, and such changes should really be at the SyncedCollection level from a conceptual standpoint unless there are implementation specifics that need to be handled by different backend subclasses.

I thought the caching will be added at the `JSONCollection` level and `BufferedJSONCollection` will add buffering to that. But I think both caching and buffering are required to be added in `BufferedJSONCollection`. Is it correct?
vyasr: > > Why is there both a `_JSONCache` and a `_JSONBuffer`? How do they differ, and are they both really necessary?
> 
> `_JSONCache` stores serialized data of files which is used in caching and buffering and `_JSON_Buffer` store metadata of files which is used while flushing the buffer (maybe name can be changed ).
> 
The entire purpose of the metadata is to provide a safety check, right? We just use it to determine whether the file has been modified in memory (i.e. to validate cache integrity). So it's really a function of the cache itself. Whether or not a given caching method provides this error-checking functionality is dependent on the specific cache, and therefore _it is an implementation detail of the cache_. In particular, the `_JSONCache` implements this using this metadata check, but other caches (like a Redis cache) may do so differently (or not at all). That means that such information should be stored within the `_JSONCache` object and used by that object when it tries to flush (to determine whether or not flushing is safe).

> > Why does the JSONCollection have a cache at all? The whole point of introducing the extra BufferedSyncedCollection layer into the hierarchy is so that the JSONCollection wouldn't have any such logic. I think that the JSONCollection in this PR should be modified as little as possible from the upstream, i.e. the JSONCollection should look almost the same after this PR. The only changes should be things that are absolutely required to enable buffering in children, and such changes should really be at the SyncedCollection level from a conceptual standpoint unless there are implementation specifics that need to be handled by different backend subclasses.
> 
> I thought the caching will be added at the `JSONCollection` level and `BufferedJSONCollection` will add buffering to that. But I think both caching and buffering are required to be added in `BufferedJSONCollection`. Is it correct?

It's important to understand the difference between caching and buffering. A cache is nothing more than a data store that is used to improve performance by providing faster read-write access than the primary data store (in this case, a file). Buffering is a means by which we improve our performance by delaying I/O operations. The `BufferedJSONCollection` implements buffering _using the JSONCache_, so the cache is an internal detail. So from a user perspective, the `BufferedJSONCollection` adds buffering, but the caches are completely hidden from the user. Does that make sense? If not, would you want to cache at the `JSONCollection` level?
vyasr: I left a few comments, but I don't have the mental capacity to work through the full design right now. I'll try and review this again soon.
vyasr: > > > Why is there both a `_JSONCache` and a `_JSONBuffer`? How do they differ, and are they both really necessary?
> > 
> > 
> > `_JSONCache` stores serialized data of files which is used in caching and buffering and `_JSON_Buffer` store metadata of files which is used while flushing the buffer (maybe name can be changed ).
> 
> The entire purpose of the metadata is to provide a safety check, right? We just use it to determine whether the file has been modified in memory (i.e. to validate cache integrity). So it's really a function of the cache itself. Whether or not a given caching method provides this error-checking functionality is dependent on the specific cache, and therefore _it is an implementation detail of the cache_. In particular, the `_JSONCache` implements this using this metadata check, but other caches (like a Redis cache) may do so differently (or not at all). That means that such information should be stored within the `_JSONCache` object and used by that object when it tries to flush (to determine whether or not flushing is safe).
> 

@vishav1771 we still need to find a way to refactor this part of the code. It will depend on how we address the buffered collections, but we may end up wanting to extract some helper functions from the JSON buffering logic so that they can be easily reused by other back ends for buffering if necessary.
vishav1771: @vyasr  I have updated the implementation as suggested. But, I have a few doubts:

- I think the main reason to create a different backend `BufferedJSONCollection` than `JSONCollection` was to mimic the behaviour of  `statepoints` (should not support buffering, previously implemented by `JSONCollection`) and `document` (supports buffering, previously implemented by `BufferedJSONCollection` ). But the current implementation will always support buffering.  Should we add a flag `disable_buffering`?

-  

> > > > Why is there both a `_JSONCache` and a `_JSONBuffer`? How do they differ, and are they both really necessary?
> > > 
> > > 
> > > `_JSONCache` stores serialized data of files which is used in caching and buffering and `_JSON_Buffer` store metadata of files which is used while flushing the buffer (maybe name can be changed ).
> > 
> > 
> > The entire purpose of the metadata is to provide a safety check, right? We just use it to determine whether the file has been modified in memory (i.e. to validate cache integrity). So it's really a function of the cache itself. Whether or not a given caching method provides this error-checking functionality is dependent on the specific cache, and therefore _it is an implementation detail of the cache_. In particular, the `_JSONCache` implements this using this metadata check, but other caches (like a Redis cache) may do so differently (or not at all). That means that such information should be stored within the `_JSONCache` object and used by that object when it tries to flush (to determine whether or not flushing is safe).
> 
> @vishav1771 we still need to find a way to refactor this part of the code. It will depend on how we address the buffered collections, but we may end up wanting to extract some helper functions from the JSON buffering logic so that they can be easily reused by other back ends for buffering if necessary.

I tried to add metadata and hashes in the cache. But as Redis-client does not have `popitem` or `__iter__` so wasn't able to implement flush. The simple solution that I could think of was to have a set or list to store filenames but that would be similar to old implementation (storing metadata in `_JSON_Meta`). Please guide.
vyasr: > @vyasr I have updated the implementation as suggested. But, I have a few doubts:
> 
>     * I think the main reason to create a different backend `BufferedJSONCollection` than `JSONCollection` was to mimic the behaviour of  `statepoints` (should not support buffering, previously implemented by `JSONCollection`) and `document` (supports buffering, previously implemented by `BufferedJSONCollection` ). But the current implementation will always support buffering.  Should we add a flag `disable_buffering`?

Yes, good call. When I was conceptualizing this I was only thinking about per-object buffering, which could easily be handled by just avoiding using the `buffered` context manager for any state point objects and only using it for documents. The need for global buffering makes it necessary to add a flag to prevent that being applied to state points. I would use `allow_buffering` rather than `disable_buffering` since I'm anticipating that the relevant checks will look something like `if self._buffered or (_BUFFERED_MODE > 0 and allow_buffering)`, but that's a minor detail.

> 
> 
> > > > > Why is there both a `_JSONCache` and a `_JSONBuffer`? How do they differ, and are they both really necessary?
> > > > 
> > > > 
> > > > `_JSONCache` stores serialized data of files which is used in caching and buffering and `_JSON_Buffer` store metadata of files which is used while flushing the buffer (maybe name can be changed ).
> > > 
> > > 
> > > The entire purpose of the metadata is to provide a safety check, right? We just use it to determine whether the file has been modified in memory (i.e. to validate cache integrity). So it's really a function of the cache itself. Whether or not a given caching method provides this error-checking functionality is dependent on the specific cache, and therefore _it is an implementation detail of the cache_. In particular, the `_JSONCache` implements this using this metadata check, but other caches (like a Redis cache) may do so differently (or not at all). That means that such information should be stored within the `_JSONCache` object and used by that object when it tries to flush (to determine whether or not flushing is safe).
> > 
> > 
> > @vishav1771 we still need to find a way to refactor this part of the code. It will depend on how we address the buffered collections, but we may end up wanting to extract some helper functions from the JSON buffering logic so that they can be easily reused by other back ends for buffering if necessary.
> 
> I tried to add metadata and hashes in the cache. But as Redis-client does not have `popitem` or `__iter__` so wasn't able to implement flush. The simple solution that I could think of was to have a set or list to store filenames but that would be similar to old implementation (storing metadata in `_JSON_Meta`). Please guide.

I don't know the Redis API very well, but it looks like there are comparable methods and you just have to find the right one; I think they avoid implementing certain standards because there can be enormous performance differences for particular use cases depending on your access pattern, so they don't give a default. I'm not 100% sure what the right solution is, but it looks like [`scan_iter`](https://redis-py.readthedocs.io/en/stable/#redis.Redis.scan_iter) is pretty close. I would try that and see how that works.

Moving the buffering inline seems to have simplified things a lot though, which is great! However, I think we can modularize this a bit further. What I'm envisioning is adding another class, say `FileBuffer` (although maybe that's not the best name), which internally stores cache as well as any metadata etc that we use to ensure file integrity. This class would contain most of the logic currently in `_flush_buffer` and `_store_in_buffer` (and maybe a couple other methods that I missed). The reason that I'm suggesting this is that the buffering strategies that we've implemented so far are independent of the type of the data store, so it makes sense to keep these separate. There will be cases where this isn't the case; for instance, with a database back end buffering might entail building up a set of queries to batch; such buffering would be intrinsically tied to the back end (and the underlying back end implementation, for instance pymongo, would be responsible for ensuring database integrity) and would therefore make sense to include directly in the code for a back end. Our current buffering mechanism is just to store an in-memory copy that can be flushed at any future point in time, which is a reusable strategy across back ends and therefore makes sense to isolate into a separate module.
vyasr: @vishav1771 this is getting closer. There are still some code smells that need to get ironed out, but these iterations are slowly clearing them up. Here are some high level thoughts and proposed solutions (where I have them):

1. Inheritance is not the right solution for the `FileBuffer`. A `JSONCollection` is not a buffer, it uses one to store its data when in buffered mode. That means that rather than inheriting, the `JSONCollection` should probably store a (class-level) instance of `FileBuffer` to be used by anything in buffered mode.
2. There are two competing issues here. I would like to isolate the validation logic from the SyncedCollection classes because we're basically providing a uniform interface that works for any file. However, I don't think the buffer should be responsible for file writing, i.e. the `flush` operation is not the responsibility of the buffer. I know this is contrary to my suggestion in my previous message, but in hindsight you can see the issues that arise in your current implementation: making it work requires creating a bidirectional link between the buffer and the back end (which you are currently accomplishing via multiple inheritance, but I am proposing to change to composition in (1)). I'm not quite sure how to resolve this yet. _Maybe_ the right answer is that the `flush` method should accept a `SyncedCollection` and call its write method, but I'm not 100% sure right now. @vishav1771 can you try to think of a solution (or try that one) that meets both of these requirements (keeping the validation logic out of `SyncedCollection` and its subclasses while also preventing the `FileBuffer` from having to know about the associated `SyncedCollection`)? I'd appreciate if you could give it a shot and see what solution seems the cleanest; I'm confident that there should be an appropriate implementation here, but having a hard time coming up with it quickly at the moment.
3. The redis instance creation logic should probably live in the RedisCache initializer. Then `get_cache` becomes as simple as trying to get a RedisCache, and returning a dict if that fails. Do any of the concerns @bdice raised in his last review make this solution unviable?
4. The caching and buffering logic are now simplified enough that putting them in the same file and hiding (i.e. adding underscores to the name) most of them is probably OK. 
vishav1771: >There are two competing issues here. I would like to isolate the validation logic from the SyncedCollection classes because we're basically providing a uniform interface that works for any file. However, I don't think the buffer should be responsible for file writing, i.e. the flush operation is not the responsibility of the buffer. I know this is contrary to my suggestion in my previous message, but in hindsight you can see the issues that arise in your current implementation: making it work requires creating a bidirectional link between the buffer and the back end (which you are currently accomplishing via multiple inheritance, but I am proposing to change to composition in (1)). I'm not quite sure how to resolve this yet. Maybe the right answer is that the flush method should accept a SyncedCollection and call its write method, but I'm not 100% sure right now. @vishav1771 can you try to think of a solution (or try that one) that meets both of these requirements (keeping the validation logic out of SyncedCollection and its subclasses while also preventing the FileBuffer from having to know about the associated SyncedCollection)? I'd appreciate if you could give it a shot and see what solution seems the cleanest; I'm confident that there should be an appropriate implementation here, but having a hard time coming up with it quickly at the moment.

I simple implementation that I could think of is that to add `_popitem_from_buffer` to the buffer which would return `filename` and data (after metacheck). Then `flush` will be implemented by the `backend` using this method. So, buffer will look something like:
```
class FileBuffer:
  _cache = get_cache()

 def _store_to_buffer(self, filename, blob):
  pass

 def _load_data(self, filename): 
  pass

 def _pop_from_buffer(self, filename): 
  pass
   ```
>The caching and buffering logic are now simplified enough that putting them in the same file and hiding (i.e. adding underscores to the name) most of them is probably OK.

Can you please elaborate this part?
vyasr: @vishav1771 that solution seems OK, but it feels awkward that the metadata verification occurs on pop rather than on write; while we can't make the check+write operation truly atomic, it does feel like they should at least be in the same function so that it guarantees safety in the single-threaded case where no other simultaneously running process is modifying the file. Does that make sense? It doesn't feel right that you could pop data from the buffer and have its metadata checked, but then fail to write the data in a separate check at which point the data is already gone from the buffer.

@bdice @csadorf any thoughts on how this division should be properly accomplished? To summarize, we're trying to deal with whether the buffer is responsible for flushing itself, or whether the back end is responsible for flushing the buffer. I feel confident that the answer is the latter as I mentioned in my previous comment, but then the question becomes how to structure the integrity verification so that the "read from buffer, verify, and write" sequence is all basically an atomic operation (within the context of our Python API, i.e. a single function call does all these steps) given that the buffer object shouldn't actually know how to write the data (because that information is specific to the back end). @vishav1771's proposed solution places steps (1) and (2) within the buffer's `_popitem` method, but then step (3) would be performed within the back end, meaning that basically you'd have a `flush` method in the back end that calls `_popitem` until the buffer is empty. I previously proposed something like passing a write method (or the collection itself, which has a write method) to the buffer's `flush` method, but that is effectively the same and not really an improvement. Maybe that is the best we can do without breaking the desired encapsulation of responsibilities? If so we can move forward, but I wanted to check if you had any better ideas.
vyasr: @vishav1771 I had a helpful discussion about the design here with @csadorf this morning that helped me to clarify the best path forward, it was very useful to get a full second opinion on the design since I've been in the weeds of it for a while. I think a significant reason for the problems that we've been having is that while the _caching_ logic (which relates to having a separate object in which we can store data faster than syncing to the back end) should be completely independent of the rest of the `SyncedCollection` framework, the _buffering_ logic (which describes how a particular back end avoids I/O) cannot be completely decoupled from the back end. The nature of the `FileBuffer` class that you've currently implemented clearly reveals this problem: there's no good way to implement what I was asking for, which is a buffer with the ability to validate and flush _without_ knowing anything about the back end. That means that what I suggested in my last comment is wrong: *for the buffer (not the cache) inheritance is the correct choice, not composition*.

In light of this, the solution is to implement a hybrid of two different ideas that we've considered. [In one of your comments above](https://github.com/glotzerlab/signac/pull/363#issuecomment-678697077), I asked you to make a class diagram back when we had a separate inheritance hierarchy for buffered and non-buffered classes (e.g. `JSONCollection` and `BufferedJSONCollection`. I suggested this because I was worried about complicating the class hierarchy, but also because the way that the `buffered` context manager and all the buffer flushing worked at the time was very confusing. I think that now that we've resolved the latter concerns, *adding to the class hierarchy is the correct approach*. What I am suggesting is to basically move the logic in your `FileBuffer` class back into something like a `BufferedSyncedCollection` class. However, the reason that I suggested creating the `FileBuffer` class was to abstract out the standard buffering mechanism for any file-based back end, which needs to be separate from the interface provided by an arbitrary buffered back end. Therefore, what we want is a `BufferedSyncedCollection` that just defines the API for any buffered collection, and then a `FileBufferedSyncedCollection` that implements the relevant methods and can be used by any back end whose underlying storage is file-based. I realize that this will complicate the inheritance hierarchy substantially; however, I think it has substantial benefits that outweight this complexity: 1) every class has a clear purpose, 2) almost no code duplication would be required, and 3) the vast majority of classes will require almost no code, nearly all the logic will simply be inherited. Once we have this implemented, we can reassess the complexity, but I think doing it this way will resolve almost all of our other concerns.

Since I think my proposal requires more explanation than a class diagram, I've written up a [skeleton Python file](https://github.com/glotzerlab/signac/files/5424020/api.txt) (saved as a text file because Github doesn't support .py files...) that contains all of the relevant methods that each class should have along with docstrings explaining their purpose. In cases where I thought it would be helpful I included implementations (largely copied from existing code), but in most cases I only described how things should behave without writing the code. I've omitted any methods that aren't directly relevant to the buffering questions, hopefully that doesn't cause you any confusion. Have a look at the file and let me know if anything doesn't make sense, or if you foresee any reasons why this suggestion would cause problems.

Apologies for having you change this back and forth multiple times. I think it's very important that we get the abstraction layers here right, and it's clearly a tricky problem. Each iteration has helped bring us closer, and I am pretty confident that this solution will get us almost there!
vyasr: @bdice @csadorf since you reviewed earlier versions of this PR I would appreciate if you could have a look again. @vishav1771 if you have time your review would also be great! I have restructured the code substantially and am much happier with the overall design now. I've also added quite a few additional tests. Some notes that will be helpful while you review:

- The outstanding test failure looks to be related to #431, so I don't think there's anything to be done about that in this PR.
- There is a long laundry list of additional tasks I'd like to complete that I collected while going through this PR. However, since getting buffering just right and as bug-free as possible is already a nontrivial task, I've avoided making unrelated changes wherever possible. If as you read through this PR you identify other issues that you would like addressed, please include in your comment whether you think it's relevant for this PR or if you think it would be appropriate to do in a follow-up.
- I have attempted to add descriptive docstrings everywhere, so feel free to comment if you think that the contents are insufficient. However, I have not yet attempted to document all parameters etc everywhere. My immediate next step after this PR is completed will be merging master into feature/synced_collection, which will also involve reformatting everything again with black/isort/etc and then applying numpy-based pydocstyle. As long as the existing documentation is enough for reviewers to understand the code, I'd like to wait until that PR to do a comprehensive review and update of docstrings.
- A few of the tests have TODO comments attached to them. I've tried to make these comments very descriptive of the tests and why I think the behavior there requires further discussion. I'd appreciate explicit feedback on how you think those should behave. I think we need to resolve those one way or another before completing this PR; I don't think that we necessarily need to _support_ all the types of behavior I'm testing, but I definitely want to avoid any undefined behavior, so we should be clear on which of those cases we expect to raise exceptions if that's how we think they should work.
vyasr: > @vyasr you mentioned "There is a long laundry list of additional tasks I'd like to complete" -- is that a list you can share in an issue or comment on this PR?

I'd rather not directly paste in since it's messy and very long and not well curated, so I'll summarize the major next steps I plan before actually implementing the remaining features discussed in #249:
- Reorganize (and possibly rename) files so that all SyncedCollection files are in a separate subpackage. 
- Reduce public API and rename some methods (e.g. sync->save). There are methods that are currently exposed that I think should be internal, but that requires touching many different collection files and I didn't want to clutter this PR.
- Unify and add more tests for different backends + buffering. 
- Fix various minor usability issues. For instance, to_base does not load first, which is undesirable for an API function, but we can't currently change it because it's used internally everywhere to convert the in-memory representation _without_ overwriting it from file first. For another, the complex multiple inheritance patterns I've introduced make the constructors cumbersome to call (in particular, positional arguments are broken). Addressing this again requires modifying all collections, because unfortunately multiple inheritance + super means that the classes have to be written with a little care to support that behavior.
- Simplify some internals like `_update` and registration.
There are other items but mostly smaller. There are also some very obvious optimizations that are worthwhile enough that I wouldn't consider them premature, for instance right now I expect validation to be a huge bottleneck because it's run every single time the collection loads for all the data rather than only for things that are new/changed.
vyasr: @bdice I've addressed all of your comments that don't require further discussion, so this is ready for another round of review.

@bdice @csadorf in addition to the TODO items listed in the tests (Simon, Bradley's conversations on those are still open, feel free to add directly to those) I've identified one other issue that was covered up by a bug in my previous implementation that Bradley found. I had originally thought this might be a problem and had included an internal assert to protect against it, but since my bug hid the issue I must have deleted it. Have a look at `FileBufferedSyncedCollection._sync_buffer` as well as `FileBufferedSyncedCollection._initialize_data_in_cache`.


vyasr: > Approving, with merge conditional on this PR receiving a second approval. I can do one more round of review on this PR (or just help reply to / resolve comments) but I think it's almost time to merge it and continue work on TODOs/design iterations in a new PR.
> 
> Looks like a test is failing with `SyncedList.reset`. I didn't notice that when I approved the PR. 😕

See my last post in the overall conversation replying to you. That error also requires some thought, the relevant places to look are `FileBufferedSyncedCollection._sync_buffer` and `FileBufferedSyncedCollection._initialize_data_in_cache`
bdice: @vyasr: "I'm going to make a class diagram for this."

![meme](https://i.kym-cdn.com/photos/images/original/001/430/367/007.gif)
vyasr: @csadorf there are a few outstanding items marked as TODO in the code with some long comments describing the exact outstanding issues. When you get a chance to review, please do comment on those (there are ongoing threads with @bdice for most of them that you can add to directly).
vyasr: I'm going to go ahead and merge this as is. We can make further edits in future PRs since all of this work is going into a feature branch.

Patch:
diff --git a/signac/core/buffered_synced_collection.py b/signac/core/buffered_synced_collection.py
new file mode 100644
index 00000000..e97df54c
--- /dev/null
+++ b/signac/core/buffered_synced_collection.py
@@ -0,0 +1,198 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implement the BufferedCollection class.
+
+The BufferedCollection extends the abstract SyncedCollection to support
+transparent buffering of the synchronization process. This feature allows
+client code to indicate to the collection when it is safe to buffer reads and
+writes, which usually means guaranteeing that the synchronization destination
+(e.g. an underlying file or database entry) will not be modified by other
+processes concurrently with the set of operations within the buffered block.
+Judicious use of buffering can dramatically speed up code paths that might
+otherwise involve, for instance, heavy I/O. The specific buffering mechanism
+must be implemented by each backend since it depends on the nature of the
+underlying data format.
+"""
+import logging
+from typing import List, Any
+
+from contextlib import contextmanager
+
+from .errors import BufferedError
+from .synced_collection import SyncedCollection
+
+logger = logging.getLogger(__name__)
+
+_BUFFERED_MODE = 0
+_BUFFERED_BACKENDS: List[Any] = list()
+
+
+@contextmanager
+def buffer_reads_writes():
+    """Enter a global buffer mode for all BufferedCollection instances.
+
+    All future write operations are written to the buffer, read
+    operations are performed from the buffer whenever possible.
+
+    All write operations are deferred until the flush_all() function
+    is called, the buffer overflows, or upon exiting the buffer mode.
+
+    Raises
+    ------
+    BufferException
+    """
+    global _BUFFERED_MODE
+    assert _BUFFERED_MODE >= 0
+
+    _BUFFERED_MODE += 1
+    try:
+        yield
+    finally:
+        _BUFFERED_MODE -= 1
+        if _BUFFERED_MODE == 0:
+            _flush_all_backends()
+
+
+def _flush_all_backends():
+    """Execute all deferred write operations.
+
+    Raises
+    ------
+    BufferedFileError
+    """
+    global _BUFFERED_BACKENDS
+    logger.debug("Flushing buffer...")
+    issues = {}
+    for backend in _BUFFERED_BACKENDS:
+        try:
+            # try to sync the data to backend
+            issue = backend._flush_buffer()
+            issues.update(issue)
+        except OSError as error:
+            logger.error(str(error))
+            issues[backend] = error
+    if issues:
+        raise BufferedError(issues)
+
+
+class BufferedCollection(SyncedCollection):
+    """Adds buffering capabilities to the SyncedCollection.
+
+    *The default behavior is no buffering.* The class simply defines an
+    appropriate interface for buffering behavior so that client code can rely
+    on these methods existing, e.g. to be able to do things like
+    `with collection.buffered...`. However, in the default case the result of
+    this will be a no-op.
+
+    In addition to the methods shown here, we may also require methods to flush
+    a specific entry out of the buffer (e.g. _flush(self, item)) which has been
+    implemented in the current version of the PR. I've left out such details
+    since they're not crucial to getting the right structure here.
+    """
+
+    def __init__(self, *args, **kwargs):
+        # The `_buffered` attribute _must_ be defined prior to calling the
+        # superclass constructors in order to enable subclasses to override
+        # setattr and getattr in nontrivial ways. In particular, if setattr and
+        # getattr need to access the synced data, they may call sync and load,
+        # which depend on this parameter existing and could otherwise end up in
+        # an infinite recursion.
+        self._buffered = 0
+        super().__init__(*args, **kwargs)
+
+    @classmethod
+    def __init_subclass__(cls):
+        """Register subclasses for the purpose of global buffering.
+
+        Each subclass has its own means of buffering and must be flushed.
+        """
+        super().__init_subclass__()
+        global _BUFFERED_BACKENDS
+        _BUFFERED_BACKENDS.append(cls)
+
+    # We would like to be able to override `_sync` and `_load` rather than
+    # `sync` and `load` to avoid having to replicate the "parent" logic.
+    # However, the underscore methods are the hooks for backend-specific
+    # collections to define how synchronization with the backend behaves. What
+    # we need to control in this class is when the synchronization to the
+    # backend actually occurs and when data is simply written to a buffer, and
+    # the only way to unambiguously specific the methods to call is by
+    # overriding `sync` and `load`.
+    def sync(self):
+        """Synchronize data with the backend but buffer if needed.
+
+        This method is identical to the SyncedCollection implementation for
+        `sync` except that it determines whether data is actually synchronized
+        or instead written to a temporary buffer based on the buffering mode.
+        """
+        if self._suspend_sync_ <= 0:
+            if self._parent is None:
+                if self._is_buffered:
+                    self._sync_buffer()
+                else:
+                    self._sync()
+            else:
+                self._parent.sync()
+
+    def load(self):
+        """Load data from the backend but buffer if needed.
+
+        This method is identical to the SyncedCollection implementation for
+        `load` except that it determines whether data is actually synchronized
+        or instead read from a temporary buffer based on the buffering mode.
+        """
+        if self._suspend_sync_ <= 0:
+            if self._parent is None:
+                if self._is_buffered:
+                    data = self._load_buffer()
+                else:
+                    data = self._load()
+                with self._suspend_sync():
+                    self._update(data)
+            else:
+                self._parent.load()
+
+    def _sync_buffer(self):
+        """Store data in buffer.
+
+        There's little benefit to providing a default means of buffering across
+        all backends since the process is heavily dependent on the backend
+        data store, so the default behavior is to just sync normally.
+        """
+        self._sync()
+
+    def _load_buffer(self):
+        """Store data in buffer.
+
+        There's little benefit to providing a default means of buffering across
+        all backends since the process is heavily dependent on the backend
+        data store, so the default behavior is to just load normally.
+        """
+        self._load()
+
+    @contextmanager
+    def buffered(self):
+        """Enter buffered mode."""
+        self._buffered += 1
+        try:
+            yield
+        finally:
+            self._buffered -= 1
+            if not self._is_buffered:
+                self._flush()
+
+    @property
+    def _is_buffered(self):
+        """Check if we should write to the buffer or not."""
+        global _BUFFERED_MODE
+        return self._buffered + _BUFFERED_MODE > 0
+
+    def _flush(self):
+        """Flush data associated with this instance from the buffer."""
+        pass
+
+    @classmethod
+    def _flush_buffer(self):
+        """Flush all data in the buffer."""
+        pass
diff --git a/signac/core/caching.py b/signac/core/caching.py
new file mode 100644
index 00000000..82cd2346
--- /dev/null
+++ b/signac/core/caching.py
@@ -0,0 +1,79 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implement the caching feature to SyncedCollection API."""
+import uuid
+import logging
+import pickle
+from collections.abc import MutableMapping
+
+logger = logging.getLogger(__name__)
+
+
+def get_cache():
+    """Return the cache.
+
+    This method returns an instance of :class:`~RedisCache` if a Redis server
+    is available, or otherwise an instance of :class:`dict` for an in-memory
+    cache.
+
+    Returns
+    -------
+    cache
+        An instance of :class:`~_RedisCache` if redis-server is available,
+        otherwise a dict.
+    """
+    try:
+        import redis
+
+        REDIS = True
+    except ImportError as error:
+        logger.debug(str(error))
+        REDIS = False
+    if REDIS:
+        try:
+            # try to connect to server
+            cache = redis.Redis()
+            test_key = str(uuid.uuid4())
+            cache.set(test_key, 0)
+            assert cache.get(test_key) == b"0"  # Redis stores data as bytes
+            cache.delete(test_key)
+            logger.info("Using Redis cache.")
+            return _RedisCache(cache)
+        except (redis.exceptions.ConnectionError, AssertionError) as error:
+            logger.debug(str(error))
+    logger.info("Redis not available.")
+    return {}
+
+
+class _RedisCache(MutableMapping):
+    """Redis-based cache.
+
+    Redis restricts the types of data it can handle to bytes, strings, or
+    numbers, and it always returns responses as bytes. The RedisCache is a
+    :class:`~collections.abc.MutableMapping` that provides a convenient wrapper
+    around instances of :class:`redis.Redis`, handling conversions to and from
+    the appropriate data types.
+    """
+
+    def __init__(self, client):
+        self._client = client
+
+    def __setitem__(self, key, value):
+        self._client[key] = pickle.dumps(value)
+
+    def __getitem__(self, key):
+        return pickle.loads(self._client[key])
+
+    def __delitem__(self, key):
+        self._client.delete(key)
+
+    def __contains__(self, key):
+        return key in self._client
+
+    def __iter__(self):
+        for key in self._client.keys():
+            yield key.decode()
+
+    def __len__(self):
+        return len(self._client.keys())
diff --git a/signac/core/collection_json.py b/signac/core/collection_json.py
index e1d70e4f..d6a72b48 100644
--- a/signac/core/collection_json.py
+++ b/signac/core/collection_json.py
@@ -13,6 +13,7 @@ import errno
 import uuid
 import warnings
 
+from .file_buffered_synced_collection import FileBufferedCollection
 from .synced_collection import SyncedCollection
 from .syncedattrdict import SyncedAttrDict
 from .synced_list import SyncedList
@@ -49,7 +50,7 @@ class JSONCollection(SyncedCollection):
         self._filename = None if filename is None else os.path.realpath(filename)
         self._write_concern = write_concern
         kwargs['name'] = filename
-        super().__init__(**kwargs)
+        super().__init__(filename=filename, **kwargs)
 
     def _load(self):
         """Load the data from a JSON file."""
@@ -85,6 +86,15 @@ class JSONCollection(SyncedCollection):
 JSONCollection.add_validator(json_format_validator)
 
 
+class BufferedJSONCollection(JSONCollection, FileBufferedCollection):
+    """A JSONCollection with buffering enabled."""
+
+    _backend = __name__ + '.buffered'  # type: ignore
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+
+
 class JSONDict(JSONCollection, SyncedAttrDict):
     """A dict-like mapping interface to a persistent JSON file.
 
@@ -169,4 +179,14 @@ class JSONList(JSONCollection, SyncedList):
     """
 
 
-SyncedCollection.register(JSONDict, JSONList)
+class BufferedJSONDict(BufferedJSONCollection, SyncedAttrDict):
+    """A buffered JSONDict."""
+    _PROTECTED_KEYS = SyncedAttrDict._PROTECTED_KEYS + (
+        '_filename', '_buffered', '_is_buffered')
+
+
+class BufferedJSONList(BufferedJSONCollection, SyncedList):
+    """A buffered JSONList."""
+
+
+SyncedCollection.register(JSONDict, JSONList, BufferedJSONDict, BufferedJSONList)
diff --git a/signac/core/errors.py b/signac/core/errors.py
index b422d2ce..8566b6f9 100644
--- a/signac/core/errors.py
+++ b/signac/core/errors.py
@@ -5,3 +5,40 @@
 
 class Error(Exception):
     pass
+
+
+class BufferException(Error):
+    """An exception occured in buffered mode."""
+
+
+class BufferedError(BufferException):
+    """Raised when an error occured while flushing one or more buffered files.
+
+    Attribute
+    ---------
+    files:
+        A dictionary of names that caused issues during the flush operation,
+        mapped to a possible reason for the issue or None in case that it
+        cannot be determined.
+    """
+
+    def __init__(self, files):
+        self.files = files
+
+    def __str__(self):
+        return "{}({})".format(type(self).__name__, self.files)
+
+
+class MetadataError(BufferException):
+    """Raised when metadata check fails.
+
+    The contents of this file in the buffer can be accessed via the
+    `buffer_contents` attribute of this exception.
+    """
+
+    def __init__(self, filename, contents):
+        self.filename = filename
+        self.buffer_contents = contents
+
+    def __str__(self):
+        return f'{self.filename} appears to have been externally modified.'
diff --git a/signac/core/file_buffered_synced_collection.py b/signac/core/file_buffered_synced_collection.py
new file mode 100644
index 00000000..ed9e3c0c
--- /dev/null
+++ b/signac/core/file_buffered_synced_collection.py
@@ -0,0 +1,297 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+"""Implement the FileBufferedSyncedCollection class.
+
+The FileBufferedSyncedCollection is a concrete implementation of the buffer
+protocol established by the BufferedSyncedCollection class. It uses an
+in-memory cache to store data when in buffered mode. It is suitable for
+use with any file-based backend because it performs integrity checks based on
+whether or not the underlying file has been modified while buffering was
+activated.
+"""
+
+import errno
+import hashlib
+import json
+import os
+import platform
+import sys
+
+from typing import Dict, Tuple, Union
+
+from .buffered_synced_collection import BufferedCollection
+from .errors import MetadataError
+
+PYPY = 'PyPy' in platform.python_implementation()
+
+
+class FileBufferedCollection(BufferedCollection):
+    """Implement buffering for SyncedCollections with file-based backends.
+
+    All file-based backends can use the same set of integrity checks prior to a
+    buffer flush. This class standardizes that protocol. This class also
+    centralizes the storage of buffered data, i.e. all subclasses share a
+    single cache. This choice is so that that users can reliably get and set
+    the buffer capacity without worrying about the number of distinct internal
+    data buffers that might be present.
+    """
+
+    _cache: Dict[str, Dict[str, Union[bytes, str, Tuple[int, float]]]] = {}
+    _cached_collections: Dict[int, BufferedCollection] = {}
+    _BUFFER_CAPACITY = 32 * 2 ** 20  # 32 MB
+    _CURRENT_BUFFER_SIZE = 0
+
+    def __init__(self, filename, *args, **kwargs):
+        if PYPY:
+            raise NotImplementedError(
+                "File-based buffering is not supported on PyPy.")
+        super().__init__(*args, **kwargs)
+        self._filename = filename
+
+    @staticmethod
+    def _hash(blob):
+        """Calculate and return the md5 hash value for the file data.
+
+        Parameters
+        ----------
+        blob : bytes
+            Byte literal to be hashed.
+
+        Returns
+        -------
+        str
+            The md5 hash of the input bytes.
+        """
+        if blob is not None:
+            m = hashlib.md5()
+            m.update(blob)
+            return m.hexdigest()
+
+    def _get_file_metadata(self):
+        """Return metadata of file.
+
+        Returns
+        -------
+        Tuple[int, float] or None
+            The size and last modification time of the associated file. If the
+            file does not exist, returns :code:`None`.
+        """
+        try:
+            metadata = os.stat(self._filename)
+            return metadata.st_size, metadata.st_mtime_ns
+        except OSError as error:
+            if error.errno != errno.ENOENT:
+                raise
+            # A return value of None indicates that the file does not
+            # exist. Since any non-``None`` value will return `False` when
+            # compared to ``None``, returning ``None`` provides a
+            # reasonable value to compare against for metadata-based
+            # validation checks.
+            return None
+
+    @staticmethod
+    def get_buffer_capacity():
+        """Get the current buffer capacity."""
+        return FileBufferedCollection._BUFFER_CAPACITY
+
+    @staticmethod
+    def set_buffer_capacity(new_capacity):
+        """Update the buffer capacity."""
+        FileBufferedCollection._BUFFER_CAPACITY = new_capacity
+        if new_capacity < FileBufferedCollection._CURRENT_BUFFER_SIZE:
+            FileBufferedCollection._flush_buffer()
+
+    @staticmethod
+    def get_current_buffer_size():
+        """Get the total amount of data currently stored in the buffer."""
+        return FileBufferedCollection._CURRENT_BUFFER_SIZE
+
+    def _flush(self, force=False):
+        """Save buffered changes to the underlying file.
+
+        Parameters
+        ----------
+        force : bool
+            If True, force a flush even in buffered mode (defaults to False).
+        """
+        if not self._is_buffered or force:
+            try:
+                cached_data = self._cache[self._filename]
+            except KeyError:
+                # There are valid reasons for nothing to be in the cache (the
+                # object was never actually accessed during global buffering,
+                # multiple collections pointing to the same file, etc).
+                pass
+            else:
+                blob = self._encode(self.to_base())
+
+                # If the contents have not been changed since the initial read,
+                # we don't need to rewrite it.
+                try:
+                    if self._hash(blob) != cached_data["hash"]:
+                        # Validate that the file hasn't been changed by
+                        # something else.
+                        if cached_data["metadata"] != self._get_file_metadata():
+                            raise MetadataError(self._filename,
+                                                cached_data['contents'])
+                        self._data = self._decode(cached_data["contents"])
+                        self._sync()
+                finally:
+                    # Whether or not an error was raised, the cache must be
+                    # cleared to ensure a valid final buffer state.
+                    del self._cache[self._filename]
+                    data_size = sys.getsizeof(cached_data)
+                    FileBufferedCollection._CURRENT_BUFFER_SIZE -= data_size
+
+    @staticmethod
+    def _encode(data):
+        """Encode the data into a serializable form.
+
+        This method assumes JSON-serializable data, but subclasses can override
+        this hook method to change the encoding behavior as needed.
+
+        Returns
+        -------
+        bytes
+            The underlying encoded data.
+        """
+        return json.dumps(data).encode()
+
+    @staticmethod
+    def _decode(blob):
+        """Decode serialized data.
+
+        This method assumes JSON-serializable data, but subclasses can override
+        this hook method to change the encoding behavior as needed.
+
+        Parameters
+        ----------
+        blob : bytes
+            Byte literal to be decoded.
+        """
+        return json.loads(blob.decode())
+
+    def _sync_buffer(self):
+        """Store data in buffer.
+
+        We can reasonably provide a default implementation for all file-based
+        backends that simply entails storing data to an in-memory cache (which
+        could also be a Redis instance, etc).
+        """
+        if self._filename in self._cache:
+            blob = self._encode(self.to_base())
+            cached_data = self._cache[self._filename]
+            buffer_size_change = sys.getsizeof(blob) - sys.getsizeof(
+                cached_data["contents"]
+            )
+            FileBufferedCollection._CURRENT_BUFFER_SIZE += buffer_size_change
+            cached_data["contents"] = blob
+        else:
+            # The only methods that could safely call sync without a load are
+            # destructive operations like `reset` or `clear` that completely
+            # wipe out previously existing data. Therefore, the safest choice
+            # for ensuring consistency of the buffer is to modify the stored
+            # hash (which is used for the consistency check) with the hash of
+            # the current data on disk. _initialize_data_in_cache always uses
+            # the current metadata, so the only extra work here is to modify
+            # the hash after it's called (since it uses self.to_base()) to get
+            # the data to initialize the cache with.
+            self._initialize_data_in_cache()
+            disk_data = self._load()
+            self._cache[self._filename]["hash"] = self._hash(self._encode(disk_data))
+
+        if (
+            FileBufferedCollection._CURRENT_BUFFER_SIZE
+            > FileBufferedCollection._BUFFER_CAPACITY
+        ):
+            FileBufferedCollection._flush_buffer(force=True)
+        # If multiple collections point to the same data, just checking that
+        # the file contents are cached is not a sufficient check.
+        if id(self) not in self._cached_collections:
+            self._cached_collections[id(self)] = self
+
+    def _load_buffer(self):
+        """Read data from buffer.
+
+        We can reasonably provide a default implementation for all file-based
+        backends that simply entails reading data from an in-memory cache
+        (which could also be a Redis instance, etc).
+
+        Returns
+        -------
+        Collection
+            A collection of the same base type as the SyncedCollection this
+            method is called for, corresponding to data loaded from the
+            underlying file.
+        """
+        if self._filename in self._cache:
+            # If multiple collections point to the same data, just checking
+            # that the file contents are cached is not a sufficient check.
+            if id(self) not in self._cached_collections:
+                self._cached_collections[id(self)] = self
+        else:
+            self._initialize_data_in_cache()
+
+        # Load from buffer
+        blob = self._cache[self._filename]["contents"]
+
+        if (
+            FileBufferedCollection._CURRENT_BUFFER_SIZE
+            > FileBufferedCollection._BUFFER_CAPACITY
+        ):
+            FileBufferedCollection._flush_buffer(force=True)
+        return self._decode(blob)
+
+    def _initialize_data_in_cache(self):
+        """Create the initial entry for the data in the cache.
+
+        This method always populates the cache with the encoded contents of the
+        provided data, the hash of said data, and the current metadata of the
+        file on disk.
+        """
+        blob = self._encode(self.to_base())
+        metadata = self._get_file_metadata()
+
+        self._cache[self._filename] = {
+            "contents": blob,
+            "hash": self._hash(blob),
+            "metadata": metadata,
+        }
+        FileBufferedCollection._CURRENT_BUFFER_SIZE += sys.getsizeof(
+            self._cache[self._filename]
+        )
+        self._cached_collections[id(self)] = self
+
+    @classmethod
+    def _flush_buffer(cls, force=False):
+        """Flush the data in the file buffer.
+
+        Returns
+        -------
+        dict
+            Mapping of filename and errors occured during flushing data.
+        """
+        # All subclasses share a single cache rather than having separate
+        # caches for each instance, so we can exit early in subclasses.
+        if cls != FileBufferedCollection:
+            return {}
+
+        issues = {}
+
+        # We need to use the list of buffered objects rather than directly
+        # looping over the local cache so that each collection can
+        # independently decide whether or not to flush based on whether it's
+        # still buffered (if buffered contexts are nested).
+        remaining_collections = {}
+        while FileBufferedCollection._cached_collections:
+            col_id, collection = FileBufferedCollection._cached_collections.popitem()
+            if collection._is_buffered and not force:
+                remaining_collections[col_id] = collection
+                continue
+            try:
+                collection._flush(force=force)
+            except (OSError, MetadataError) as err:
+                issues[collection._filename] = err
+        FileBufferedCollection._cached_collections = remaining_collections
+        return issues
diff --git a/signac/core/synced_collection.py b/signac/core/synced_collection.py
index b9f6f0e3..9e90078a 100644
--- a/signac/core/synced_collection.py
+++ b/signac/core/synced_collection.py
@@ -8,15 +8,16 @@ These features are implemented in different subclasses which enable us to use a
 backend with different data-structures or vice-versa. It declares as abstract
 methods the methods that must be implemented by any subclass to match the API.
 """
-
-from typing import List, Callable
+from typing import List, Callable, DefaultDict, Any
 from contextlib import contextmanager
 from abc import abstractmethod
 from collections import defaultdict
 from collections.abc import Collection
 
+
 try:
     import numpy
+
     NUMPY = True
 except ImportError:
     NUMPY = False
@@ -31,9 +32,10 @@ class SyncedCollection(Collection):
     """
 
     _backend = None
+    registry: DefaultDict[str, List[Any]] = defaultdict(list)
     _validators: List[Callable] = []
 
-    def __init__(self, name=None, parent=None):
+    def __init__(self, name=None, parent=None, *args, **kwargs):
         self._data = None
         self._parent = parent
         self._name = name
@@ -41,7 +43,8 @@ class SyncedCollection(Collection):
         if (name is None) == (parent is None):
             raise ValueError(
                 "Illegal argument combination, one of the two arguments, "
-                "parent or name must be None, but not both.")
+                "parent or name must be None, but not both."
+            )
 
     @classmethod
     def __init_subclass__(cls):
@@ -69,8 +72,6 @@ class SyncedCollection(Collection):
         \*args
             Classes to register
         """
-        if not hasattr(cls, 'registry'):
-            cls.registry = defaultdict(list)
         for base_cls in args:
             cls.registry[base_cls._backend].append(base_cls)
 
@@ -80,8 +81,10 @@ class SyncedCollection(Collection):
         validators = []
         # Classes inherit the validators of their parent classes.
         for base_cls in type(self).__mro__:
-            if hasattr(base_cls, '_validators'):
-                validators.extend([v for v in base_cls._validators if v not in validators])
+            if hasattr(base_cls, "_validators"):
+                validators.extend(
+                    [v for v in base_cls._validators if v not in validators]
+                )
         return validators
 
     @classmethod
@@ -177,11 +180,13 @@ class SyncedCollection(Collection):
 
     # The following methods share a common implementation for
     # all data structures and regardless of backend.
+
     def __getitem__(self, key):
         self.load()
         return self._data[key]
 
     def __delitem__(self, item):
+        self.load()
         del self._data[item]
         self.sync()
 
@@ -198,6 +203,7 @@ class SyncedCollection(Collection):
         return self.to_base()
 
     def __eq__(self, other):
+        self.load()
         if isinstance(other, type(self)):
             return self() == other()
         else:
diff --git a/signac/core/syncedattrdict.py b/signac/core/syncedattrdict.py
index 88fea769..75a58c1f 100644
--- a/signac/core/syncedattrdict.py
+++ b/signac/core/syncedattrdict.py
@@ -11,6 +11,7 @@ named by keys, including nested keys.
 
 from collections.abc import Mapping
 from collections.abc import MutableMapping
+from typing import Tuple
 
 from .synced_collection import SyncedCollection
 from .validators import no_dot_in_key
@@ -34,8 +35,11 @@ class SyncedAttrDict(SyncedCollection, MutableMapping):
         :class:`dict` representation, and if necessary construct a new SyncedAttrDict.
     """
 
-    _PROTECTED_KEYS = ('_data', '_name', '_suspend_sync_', '_load', '_sync', '_parent',
-                       '_validators')
+    # Must specify this as a variable length tuple to allow subclasses to
+    # extend the list of protected keys.
+    _PROTECTED_KEYS: Tuple[str, ...] = (
+        '_data', '_name', '_suspend_sync_', '_load', '_sync', '_parent',
+        '_validators')
 
     def __init__(self, data=None, **kwargs):
         super().__init__(**kwargs)
@@ -173,7 +177,6 @@ class SyncedAttrDict(SyncedCollection, MutableMapping):
         return ret
 
     def clear(self):
-        self.load()
         self._data = {}
         self.sync()
 


Test patch:
diff --git a/tests/test_buffered_synced_collection.py b/tests/test_buffered_synced_collection.py
new file mode 100644
index 00000000..b54c9ead
--- /dev/null
+++ b/tests/test_buffered_synced_collection.py
@@ -0,0 +1,420 @@
+# Copyright (c) 2020 The Regents of the University of Michigan
+# All rights reserved.
+# This software is licensed under the BSD 3-Clause License.
+import pytest
+import os
+import json
+from tempfile import TemporaryDirectory
+import time
+import itertools
+import platform
+
+from signac.core.synced_collection import SyncedCollection
+from signac.core.collection_json import BufferedJSONDict
+from signac.core.collection_json import BufferedJSONList
+from signac.core.buffered_synced_collection import buffer_reads_writes
+from signac.core.errors import MetadataError, BufferedError
+
+from test_synced_collection import TestJSONDict, TestJSONList
+
+FN_JSON = 'test.json'
+
+
+PYPY = 'PyPy' in platform.python_implementation()
+
+
+@pytest.mark.skipif(PYPY, reason="Buffered mode not supported for PyPy.")
+class TestJSONCollectionBase:
+
+    # this fixture sets temporary directory for tests
+    @pytest.fixture(autouse=True)
+    def synced_collection(self):
+        self._tmp_dir = TemporaryDirectory(prefix='synced_collection_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        yield
+        self._tmp_dir.cleanup()
+
+    def test_from_base_json(self):
+        sd = SyncedCollection.from_base(
+            filename=self._fn_,
+            data={'a': 0}, backend='signac.core.collection_json.buffered')
+        assert isinstance(sd, BufferedJSONDict)
+        assert 'a' in sd
+        assert sd['a'] == 0
+
+    def test_from_base_no_backend(self):
+        with pytest.raises(ValueError):
+            SyncedCollection.from_base(data={'a': 0})
+
+
+@pytest.mark.skipif(PYPY, reason="Buffered mode not supported for PyPy.")
+class BufferedJSONCollectionTest:
+    def load(self, collection):
+        """Load the data corresopnding to a SyncedCollection from disk."""
+        with open(collection._filename) as f:
+            return json.load(f)
+
+
+@pytest.mark.skipif(PYPY, reason="Buffered mode not supported for PyPy.")
+class TestBufferedJSONDict(TestJSONDict, BufferedJSONCollectionTest):
+    """Tests of buffering JSONDicts."""
+
+    @pytest.fixture
+    def synced_dict(self):
+        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        self._backend_kwargs = {'filename': self._fn_, 'write_concern': False}
+        tmp = BufferedJSONDict(**self._backend_kwargs)
+        yield tmp
+        self._tmp_dir.cleanup()
+
+    @pytest.fixture
+    def synced_dict2(self):
+        _tmp_dir2 = TemporaryDirectory(prefix='jsondict2_')
+        _fn_2 = os.path.join(_tmp_dir2.name, 'test2.json')
+        _backend_kwargs2 = {'filename': _fn_2, 'write_concern': False}
+        tmp = BufferedJSONDict(**_backend_kwargs2)
+        yield tmp
+        _tmp_dir2.cleanup()
+
+    def test_buffered(self, synced_dict, testdata):
+        """Test basic per-instance buffering behavior."""
+        assert len(synced_dict) == 0
+        synced_dict['buffered'] = testdata
+        assert 'buffered' in synced_dict
+        assert synced_dict['buffered'] == testdata
+        with synced_dict.buffered():
+            assert 'buffered' in synced_dict
+            assert synced_dict['buffered'] == testdata
+            synced_dict['buffered2'] = 1
+            assert 'buffered2' in synced_dict
+            assert synced_dict['buffered2'] == 1
+        assert len(synced_dict) == 2
+        assert 'buffered2' in synced_dict
+        assert synced_dict['buffered2'] == 1
+        with synced_dict.buffered():
+            del synced_dict['buffered']
+            assert len(synced_dict) == 1
+            assert 'buffered' not in synced_dict
+        assert len(synced_dict) == 1
+        assert 'buffered' not in synced_dict
+        assert 'buffered2' in synced_dict
+        assert synced_dict['buffered2'] == 1
+
+        # Explicitly check that the file has not been changed when buffering.
+        raw_dict = synced_dict.to_base()
+        with synced_dict.buffered():
+            synced_dict['buffered3'] = 1
+            on_disk_dict = self.load(synced_dict)
+            assert 'buffered3' not in on_disk_dict
+            assert on_disk_dict == raw_dict
+
+        on_disk_dict = self.load(synced_dict)
+        assert 'buffered3' in on_disk_dict
+        assert on_disk_dict == synced_dict
+
+    def test_two_buffered(self, synced_dict, testdata):
+        """Test that a non-buffered copy is not modified."""
+        synced_dict['buffered'] = testdata
+        synced_dict2 = BufferedJSONDict(filename=synced_dict._filename)
+
+        # Check that the non-buffered object is not modified.
+        with synced_dict.buffered():
+            synced_dict['buffered2'] = 1
+            assert 'buffered2' not in synced_dict2
+
+    def test_two_buffered_modify_unbuffered(self, synced_dict, testdata):
+        """Test that in-memory changes raise errors in buffered mode."""
+        synced_dict['buffered'] = testdata
+        synced_dict2 = BufferedJSONDict(filename=synced_dict._filename)
+
+        # Check that the non-buffered object is not modified.
+        with pytest.raises(MetadataError):
+            with synced_dict.buffered():
+                synced_dict['buffered2'] = 1
+                synced_dict2['buffered2'] = 2
+                assert synced_dict['buffered2'] == 1
+                synced_dict['buffered2'] = 3
+                assert synced_dict2['buffered2'] == 2
+                synced_dict2['buffered2'] = 3
+                assert synced_dict['buffered2'] == 3
+
+    def test_two_buffered_modify_unbuffered_first(self, synced_dict, testdata):
+        # TODO: What is the expected behavior in this test? Data is only loaded
+        # into the buffer the first time anything happens, so if we enter the
+        # buffered context for one collection then modify an unbuffered
+        # collection within that context _before_ doing anything with the
+        # buffered collection, the buffered version will overwrite it. This
+        # behavior feels slightly unexpected, but I don't know if there's any
+        # way to fix it. For object-local buffering, we could load into the
+        # buffer when entering the context instead of waiting until the first
+        # call to load, but for global buffering there's no equivalent.
+        synced_dict['buffered'] = testdata
+        synced_dict2 = BufferedJSONDict(filename=synced_dict._filename)
+
+        # Check that the non-buffered object is not modified.
+        with synced_dict.buffered():
+            synced_dict2['buffered2'] = 1
+            assert 'buffered2' not in synced_dict
+            synced_dict['buffered2'] = 3
+        assert synced_dict == {'buffered': testdata, 'buffered2': 3}
+
+    def test_global_buffered(self, synced_dict, testdata):
+        assert len(synced_dict) == 0
+        synced_dict['buffered'] = testdata
+        assert 'buffered' in synced_dict
+        assert synced_dict['buffered'] == testdata
+        with buffer_reads_writes():
+            assert 'buffered' in synced_dict
+            assert synced_dict['buffered'] == testdata
+            synced_dict['buffered2'] = 1
+            assert 'buffered2' in synced_dict
+            assert synced_dict['buffered2'] == 1
+        assert len(synced_dict) == 2
+        assert 'buffered2' in synced_dict
+        assert synced_dict['buffered2'] == 1
+        with buffer_reads_writes():
+            del synced_dict['buffered']
+            assert len(synced_dict) == 1
+            assert 'buffered' not in synced_dict
+        assert len(synced_dict) == 1
+        assert 'buffered' not in synced_dict
+        assert 'buffered2' in synced_dict
+        assert synced_dict['buffered2'] == 1
+
+        with pytest.raises(BufferedError):
+            with buffer_reads_writes():
+                synced_dict['buffered2'] = 2
+                self.store({'test': 1})
+                assert synced_dict['buffered2'] == 2
+        assert 'test' in synced_dict
+        assert synced_dict['test'] == 1
+
+    def test_nested_same_collection(self, synced_dict):
+        """Test nesting global buffering."""
+        assert len(synced_dict) == 0
+
+        for outer_buffer, inner_buffer in itertools.product(
+                [synced_dict.buffered, buffer_reads_writes], repeat=2):
+            err_msg = (f"outer_buffer: {outer_buffer.__qualname__}, "
+                       f"inner_buffer: {inner_buffer.__qualname__}")
+            synced_dict.reset({'outside': 1})
+            with outer_buffer():
+                synced_dict['inside_first'] = 2
+                with inner_buffer():
+                    synced_dict['inside_second'] = 3
+
+                on_disk_dict = self.load(synced_dict)
+                assert 'inside_first' not in on_disk_dict, err_msg
+                assert 'inside_second' not in on_disk_dict, err_msg
+                assert 'inside_first' in synced_dict, err_msg
+                assert 'inside_second' in synced_dict, err_msg
+
+        assert self.load(synced_dict) == synced_dict
+
+    def test_nested_different_collections(self, synced_dict, synced_dict2):
+        """Test nested buffering for different collections."""
+        assert len(synced_dict) == 0
+        assert len(synced_dict2) == 0
+
+        synced_dict['outside'] = 1
+        synced_dict2['outside'] = 1
+        with synced_dict.buffered():
+            synced_dict['inside_first'] = 2
+            on_disk_dict = self.load(synced_dict)
+            assert 'inside_first' in synced_dict
+            assert 'inside_first' not in on_disk_dict
+
+            synced_dict2['inside_first'] = 2
+            on_disk_dict2 = self.load(synced_dict2)
+            assert 'inside_first' in synced_dict2
+            assert 'inside_first' in on_disk_dict2
+
+            with buffer_reads_writes():
+                synced_dict['inside_second'] = 3
+                synced_dict2['inside_second'] = 3
+
+                on_disk_dict = self.load(synced_dict)
+                assert 'inside_second' in synced_dict
+                assert 'inside_second' not in on_disk_dict
+                on_disk_dict2 = self.load(synced_dict2)
+                assert 'inside_second' in synced_dict2
+                assert 'inside_second' not in on_disk_dict2
+
+            on_disk_dict = self.load(synced_dict)
+            on_disk_dict2 = self.load(synced_dict2)
+
+            assert 'inside_first' in synced_dict
+            assert 'inside_first' not in on_disk_dict
+
+            assert 'inside_second' in synced_dict
+            assert 'inside_second' not in on_disk_dict
+            assert 'inside_second' in synced_dict2
+            assert 'inside_second' in on_disk_dict2
+
+        on_disk_dict = self.load(synced_dict)
+        on_disk_dict2 = self.load(synced_dict2)
+
+        assert 'inside_first' in synced_dict
+        assert 'inside_first' in on_disk_dict
+
+        assert 'inside_second' in synced_dict
+        assert 'inside_second' in on_disk_dict
+        assert 'inside_second' in synced_dict2
+        assert 'inside_second' in on_disk_dict2
+
+    def test_nested_copied_collection(self, synced_dict):
+        """Test modifying two collections pointing to the same data."""
+        synced_dict2 = BufferedJSONDict(filename=synced_dict._filename)
+
+        assert len(synced_dict) == 0
+        assert len(synced_dict2) == 0
+
+        synced_dict['outside'] = 1
+        with synced_dict.buffered():
+            synced_dict['inside_first'] = 2
+
+            on_disk_dict = self.load(synced_dict)
+            assert synced_dict['inside_first'] == 2
+            assert 'inside_first' not in on_disk_dict
+
+            with buffer_reads_writes():
+                synced_dict['inside_second'] = 3
+                synced_dict2['inside_second'] = 4
+
+                on_disk_dict = self.load(synced_dict)
+                assert synced_dict['inside_second'] == 4
+                assert synced_dict2['inside_second'] == 4
+                assert 'inside_second' not in on_disk_dict
+
+            on_disk_dict = self.load(synced_dict)
+            assert on_disk_dict['inside_second'] == 4
+
+    @pytest.mark.skip("Not currently sure what the expected behavior is.")
+    def test_nested_copied_collection_invalid(self, synced_dict):
+        """Test the behavior of invalid modifications of copied objects."""
+        synced_dict2 = BufferedJSONDict(filename=synced_dict._filename)
+
+        assert len(synced_dict) == 0
+        assert len(synced_dict2) == 0
+
+        synced_dict['outside'] = 1
+        finished = False
+        with pytest.raises(MetadataError):
+            with synced_dict.buffered():
+                synced_dict['inside_first'] = 2
+                # TODO: Currently, modifying synced_dict2 here causes problems.
+                # It is unbuffered, so it directly writes to file. Then, when
+                # entering global buffering in the context below, synced_dict2
+                # sees that synced_dict has already saved data for this file to
+                # the buffer, so it loads that data, which also means that
+                # synced_dict2 becomes associated with the metadata stored when
+                # synced_dict entered buffered mode. As a result, when the
+                # global buffering exits, we see metadata errors because
+                # synced_dict2 lost track of the fact that it saved changes to
+                # filemade prior to entering the global buffer. We _could_ fix
+                # this by changing the behavior of _load_buffer to not load the
+                # data from the cache if it exists, if the object is new to
+                # cached_collections then we would save a new version. However,
+                # I'm not sure that's the correct answer. Is there a true
+                # canonical source of truth in this scenario?
+                synced_dict2['inside_first'] = 3
+
+                on_disk_dict = self.load(synced_dict)
+                assert synced_dict['inside_first'] == 2
+                assert on_disk_dict['inside_first'] == 3
+
+                with buffer_reads_writes():
+                    synced_dict['inside_second'] = 3
+                    synced_dict2['inside_second'] = 4
+
+                    on_disk_dict = self.load(synced_dict)
+                    assert synced_dict['inside_second'] == 4
+                    assert synced_dict2['inside_second'] == 4
+                    assert 'inside_second' not in on_disk_dict
+
+                on_disk_dict = self.load(synced_dict)
+                assert on_disk_dict['inside_second'] == 4
+                # Check that all the checks ran before the assertion failure.
+                finished = True
+        assert finished
+
+
+@pytest.mark.skipif(PYPY, reason="Buffered mode not supported for PyPy.")
+class TestBufferedJSONList(TestJSONList, BufferedJSONCollectionTest):
+    """Tests of buffering JSONLists."""
+
+    @pytest.fixture
+    def synced_list(self):
+        self._tmp_dir = TemporaryDirectory(prefix='jsonlist_')
+        self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
+        self._backend_kwargs = {'filename': self._fn_}
+        yield BufferedJSONList(**self._backend_kwargs)
+        self._tmp_dir.cleanup()
+
+    def test_buffered(self, synced_list):
+        synced_list.extend([1, 2, 3])
+        assert len(synced_list) == 3
+        assert synced_list == [1, 2, 3]
+        with synced_list.buffered():
+            assert len(synced_list) == 3
+            assert synced_list == [1, 2, 3]
+            synced_list[0] = 4
+            assert len(synced_list) == 3
+            assert synced_list == [4, 2, 3]
+        assert len(synced_list) == 3
+        assert synced_list == [4, 2, 3]
+        with synced_list.buffered():
+            assert len(synced_list) == 3
+            assert synced_list == [4, 2, 3]
+            del synced_list[0]
+            assert len(synced_list) == 2
+            assert synced_list == [2, 3]
+        assert len(synced_list) == 2
+        assert synced_list == [2, 3]
+
+        # Explicitly check that the file has not been changed when buffering.
+        raw_list = synced_list.to_base()
+        with synced_list.buffered():
+            synced_list.append(10)
+            on_disk_list = self.load(synced_list)
+            assert 10 not in on_disk_list
+            assert on_disk_list == raw_list
+
+        on_disk_list = self.load(synced_list)
+        assert 10 in on_disk_list
+        assert on_disk_list == synced_list
+
+    def test_global_buffered(self, synced_list):
+        assert len(synced_list) == 0
+        with buffer_reads_writes():
+            synced_list.reset([1, 2, 3])
+            assert len(synced_list) == 3
+        assert len(synced_list) == 3
+        assert synced_list == [1, 2, 3]
+        with buffer_reads_writes():
+            assert len(synced_list) == 3
+            assert synced_list == [1, 2, 3]
+            synced_list[0] = 4
+            assert len(synced_list) == 3
+            assert synced_list == [4, 2, 3]
+        assert len(synced_list) == 3
+        assert synced_list == [4, 2, 3]
+
+        # metacheck failure
+        with pytest.raises(BufferedError):
+            with buffer_reads_writes():
+                synced_list.reset([1])
+                assert synced_list == [1]
+                # Unfortunately the resolution of os.stat is
+                # platform dependent and may not always be
+                # high enough for our check to work. Since
+                # this unit test is artificially simple we
+                # must add some amount of minimum waiting time
+                # to ensure that the change in time will be
+                # detected.
+                time.sleep(0.01)
+                self.store([1, 2, 3])
+                assert synced_list == [1]
+        assert len(synced_list) == 3
+        assert synced_list == [1, 2, 3]
diff --git a/tests/test_synced_collection.py b/tests/test_synced_collection.py
index 594d09d9..6da12c73 100644
--- a/tests/test_synced_collection.py
+++ b/tests/test_synced_collection.py
@@ -9,7 +9,8 @@ from collections.abc import MutableMapping
 from collections.abc import MutableSequence
 from copy import deepcopy
 
-from signac.core.synced_list import SyncedCollection
+from signac.core.synced_collection import SyncedCollection
+from signac.core.collection_json import JSONCollection
 from signac.core.collection_json import JSONDict
 from signac.core.collection_json import JSONList
 from signac.errors import InvalidKeyError
@@ -43,7 +44,8 @@ class TestJSONCollectionBase:
 
     def test_from_base_no_backend(self):
         with pytest.raises(ValueError):
-            SyncedCollection.from_base(data={'a': 0})
+            SyncedCollection.from_base(filename=self._fn_, data={'a': 0})
+        JSONCollection.from_base(filename=self._fn_, data={'a': 0})
 
 
 class TestJSONDict:
@@ -406,13 +408,11 @@ class TestJSONDict:
 
 class TestJSONList:
 
-    _write_concern = False
-
     @pytest.fixture
     def synced_list(self):
         self._tmp_dir = TemporaryDirectory(prefix='jsonlist_')
         self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
-        self._backend_kwargs = {'filename': self._fn_, 'write_concern': self._write_concern}
+        self._backend_kwargs = {'filename': self._fn_}
         yield JSONList(**self._backend_kwargs)
         self._tmp_dir.cleanup()
 
@@ -656,7 +656,7 @@ class TestJSONListWriteConcern(TestJSONList):
 
     @pytest.fixture
     def synced_list(self):
-        self._tmp_dir = TemporaryDirectory(prefix='jsondict_')
+        self._tmp_dir = TemporaryDirectory(prefix='jsonlist_')
         self._fn_ = os.path.join(self._tmp_dir.name, FN_JSON)
         self._backend_kwargs = {'filename': self._fn_, 'write_concern': True}
         yield JSONList(**self._backend_kwargs)