Part 1 — The Database. How to build a real time data sync, multi platform app with CouchDB and PouchDB
By: Tudor Dumitriu
Tags: Tech, Pouchdb, Mobile App Development, Couchdb, NoSQL

Design

Before diving into the specifics of our Database design, here are a few words about the product being built.

The app is basically a task management system, that allows organizing tasks in lists and calendars, sharing them with friends and more, all while being focused on your goals. It also needs to be able to run simultaneously on browsers and mobile devices.

That should explain pretty well why the app should be available in offline mode, and why the data needs to sync in real time.

Also, only the shared lists of tasks need to be kept in sync between friends, and since the data is entirely synced with the client, each user needs their own DB, with restricted access, but the common records (and only the common records) need to by synced automatically between per-user DBs.

Image 1: Database Architecture

Image 1 presents the high-level database architecture, but what is very important here is that the standard communication paradigm has shifted from UI-API-DB to UI-DB-SERVICES.

This approach has its challenges, because on the server side we need listener services always running to track events that trigger business logic. Also this forced us to have quite a lot of the business logic residing on the client side, which is not necessarily a bad thing but needs to be kept in mind.

In order to understand it better, a general data flow process should shed some light. Assuming that User 1 and User 2 are friends and have a shared list of tasks, when User 1 updates one of the shared tasks, the following data flow occurs:

Task gets replicated from PouchDB Client User DB1 to CouchDB User DB1 Spiegel listens to CouchDB User DB1 changes and triggers a CouchDB replication from User DB1 to CentralDB Spiegel listens to CentralDB changes and determines (via an API call, but let’s keep things simple) what changes go where and if the User 2 has the required permissions Spiegel triggers a CouchDB replication from CentralDB to the User DB2 PouchDB long polling via HTTP mechanism is getting the changes in Couch DB User DB 2 and updates the PouchDB Client User DB 2

Quirks

The process, as always, wasn’t defined straight through, the design emerged gradually while adding more features, facing more and more challenges and limitations. We had to come up with various solutions, listeners, complementing services, even cleanup jobs for the deleted revisions and here are some of them:

Conflicts will happen when 2 users update the same document at the same time. Then, scheduled and constantly running jobs need to be created, to watch those conflicts and to gracefully (merge) solve them (otherwise CouchDB will select one of the revisions and that’s it)

Solving conflicts on the client side: in PouchDB conflicts need to be considered and solved, but it might not be enough

If you need more granular control when replicating a document according to specific business rules, Spiegel can call an API endpoint where you can implement specific logic

A CentralDB, holding ALL the important documents is needed because otherwise it would be very difficult to query documents for specific reports. But also, listening to document changes in EVERY User DB is highly impractical

Because the DB is responsible of Client-Server communication, this leads to a CQRS-ish way of doing things, meaning that the services are listening to changes and if there is a business-concerning event, the event is raised to the proper service that in turn will be responsible for generating a notification that will finally (via PouchDB to CouchDB replication) arrive to the client. This forced us, for immediate response requests (like adding a friend) to create an API endpoint as well

Because even though PouchDB is capable of using various plugins for local storage, the storage footprint needs to be monitored, so we are keeping a limited number of revisions and auto-compaction enabled. Also, since the entire client database gets replicated automatically to the CouchDB server, that too needs to be configured carefully to limit the number of revisions, to enable auto-compaction as well and even more special jobs might have to periodically purge the deleted documents, because otherwise will end up with a huge storage usage

We don’t recommend using CouchDB attachments for file storage, because behind the scenes each file would be converted to base64 and replicated via the standard mechanism, which is fantastic for small documents but not so great (actually bad) for larger ones, especially files, so the file, images and attachments support should be handled separately

Conclusion

Both CouchDB and PouchDB are mature technologies, developed by strong teams with active communities. They are most definitely alternatives worth considering and both of them are capable of being used independently.

There is so much more than this, CouchDB is designed with multi cluster replication and sharding in mind, PouchDB has a rich set of plugins allowing it to run from Web Workers or in-memory to transforming the DB into an Event Sourcing one with Delta Pouch. All these are topics for the advanced classes and will require a deeper understanding of the two.

Hoping that we’ve scratched the surface and made you curious, please join the discussion if there’s anything you want to share!