# Realm Data and Partitioning Strategy Behind the WildAid O-FISH Mobile Apps

In 2020, MongoDB partnered with the WildAid Marine Protection Program to create a mobile app for officers to use while out at sea patrolling Marine Protected Areas (MPAs) worldwide. We implemented apps for iOS, Android, and web, where they all share the same Realm back end, schema, and sync strategy. This article explains the data architecture, schema, and partitioning strategy we used. If you're developing a mobile app with Realm, this post will help you design and implement your data architecture.

MPAs—like national parks on land—set aside dedicated coastal and marine environments for conservation. WildAid helps enable local agencies to protect their MPAs. We developed the O-FISH app for enforcement officers to search and create boarding reports while they're out at sea, patrolling the MPAs and boarding vessels for inspection.

O-FISH needed to be a true offline-first application as officers will typically be without network access when they're searching and creating boarding reports. It's a perfect use case for the Realm mobile database and MongoDB Realm Sync.

This video gives a great overview of the WildAid Marine Program, the requirements for the O-FISH app, and the technologies behind the app:

:youtube]{vid=54E9wfFHjiw}

This article is broken down into these sections:

- [The O-FISH Application
- System Architecture
- Data Partitioning
- Data Schema
- Handling images
- Summary
- Resources

## The O-FISH Application

There are three frontend applications.

The two mobile apps (iOS and Android) provide the same functionality. An officer logs in and can search existing boarding reports, for example, to check on past reports for a vessel before boarding it. After boarding the boat, the officer uses the app to create a new boarding report. The report contains information about the vessel, equipment, crew, catch, and any laws they're violating.

Crucially, the mobile apps need to allow users to view and create reports even when there is no network coverage (which is the norm while at sea). Data is synchronized with other app instances and the backend database when it regains network access.

    
    iOS O-FISH App in Action

The web app also allows reports to be viewed and edited. It provides dashboards to visualize the data, including plotting boardings on a map.  User accounts are created and managed through the web app.

All three frontend apps share a common backend Realm application. The Realm app is responsible for authenticating users, controlling what data gets synced to each mobile app instance, and persisting the data to MongoDB Atlas.  Multiple "agencies" share the same frontend and backend apps. An officer should have access to the reports belonging to their agency. An agency is an authority responsible for enforcing the rules for one or more regional MPAs. Agencies are often named after the country they operate in. Examples of agencies would be Galapogas or Tanzania.

## System Architecture

The iOS and Android mobile apps both contain an embedded Realm mobile database. The app reads and writes data to that Realm database-whether the device is connected to the network or not. Whenever the device has network coverage, Realm synchronizes the data with other devices via the Realm backend service.

    
    O-FISH System Architecture

The Realm database is embedded within the mobile apps, each instance storing a partition of the O-FISH data. We also need a consolidated view of all of the data that the O-FISH web app can access, and we use MongoDB Atlas for that. MongoDB Realm is also responsible for synchronizing the data with the MongoDB Atlas database.

The web app is stateless, accessing data from Atlas as needed via the Realm SDK.

MongoDB Charts dashboards are embedded in the web app to provide richer, aggregated views of the data.

## Data Partitioning

MongoDB Realm Sync uses partitions to control what data it syncs to instances of a mobile app. You typically partition data to limit the amount of space used on the device and prevent users from accessing information they're not entitled to see or change.

When a mobile app opens a synced Realm, it can provide a partition value to specify what data should be synced to the device.

As a developer, you must specify an attribute to use as your partition key.  The rules for partition keys have some restrictions:

-   All synced collections use the same attribute name and type for the partition key.
-   The key can be a `string`, `objectId`, or a `long`.
-   When the app provides a partition key, only documents that have an exact match will be synced. For example, the app can't specify a set or range of partition key values.

A common use case would be to use a string named "username" as the partition key. The mobile app would then open a Realm by setting the partition to the current user's name, ensuring that the user's data is available (but no data for other users).

If you want to see an example of creating a sophisticated partitioning strategy, then Building a Mobile Chat App Using Realm – Data Architecture describes RChat's approach (RChat is a reference mobile chat app built on Realm and MongoDB Realm). O-FISH's method is straightforward in comparison.

WildAid works with different agencies around the world. Each officer within an agency needs access to any boarding report created by other officers in the same agency. Photos added to the app by one officer should be visible to the other officers. Officers should be offered menu options tailored to their agency—an agency operating in the North Sea would want cod to be in the list of selectable species, but including clownfish would clutter the menu.

We use a string attribute named `agency` as the partitioning key to meet those requirements.

As an extra level of security, we want to ensure that an app doesn't open a Realm for the wrong partition. This could result from a coding error or because someone hacks a version of the app. When enabling Realm Sync, we can provide expressions to define whether the requesting user should be able to access a partition or not.

    
    Expression to Limit Sync Access to Partitions

For O-FISH, the rule is straightforward. We compare the logged-in user's agency name with the partition they're requesting to access. The Realm will be synced if and only if they're the same:

``` json
{
   "%%user.custom_data.agency.name": "%%partition"
}
```

## Data Schema

At the highest level, the O-FISH schema is straightforward with four Realms (each with an associated MongoDB Atlas collection):

-   `DutyChange` records an officer going on-duty or back off-duty.
-   `Report` contains all of the details associated with the inspection of a vessel.
-   `Photo` represents a picture (either of one of the users or a photo that was taken to attach to a boarding report).
-   `MenuData` contains the agency-specific values that officers can select through the app's menus.

    

You might want to right-click that diagram so that you can open it in a new tab!

Let's take a look at each of those four objects.

### DutyChange

The app creates a `DutyChange` object when a user toggles a switch to flag that they are going on or off duty (at sea or not).

    

These are the Swift and Kotlin versions of the `DutyChange` class:

::::tabs
:::tab]{tabid="Swift"}
``` swift
import RealmSwift

class DutyChange: Object {
   @objc dynamic var _id: ObjectId = ObjectId.generate()
   @objc dynamic var user: User? = User()
   @objc dynamic var date = Date()
   @objc dynamic var status = ""

   override static func primaryKey() -> String? {
      return "_id"
   }
}
```
:::
:::tab[]{tabid="Kotlin"}
``` kotlin
import io.realm.RealmObject
import io.realm.annotations.PrimaryKey
import io.realm.annotations.RealmClass
import org.bson.types.ObjectId
import java.util.Date

@RealmClass
open class DutyChange : RealmObject() {
   @PrimaryKey
   var _id: ObjectId = ObjectId.get()
   var user: User? = User()
   var date: Date = Date()
   var status: String = ""
}
```
:::
::::

On iOS, `DutyChange` inherits from the Realm `Object` class, and the attributes need to be made accessible to the Realm SDK by making them `dynamic` and adding the `@objc` annotation. The Kotlin app uses the `@RealmClass` annotation and inheritance from `RealmObject`.

Note that there is no need to include the partition key as an attribute.

In addition to primitive attributes, `DutyChange` contains `user` which is of type `User`:

::::tabs
:::tab[]{tabid="Swift"}
``` swift
import RealmSwift

class User: EmbeddedObject, ObservableObject {
   @objc dynamic var name: Name? = Name()
   @objc dynamic var email = ""
}
```
:::
:::tab[]{tabid="Kotlin"}
``` kotlin
import io.realm.RealmObject
import io.realm.annotations.RealmClass

@RealmClass(embedded = true)
open class User : RealmObject() {
   var name: Name? = Name()
   var email: String = ""
}
```
:::
::::

`User` objects are always embedded in higher-level objects rather than being top-level Realm objects. So, the class inherits from `EmbeddedObject` rather than `Object` in Swift. The Kotlin app extends the `@RealmClass` annotation to include `(embedded = true)`.

Whether created in the iOS or Android app, the `DutyChange` object is synced to MongoDB Atlas as a single `DutyChange` document that contains a `user` sub-document:

``` json
{
   "_id" : ObjectId("6059c9859a545bbceeb9e881"),
   "agency" : "Ecuadorian Galapagos",
   "date" : ISODate("2021-03-23T10:57:09.777Z"),
   "status" : "At Sea",
   "user" : {
      "email" : "global-admin@clusterdb.com",
      "name" : {
         "first" : "Global",
         "last" : "Admin"
      }
   }
}
```

There's a [Realm schema associated with each collection that's synced with Realm Sync. The schema can be viewed and managed through the Realm UI:

    

### Report

The `Report` object is at the heart of the O-FISH app. A report is what an officer reviews for relevant data before boarding a boat. A report is where the officer records all of the details when they've boarded a vessel for an inspection.

In spite of appearances, it pretty straightforward. It looks complex because there's a lot of information that an officer may need to include in their report.

Starting with the top-level object - `Report`:

::::tabs
:::tab]{tabid="Swift"}
``` swift
import RealmSwift

class Report: Object, Identifiable {
   @objc dynamic var _id: ObjectId = ObjectId.generate()
   let draft = RealmOptional()
   @objc dynamic var reportingOfficer: User? = User()
   @objc dynamic var timestamp = NSDate()
   let location = List()
   @objc dynamic var date: NSDate? = NSDate()
   @objc dynamic var vessel: Boat? = Boat()
   @objc dynamic var captain: CrewMember? = CrewMember()
   let crew = List()
   let notes = List()
   @objc dynamic var inspection: Inspection? = Inspection()

   override static func primaryKey() -> String? {
      return "_id"
   }
}
```
:::
:::tab[]{tabid="Kotlin"}
``` kotlin
@RealmClass
open class Report : RealmObject() {
   @PrimaryKey
   var _id: ObjectId = ObjectId.get()
   var reportingOfficer: User? = User()
   var timestamp: Date = Date()
   @Required
   var location: RealmList = RealmList() // In order longitude, latitude
   var date: Date? = Date()
   var vessel: Boat? = Boat()
   var captain: CrewMember? = CrewMember()
   var crew: RealmList = RealmList()
   var notes: RealmList = RealmList()
   var draft: Boolean? = false
   var inspection: Inspection? = Inspection()
}
```
:::
::::

The `Report` class contains Realm `List` s (`RealmList` in Kotlin) to store lists of instances of classes such as `CrewMember`.

Some of the classes embedded in `Report` contain further embedded classes. There are 19 classes in total that make up a `Report`. You can view all of the component classes in the [iOS and Android repos.

Once synced to Atlas, the Report is represented as a single `BoardingReports` document (the name change is part of the schema definition):

    

Note that Realm lists are mapped to JSON/BSON arrays.

### Photo

A single boarding report could contain many large photographs, and so we don't want to embed those within the `Report` object (as an object could grow very large and even exceed MongoDB's 16 MB document limit).  Instead, the `Report` object (and its embedded objects) store references to `Photo` objects. Each photo is represented by a top-level `Photo` Realm object. As an example, `Attachments` contains a Realm `List` of strings, each of which identifies a `Photo` object. Handling images will step through how we implemented this.

::::tabs
:::tab]{tabid="Swift"}
``` swift
class Attachments: EmbeddedObject, ObservableObject {
   let notes = List()
   let photoIDs = List()
}
```
:::
:::tab[]{tabid="Kotlin"}
``` kotlin
@RealmClass(embedded = true)
open class Attachments : RealmObject() {
   @Required
   var notes: RealmList = RealmList()

   @Required
   var photoIDs: RealmList = RealmList()
}
```
:::
::::

The general rule is that it isn't the best practice to store images in a database as they consume a lot of valuable storage space. A typical solution is to keep the image in some store with plentiful, cheap capacity (e.g., a block store such as cloud storage - Amazon S3 of Microsoft Blob Storage.) The O-FISH app's issue is that it's probable that the officer's phone has no internet connectivity when they create the boarding report and attach photos, so uploading them to cloud object storage can't be done at that time. As a compromise, O-FISH stores the image in the `Photo` object, but when the device has internet access, that image is uploaded to cloud object storage, removed from the `Photo` object and replaced with the S3 link. This is why the `Photo` includes both an optional binary `picture` attribute and a `pictureURL` field for the S3 link:

::::tabs
:::tab[]{tabid="Swift"}
``` swift
class Photo: Object {
   @objc dynamic var _id: ObjectId = ObjectId.generate()
   @objc dynamic var thumbNail: NSData?
   @objc dynamic var picture: NSData?
   @objc dynamic var pictureURL = ""
   @objc dynamic var referencingReportID = ""
   @objc dynamic var date = NSDate()

   override static func primaryKey() -> String? {
      return "_id"
   }
}
```
:::
:::tab[]{tabid="Kotlin"}
``` kotlin
open class Photo : RealmObject() {
   @PrimaryKey
   var _id: ObjectId = ObjectId.get()
   var thumbNail: ByteArray? = null
   var picture: ByteArray? = null
   var pictureURL: String = ""
   var referencingReportID: String = ""
   var date: Date = Date()
}
```
:::
::::

Note that we include the `referencingReportID` attribute to make it easy to delete all `Photo` objects associated with a `Report`.

The officer also needs to review past boarding reports (and attached photos), and so the `Photo` object also includes a thumbnail image for off-line use.

### MenuData

Each agency needs the ability to customize what options are added in the app's menus. For example, agencies operating in different countries will need to define the list of locally applicable laws. Each agency has a `MenuData` instance with a list of strings for each of the customizable menus:

::::tabs
:::tab[]{tabid="Swift"}
``` swift
class MenuData: Object {
   @objc dynamic var _id = ObjectId.generate()
   let countryPickerPriorityList = List()
   let ports = List()
   let fisheries = List()
   let species = List()
   let emsTypes = List()
   let activities = List()
   let gear = List()
   let violationCodes = List()
   let violationDescriptions = List()

   override static func primaryKey() -> String? {
      return "_id"
   }
}
```
:::
:::tab[]{tabid="Kotlin"}
``` kotlin
@RealmClass
open class MenuData : RealmModel {
   @PrimaryKey
   var _id: ObjectId = ObjectId.get()
   @Required
   var countryPickerPriorityList: RealmList = RealmList()
   @Required
   var ports: RealmList = RealmList()
   @Required
   var fisheries: RealmList = RealmList()
   @Required
   var species: RealmList = RealmList()
   @Required
   var emsTypes: RealmList = RealmList()
   @Required
   var activities: RealmList = RealmList()
   @Required
   var gear: RealmList = RealmList()
   @Required
   var violationCodes: RealmList = RealmList()
   @Required
   var violationDescriptions: RealmList = RealmList()
}
```
:::
::::

## Handling images

When MongoDB Realm Sync writes a new `Photo` document to Atlas, it contains the full-sized image in the `picture` attribute. It consumes space that we want to free up by moving that image to Amazon S3 and storing the resulting S3 location in `pictureURL`. Those changes are then synced back to the mobile apps, which can then decide how to get an image to render using this algorithm:

1.  If `picture` contains an image, use it.
2.  Else, if `pictureURL` is set and the device is connected to the internet, then fetch the image from cloud object storage and use the returned image.
3.  Else, use the `thumbNail`.

    

When the `Photo` document is written to Atlas, the `newPhoto` database trigger fires, which invokes a function named `newPhoto` function.

    

The trigger passes the `newPhoto` Realm function the `changeEvent`, which contains the new `Photo` document. The function invokes the `uploadImageToS3` Realm function and then updates the `Photo` document by removing the image and setting the URL:

``` javascript
exports = function(changeEvent){
const fullDocument = changeEvent.fullDocument;
const image = fullDocument.picture;
const agency = fullDocument.agency;
const id = fullDocument._id;
const imageName = `${id}`;

if (typeof image !== 'undefined') {
   console.log(`Requesting upload of image: ${imageName}`);
   context.functions.execute("uploadImageToS3", imageName, image)
   .then (() => {
      console.log('Uploaded to S3');
      const bucketName = context.values.get("photoBucket");
      const imageLink = `https://${bucketName}.s3.amazonaws.com/${imageName}`;
      const collection = context.services.get('mongodb-atlas').db("wildaid").collection("Photo");
      collection.updateOne({"_id": fullDocument._id}, {$set: {"pictureURL": imageLink}, $unset: {picture: null}});
   },
   (error) => {
      console.error(`Failed to upload image to S3: ${error}`);
   });
} else {
   console.log("No new photo to upload this time");
}
};
```

`uploadImageToS3` uses Realm's AWS integration to upload the image:

``` javascript
exports = function(name, image) {
   const s3 = context.services.get('AWS').s3(context.values.get("awsRegion"));
   const bucket = context.values.get("photoBucket");
   console.log(`Bucket: ${bucket}`);
   return s3.PutObject({
      "Bucket": bucket,
      "Key": name,
      "ACL": "public-read",
      "ContentType": "image/jpeg",
      "Body": image
   });
};
```

## Summary

We've covered the common data model used across the iOS, Android, and backend Realm apps. (The [web app also uses it, but that's beyond the scope of this article.)

The data model is deceptively simple. There's a lot of nested information that can be captured in each boarding report, resulting in 20+ classes, but there are only four top-level classes in the app, with the rest accounted for by embedding. The only other type of relationship is the references to instances of the `Photo` class from other classes (required to prevent the `Report` objects from growing too large).

The partitioning strategy is straightforward. Partitioning for every class is based on the name of the user's agency. That pattern is going to appear in many apps—just substitute "agency" with "department," "team," "user," "country," ...

Suppose you determine that your app needs a different partitioning strategy for different classes. In that case, you can implement a more sophisticated partitioning strategy by encoding a key-value pair in a string partition key.

For example, if we'd wanted to partition the reports by username (each officer can only access reports they created) and the menu items by agency, then you could partition on a string attribute named `partition`. For the `Report` objects, it would be set to pairs such as `partition = "user=bill@some-domain.com"` whereas for a `MenuData` object it might be set to `partition = "agency=Galapagos"`. Building a Mobile Chat App Using Realm – Data Architecture steps through designing these more sophisticated strategies.

## Resources

-   O-FISH GitHub repos
    -   iOS.
    -   Android.
    -   Web.
    -   Realm Backend.
-   Read Building a Mobile Chat App Using Realm – Data Architecture to understand the data model and partitioning strategy behind the RChat app-an example of a more sophisticated partitioning strategy.
-   If you're building your first SwiftUI/Realm app, then check out Build Your First iOS Mobile App Using Realm, SwiftUI, & Combine.

>If you have questions, please head to our developer community website where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.