# Improved Error Messages for Schema Validation in MongoDB 5.0

## Intro

Many MongoDB users rely on schema
validation to
enforce rules governing the structure and integrity of documents in
their collections. But one of the challenges they faced was quickly
understanding why a document that did not match the schema couldn't be
inserted or updated. This is changing in the upcoming MongoDB 5.0
release.

Schema validation ease-of-use will be significantly improved by
generating descriptive error messages whenever an operation fails
validation. This additional information provides valuable insight into
which parts of a document in an insert/update operation failed to
validate against which parts of a collection's validator, and how. From
this information, you can quickly identify and remediate code errors
that are causing documents to not comply with your validation rules. No
more tedious debugging by slicing your document into pieces to isolate
the problem!

>
>
>If you would like to evaluate this feature and provide us early
>feedback, fill in this
>form to
>participate in the preview program.
>
>

The most popular way to express the validation rules is JSON
Schema.
It is a widely adopted standard that is also used within the REST API
specification and validation. And in MongoDB, you can combine JSON
Schema with the MongoDB Query Language (MQL) to do even more.

In this post, I would like to go over a few examples to reiterate the
capabilities of schema validation and showcase the addition of new
detailed error messages.

## What Do the New Error Messages Look Like?

First, let's look at the new error message. It is a structured message
in the BSON format, explaining which part of the document didn't match
the rules and which validation rule caused this.

Consider this basic validator that ensures that the price field does not
accept negative values. In JSON Schema, the property is the equivalent
of what we call "field" in MongoDB.

``` json
{
   "$jsonSchema": {
       "properties": {
           "price": {
               "minimum": 0
           }
       }
   }
}
```

When trying to insert a document with `{price: -2}`, the following error
message will be returned.

``` json
{
   "code": 121,
   "errmsg": "Document failed validation",
   "errInfo": {
       "failingDocumentId": ObjectId("5fe0eb9642c10f01eeca66a9"),
       "details": {
           "operatorName": "$jsonSchema",
           "schemaRulesNotSatisfied": 
               {
                   "operatorName": "properties",
                   "propertiesNotSatisfied": [
                       {
                           "propertyName": "price",
                           "details": [
                               {
                                   "operatorName": "minimum",
                                   "specifiedAs": {
                                       "minimum": 0
                                   },
                                   "reason": "comparison failed",
                                   "consideredValue": -2
                               }
                           ]
                       }
                   ]
               }
           ]
       }
   }
}
```

Some of the key fields in the response are:

-   `failingDocumentId` - the \_id of the document that was evaluated
-   `operatorName` - the operator used in the validation rule
-   `propertiesNotSatisfied` - the list of fields (properties) that
    failed validation checks
-   `propertyName` - the field of the document that was evaluated
-   `specifiedAs` - the rule as it was expressed in the validator
-   `reason - explanation` of how the rule was not satisfied
-   `consideredValue` - value of the field in the document that was
    evaluated

The error may include more fields depending on the specific validation
rule, but these are the most common. You will likely find the
`propertyName` and `reason` to be the most useful fields in the
response.

Now we can look at the examples of the different validation rules and
see how the new detailed message helps us identify the reason for the
validation failure.

## Exploring a Sample Collection

As an example, we'll use a collection of real estate properties in NYC
managed by a team of real estate agents.

Here is a sample document:

``` json
{
   "PID": "EV10010A1",
   "agents": [ { "name": "Ana Blake", "email": "anab@rcgk.com" } ],
   "description": "Spacious 2BR apartment",
   "localization": { "description_es": "Espacioso apartamento de 2 dormitorios" },
   "type": "Residential",
   "address": {
       "street1": "235 E 22nd St",
       "street2": "Apt 42",
       "city": "New York",
       "state": "NY",
       "zip": "10010"
   },
   "originalPrice": 990000,
   "discountedPrice": 980000,
   "geoLocation": [ -73.9826509, 40.737499 ],
   "listedDate": "Wed Dec 11 2020 10:05:10 GMT-0500 (EST)",
   "saleDate": "Wed Dec 21 2020 12:00:04 GMT-0500 (EST)",
   "saleDetails": {
       "price": 970000,
       "buyer": { "id": "24434" },
       "bids": [
           {
               "price": 950000,
               "winner": false,
               "bidder": {
                   "id": "24432",
                   "name": "Sam James",
                   "contact": { "email": "sjames@gmail.com" }
               }
           },
           {
               "price": 970000,
               "winner": true,
               "bidder": {
                   "id": "24434",
                   "name": "Joana Miles",
                   "contact": { "email": "jm@gmail.com" }
               }
           }
       ]
   }
}
```

## Using the Value Pattern

Our real estate properties are identified with property id (PID) that
has to follow a specific naming format: It should start with two letters
followed by five digits, and some letters and digits after, like this:
WS10011FG4 or EV10010A1.

We can use JSON Schema `pattern` operator to create a rule for this as a
regular expression.

Validator:

``` json
{
   "$jsonSchema": {
       "properties": {
           "PID": {
               "bsonType": "string",
               "pattern": "^[A-Z]{2}[0-9]{5}[A-Z]+[0-9]+$"
           }
       }
   }
}
```

If we try to insert a document with a PID field that doesn't match the
pattern, for example `{ PID: "apt1" }`, we will receive an error.

The error states that the field `PID` had the value of `"apt1"` and it
did not match the regular expression, which was specified as
`"^[A-Z]{2}[0-9]{5}[A-Z]+[0-9]+$"`.

``` json
{  ...
       "schemaRulesNotSatisfied": [
         {
           "operatorName": "properties",
           "propertiesNotSatisfied": [
             {
               "propertyName": "PID",
               "details": [
                 {
                   "operatorName": "pattern",
                   "specifiedAs": {
                     "pattern": "^[A-Z]{2}[0-9]{5}[A-Z]+[0-9]+$"
                   },
                   "reason": "regular expression did not match",
                   "consideredValue": "apt1"
                 }
               ]
             }
           ]
   ...
}
```

## Additional Properties and Property Pattern

The description may be localized into several languages. Currently, our
application only supports Spanish, German, and French, so the
localization object can only contain fields `description_es`,
`description_de`, or `description_fr`. Other fields will not be allowed.

We can use operator `patternProperties` to describe this requirement as
regular expression and indicate that no other fields are expected here
with `"additionalProperties": false`.

Validator:

``` json
{
   "$jsonSchema": {
       "properties": {
           "PID": {...},
           "localization": {
               "additionalProperties": false,
               "patternProperties": {
                   "^description_(es|de|fr)+$": {
                       "bsonType": "string"
                   }
               }
           }
       }
   }
} 
```

Document like this can be inserted successfully:

``` json
{
   "PID": "TS10018A1",
   "type": "Residential",
   "localization": {
       "description_es": "Amplio apartamento de 2 dormitorios",
       "description_de": "Geräumige 2-Zimmer-Wohnung",
   }
}
```

Document like this will fail the validation check:

``` json
{
   "PID": "TS10018A1",
   "type": "Residential",
   "localization": {
       "description_cz": "Prostorný byt 2 + kk"
   }
}
```

The error below indicates that field `localization` contains additional
property `description_cz`. `description_cz` does not match the expected
pattern, so it is considered an additional property.

``` json
{ ...
   "propertiesNotSatisfied": [
       {
           "propertyName": "localization",
           "details": [
               {
                   "operatorName": "additionalProperties",
                   "specifiedAs": {
                       "additionalProperties": false
                   },
                   "additionalProperties": [
                       "description_cz"
                   ]
               }
           ]
       }
   ]
...
}
```

## Enumeration of Allowed Options

Each real estate property in our collection has a type, and we want to
use one of the four types: "Residential," "Commercial," "Industrial," or
"Land." This can be achieved with the operator `enum`.

Validator:

``` json
{
   "$jsonSchema": {
       "properties": {
           "type": {
               "enum": [ "Residential", "Commercial", "Industrial", "Land" ]
           }
       }
   }
}
```

The following document will be considered invalid:

``` json
{
   "PID": "TS10018A1", "type": "House"
}
```

The error states that field `type` failed validation because "value was
not found in enum."

``` json
{...
   "propertiesNotSatisfied": [
       {
           "propertyName": "type",
           "details": [
               {
                   "operatorName": "enum",
                   "specifiedAs": {
                       "enum": [
                           "Residential",
                           "Commercial",
                           "Industrial",
                           "Land"
                       ]
                   },
                   "reason": "value was not found in enum",
                   "consideredValue": "House"
               }
           ]
       }
   ]
...
}
```

## Arrays: Enforcing Number of Elements and Uniqueness

Agents who manage each real estate property are stored in the `agents`
array. Let's make sure there are no duplicate elements in the array, and
no more than three agents are working with the same property. We can use
`uniqueItems` and `maxItems` for this.

``` json
{
   "$jsonSchema": {
       "properties": {
           "agents": {
               "bsonType": "array",
               "uniqueItems": true,
               "maxItems": 3
           }
       }
   }
}
```

The following document violates both if the validation rules.

``` json
{
   "PID": "TS10018A1",
   "agents": [
       { "name": "Ana Blake" },
       { "name": "Felix Morin" },
       { "name": "Dilan Adams" },
       { "name": "Ana Blake" }
   ]
}
```

The error returns information about failure for two rules: "array did
not match specified length" and "found a duplicate item," and it also
points to what value was a duplicate.

``` json
{
   ...
           "propertiesNotSatisfied": [
             {
               "propertyName": "agents",
               "details": [
                 {
                   "operatorName": "maxItems",
                   "specifiedAs": { "maxItems": 3 },
                   "reason": "array did not match specified length",
                   "consideredValue": [
                     { "name": "Ana Blake" },
                     { "name": "Felix Morin" },
                     { "name": "Dilan Adams" },
                     {  "name": "Ana Blake" }
                   ]
                 },
                 {
                   "operatorName": "uniqueItems",
                   "specifiedAs": { "uniqueItems": true },
                   "reason": "found a duplicate item",
                   "consideredValue": [
                     { "name": "Ana Blake" },
                     { "name": "Felix Morin" },
                     { "name": "Dilan Adams" },
                     { "name": "Ana Blake" }
                   ],
                   "duplicatedValue": { "name": "Ana Blake" }
                 }
               ]
             ...
 }
```

## Enforcing Required Fields

Now, we want to make sure that there's contact information available for
the agents. We need each agent's name and at least one way to contact
them: phone or email. We will use `required`and `anyOf` to create this
rule.

Validator:

``` json
{
   "$jsonSchema": {
       "properties": {
           "agents": {
               "bsonType": "array",
               "uniqueItems": true,
               "maxItems": 3,
               "items": {
                   "bsonType": "object",
                   "required": [ "name" ],
                   "anyOf": [ { "required": [ "phone" ] }, { "required": [ "email" ] } ]
               }
           }
       }
   }
}
```

The following document will fail validation:

``` json
{
   "PID": "TS10018A1",
   "agents": [
       { "name": "Ana Blake", "email": "anab@rcgk.com" },
       { "name": "Felix Morin", "phone": "+12019878749" },
       { "name": "Dilan Adams" }
   ]
}
```

Here the error indicates that the third element of the array
(`"itemIndex": 2`) did not match the rule.

``` json
{
   ...
   "propertiesNotSatisfied": [
       {
           "propertyName": "agents",
           "details": [
               {
                   "operatorName": "items",
                   "reason": "At least one item did not match the sub-schema",
                   "itemIndex": 2,
                   "details": [
                       {
                           "operatorName": "anyOf",
                           "schemasNotSatisfied": [
                               {
                                   "index": 0,
                                   "details": [
                                       {
                                           "operatorName": "required",
                                           "specifiedAs": { "required": [ "phone" ] },
                                           "missingProperties": [ "phone" ]
                                       }
                                   ]
                               },
                               {
                                   "index": 1,
                                   "details": [
                                       {
                                           "operatorName": "required",
                                           "specifiedAs": { "required": [ "email" ] },
                                           "missingProperties": [ "email" ]
                                       }
                                   ]
                               }
                           ]
                       }
                   ]
               }
           ]
       }
   ]
...
}
```

## Creating Dependencies

Let's create another rule to ensure that if the document contains the
`saleDate` field, `saleDetails` is also present, and vice versa: If
there is `saleDetails`, then `saleDate` also has to exist.

``` json
{
   "$jsonSchema": {
       "dependencies": {
           "saleDate": [ "saleDetails"],
           "saleDetails": [ "saleDate"]
       }
   }
}
```

Now, let's try to insert the document with `saleDate` but with no
`saleDetails`:

``` json
{
   "PID": "TS10018A1",
   "saleDate": Date("2020-05-01T04:00:00.000Z")
}
```

The error now includes the property with dependency `saleDate` and a
property missing from the dependencies: `saleDetails`.

``` json
{ 
 ...
   "details": {
       "operatorName": "$jsonSchema",
       "schemaRulesNotSatisfied": [
           {
               "operatorName": "dependencies",
               "failingDependencies": [
                   {
                       "conditionalProperty": "saleDate",
                       "missingProperties": [ "saleDetails" ]
                   }
               ]
           }
       ]
   }
...
}
```

Notice that in JSON Schema, the field `dependencies` is in the root
object, and not inside of the specific property. Therefore in the error
message, the `details` object will have a different structure:

``` json
{ "operatorName": "dependencies", "failingDependencies": [...]}
```

In the previous examples, when the JSON Schema rule was inside of the
"properties" object, like this:

``` json
"$jsonSchema": { "properties": { "price": { "minimum": 0 } } }
```

the details of the error message contained
`"operatorName": "properties"` and a `"propertyName"`:

``` json
{ "operatorName": "properties",
   "propertiesNotSatisfied": [ { "propertyName": "...", "details": [] } ]
}
```

## Adding Business Logic to Your Validation Rules

You can use MongoDB Query Language (MQL) in your validator right next to
JSON Schema to add richer business logic to your rules.

As one example, you can use
[$expr
to add a check for a `discountPrice` to be less than `originalPrice`
just like this:

``` json
{
   "$expr": {
       "$lt":  "$discountedPrice", "$originalPrice" ]
   },
   "$jsonSchema": {...}
}
```

[$expr
resolves to `true` or `false`, and allows you to use aggregation
expressions to create sophisticated business rules.

For a little more complex example, let's say we keep an array of bids in
the document of each real estate property, and the boolean field
`isWinner` indicates if a particular bid is a winning one.

Sample document:

``` json
{
   "PID": "TS10018A1",
   "type": "Residential",
   "saleDetails": {
       "bids": 
           {
               "price": 500000,
               "isWinner": false,
               "bidder": {...}
           },
           {
               "price": 530000,
               "isWinner": true,
               "bidder": {...}
           }
       ]
   }
}
```

Let's make sure that only one of the `bids` array elements can be marked
as the winner. The validator will have an expression where we apply a
filter to the array of bids to only keep the elements with `"isWinner":`
true, and check the size of the resulting array to be less or equal to
1.

Validator:

``` json
{
   "$and": [
       {
           "$expr": {
               "$lte": [
                   {
                       "$size": {
                           "$filter": {
                               "input": "$saleDetails.bids.isWinner",
                               "cond": "$$this"
                           }
                       }
                   },
                   1
               ]
           }
       },
       {
           "$expr": {...}
       },
       {
           "$jsonSchema": {...}
       }
   ]
}
```

Let's try to insert the document with few bids having
`"isWinner": true`.

``` json
{
   "PID": "TS10018A1",
   "type": "Residential",
   "originalPrice": 600000,
   "discountedPrice": 550000,
   "saleDetails": {
       "bids": [
           { "price": 500000, "isWinner": true },
           { "price": 530000, "isWinner": true }
       ]
   }
}
```

The produced error message will indicate which expression evaluated to
false.

``` json
{
...
   "details": {
       "operatorName": "$expr",
       "specifiedAs": {
           "$expr": {
               "$lte": [
                   {
                       "$size": {
                           "$filter": {
                               "input": "$saleDetails.bids.isWinner",
                               "cond": "$$this"
                           }
                       }
                   },
                   1
               ]
           }
       },
       "reason": "expression did not match",
       "expressionResult": false
   }
...
}
```

## Geospatial Validation

As the last example, let's see how we can use the geospatial features of
MQL to ensure that all the real estate properties in the collection are
located within the New York City boundaries. Our documents include a
`geoLocation` field with coordinates. We can use `$geoWithin` to check
that these coordinates are inside the geoJSON polygon (the polygon for
New York City in this example is approximate).

Validator:

``` json
{
   "geoLocation": {
       "$geoWithin": {
           "$geometry": {
               "type": "Polygon",
               "coordinates": [
                   [ [ -73.91326904296874, 40.91091803848203 ],
                   [ -74.01626586914062, 40.75297891717686 ],
                   [ -74.05677795410156, 40.65563874006115 ],
                   [ -74.08561706542969, 40.65199222800328 ],
                   [ -74.14329528808594, 40.64417760251725 ],
                   [ -74.18724060058594, 40.643656594948524 ],
                   [ -74.234619140625, 40.556591288249905 ],
                   [ -74.26345825195312, 40.513277131087484 ],
                   [ -74.2510986328125, 40.49500373230525 ],
                   [ -73.94691467285156, 40.543026009954986 ],
                   [ -73.740234375, 40.589449604232975 ],
                   [ -73.71826171874999, 40.820045086716505 ],
                   [ -73.78829956054686, 40.8870435151357 ],
                   [ -73.91326904296874, 40.91091803848203 ] ]
               ]
           }
       }
   },
   "$jsonSchema": {...}
}
```

A document like this will be inserted successfully.

``` json
{
   "PID": "TS10018A1",
   "type": "Residential",
   "geoLocation": [ -73.9826509, 40.737499 ],
   "originalPrice": 600000,
   "discountedPrice": 550000,
   "saleDetails": {...}
}
```

The following document will fail.

``` json
{
   "PID": "TS10018A1",
   "type": "Residential",
   "geoLocation": [ -73.9826509, 80.737499 ],
   "originalPrice": 600000,
   "discountedPrice": 550000,
   "saleDetails": {...}
}
```

The error will indicate that validation failed the `$geoWithin`
operator, and the reason is "none of the considered geometries were
contained within the expression's geometry."

``` json
{
...
   "details": {
       "operatorName": "$geoWithin",
       "specifiedAs": {
           "geoLocation": {
               "$geoWithin": {...}
           }
       },
       "reason": "none of the considered geometries were contained within the   
       expression's geometry",
       "consideredValues": [ -73.9826509, 80.737499 ]
   }
...
}
```

## Conclusion and Next Steps

Schema validation is a great tool to enforce governance over your data
sets. You have the choice to express the validation rules using JSON
Schema, MongoDB Query Language, or both. And now, with the detailed
error messages, it gets even easier to use, and you can have the rules
be as sophisticated as you need, without the risk of costly maintenance.

You can find the full validator code and sample documents from this post
[here.

>
>
>If you would like to evaluate this feature and provide us early
>feedback, fill in this
>form to
>participate in the preview program.
>
>

More posts on schema validation:

-   JSON Schema Validation - Locking down your model the smart
    way
-   JSON Schema Validation - Dependencies you can depend
    on
-   JSON Schema Validation - Checking Your
    Arrays

Questions? Comments? We'd love to connect with you. Join the
conversation on the MongoDB Community
Forums.

**Safe Harbor**

The development, release, and timing of any features or functionality
described for our products remains at our sole discretion. This
information is merely intended to outline our general product direction
and it should not be relied on in making a purchasing decision nor is
this a commitment, promise or legal obligation to deliver any material,
code, or functionality.