Project: CockroachDB (App Definition and Development / Database)
File: jsonb_forward_indexing.md

JSONB Key Encoding =========================================== This document intends to build on the JSONB value encoding . Previously, it was only possible to value-encode JSONB values in CRDB. The following document describes a format for key-encoding JSONB values for the purpose of having a semantic ordering for JSON which in turn allows forward indexes on JSON columns. In order to implement this, a lexicographical order is required for the encoding of each JSONB value. Motivation JSON is stored in the JSONB format in CRDB. The motivation and use-cases for including JSONB were included in the scoping of JSONB. Previously, it was only possible to value-encode JSONB values as forward-indexes on JSON columns were not permitted. A key-encoding of JSONB now defines a valid lexicographical ordering for JSON which shall in turn allow the creation of primary indexes as well as the execution of `ORDER BY` statements on a JSON column. Moreover, it is imperative to note that JSON numbers that are decimals and JSON containers (arrays and objects) that contain such numbers could have composite encodings. Thus, these changes also allow JSON columns to now be composite in nature. Detailed Design A JSON value can be one of seven types: true, false, null, a UTF-8 string, an arbitrary-precision number, an array of zero or more JSON values, or an object consisting of key-value pairs, where the keys are unique strings and the values are JSON values. The following rules were kept in mind while designing this form of encoding, as defined in the postgres Object > Array > Boolean > Number > String > Null Object with n pairs > Object with n-1 pairs Array with n elements > Array with n - 1 elements Moreover, arrays with equal number of elements are compared in the order: `element1`, `element2`, `element3`, . Objects with an equal number of key value pairs are compared in the order: `key1`, `value1`, `key2`, `value2`, . NOTE: There is one exception to these rules, which is neither documented by Postgres, nor mentioned in the source code: empty arrays are the minimum JSON value. As far as we can tell, this is a Postgres bug that has existed for some time. We've decided to replicate this behavior to remain consistent with Postgres. We've filed a to track the issue. In order to satisfy property 1 at all times, tags are defined in an increasing order of bytes. These tags will also have to be defined in a way where the tag representing an object is a large byte representation for a hexadecimal value (such as 0xff) and the subsequent objects have a value 1 less than the previous one, where the ordering is described in point 1 above. There is a special tag for empty JSON arrays in order to handle the special case of empty arrays being ordered before all other JSON values. Additionally, tags representing terminators will also be

There will be two terminators, one for the ascending designation and the other for the descending one, and will be required to denote the end of a key encoding of the following JSON values: Objects, Arrays, Number and Strings. JSON Boolean and JSON Null are not required to have the terminator since they do not have variable length encoding due to the presence of a single tag (as explained later in this document). When converting textual JSON input into JSONB, the primitive types described by are mapped into the primitive types described in CRDB: A JSONB null is stored solely as the tag representing the type. A JSONB String is stored as: a JSONB String tag a UTF-8 string JSONB terminator tag A JSONB Number is stored as: a JSONB Number tag encoding of the number using CRDBs Decimal encoding. JSONB terminator tag It is imperative to note that the following query results in being true in postgres: select 'true'::JSONB > 'false'::JSONB; Thus, instead of a single Boolean tag in the enum (which was described above), two separate tags, one for true and one for false, shall be present. Moreover, the tag for true will have a higher value, in bytes, than the tag for false. Thus, a JSONB Boolean is stored as just the tag depicting its type. A JSONB array is stored as: a JSONB array tag number of elements present in the array encoding of the JSON values JSONB terminator tag An interesting property to note about a JSON object, while conversing about its possible encoding, is the fact that there is no order in which JSON keys are stored inside the object container. This results in the following query to be true (tested inside of psql): select '{"a": "b", "c":"d"}'::JSONB = '{"c":"d", "a":"b"}'::JSONB; Thus, it is important to sort the key-value pairs present within a JSON object. In order to achieve a sorting effect, encoding will be done in a fashion where all the key-value pairs will be sorted first according to the encodings of the current keys of these pairs. This will be followed by an encoding of each key and value present in these pairs and concatenating them to build the resulting encoding. The key and value encodings, of each such pair, will be intertwined to keep the property that the resultant encoding shall sort the same way as the original object. Thus, a JSONB object is stored as: a JSONB object tag number of key-value pairs present in the object an encoding of the key-value pairs where the pairs have been sorted first on the encodings of the keys. JSONB terminator tag | Example JSON's | Encodings | ||:--:| | enc(`NULL::JSONB`) | [JSONB_NULL] | enc(`"a"::JSONB`) | [JSONBString, enc("a"), JSONBTerminator] | | enc(`'1'::JSONB`) | [JSONBNumber, enc(1), JSONBTerminator] | | enc(`'False'::JSONB`) | [JSONB_False] | | enc(`'True'::JSONB`) | [JSONB_True] | | enc(`'[1, 2, "a"]'::JSONB`) | [JSONBArray, enc(3), JSONBNumber, enc(1), JSONBTerminator, JSONBNumber, enc(2), JSONBTerminator, JSONBString, enc("a"), JSONBTerminator, JSONBTerminator] | | enc(`'{a:b, c:d}'::JSONB`) | [ JSONBObject, enc(2), JSONBString, enc(a), JSONBTerminator, JSONBString, enc(b), JSONBTerminator, JSONBString, enc(c), JSONBTerminator, JSONBString, enc(d), JSONBTerminator, JSONBTerminator] | | enc(`'{a: [1], b: {c: 0}}'::JSONB`) | [ JSONBObject, enc(2), JSONBString, enc(a), JSONBTerminator, JSONBArray, enc(1), JSONBNumber, enc(1), JSONBTerminator, JSONBTerminator, JSONBString, enc(b), JSONBTerminator, JSONBObject, enc(1), JSONBString, enc(c), JSONBTerminator, JSONBNumber, enc(0), JSONBTerminator, JSONBTerminator, JSONBTerminator ] |