Repository: jsonata-js/jsonata (js)

Issue:
Proposal: array function to return a unique list of values
I'm looking for something like a `$distinct(array)` function, that squashes an array into a unique list of values. Alternatively, I would think it could also be called `$unique(...)`

I've managed to write a couple expressions that build an object with keys, then gets the list of key names, but that seems too heavy handed to be very useful.

Ideally, this function would also take a second argument, a function that returns true/false (like $sift or $filter), to determine whether each element is unique. Arguments to this optional arg would probably follow the pattern of the `$reduce(...) ` function, so the test logic can see what element is being checked, what its index is in the original array, and what the output array contains. But this is all just off the top of my head, so we have a starting point for any further discussions.


Pull request description:
Add function $distinct()
Upgrade the equality operators `=` and `!=` to do deep equality comparison.
Add `$distinct()` function to return sequence of values with duplicates removed
Resolves #117 

Patch:
diff --git a/docs/array-functions.md b/docs/array-functions.md
index e13fc4f..788e927 100644
--- a/docs/array-functions.md
+++ b/docs/array-functions.md
@@ -10,8 +10,8 @@ Returns the number of items in the `array` parameter.  If the `array` parameter
 
 If `array` is not specified, then the context value is used as the value of `array`.
 
-__Examples__  
-- `$count([1,2,3,1])` => `4`  
+__Examples__
+- `$count([1,2,3,1])` => `4`
 - `$count("hello")` => 1
 
 ## `$append()`
@@ -19,9 +19,9 @@ __Signature:__ `$append(array1, array2)`
 
 Returns and array containing the values in `array1` followed by the values in `array2`.  If either parameter is not an array, then it is treated as a singleton array containing that value.
 
-__Examples__  
-- `$append([1,2,3], [4,5,6])` => `[1,2,3,4,5,6]`  
-- `$append([1,2,3], 4)` => `[1,2,3,4]`  
+__Examples__
+- `$append([1,2,3], [4,5,6])` => `[1,2,3,4,5,6]`
+- `$append([1,2,3], 4)` => `[1,2,3,4]`
 - `$append("Hello", "World")` => `["Hello", "World"]`
 
 
@@ -52,8 +52,8 @@ __Signature:__ `$reverse(array)`
 
 Returns an array containing all the values from the `array` parameter, but in reverse order.
 
-__Examples__  
-- `$reverse(["Hello", "World"])` => `["World", "Hello"]`  
+__Examples__
+- `$reverse(["Hello", "World"])` => `["World", "Hello"]`
 - `[1..5] ~> $reverse()` => `[5, 4, 3, 2, 1]`
 
 ## `$shuffle()`
@@ -61,10 +61,17 @@ __Signature:__ `$shuffle(array)`
 
 Returns an array containing all the values from the `array` parameter, but shuffled into random order.
 
-__Examples__  
+__Examples__
 - `$shuffle([1..9])` => `[6, 8, 2, 3, 9, 5, 1, 4, 7]`
 
+## `$distinct()`
+__Signature__ `$distinct(array)`
 
+Returns an array containing all the values from the `array` parameter, but with any duplicates removed.  Values are tested for deep equality as if by using the [equality operator](comparison-operators#equals).
+
+__Examples__
+- `$distinct([1,2,3,3,4,3,5])` => `[1, 2, 3, 4, 5]`
+- `$distinct(Account.Order.Product.Description.Colour)` => `[ "Purple", "Orange", "Black" ]`
 
 ## `$zip()`
 __Signature:__ `$zip(array1, ...)`
@@ -73,6 +80,6 @@ Returns a convolved (zipped) array containing grouped arrays of values from the
 
 This function accepts a variable number of arguments.  The length of the returned array is equal to the length of the shortest array in the arguments.
 
-__Examples__  
-- `$zip([1,2,3], [4,5,6])` => `[[1,4] ,[2,5], [3,6]]`  
+__Examples__
+- `$zip([1,2,3], [4,5,6])` => `[[1,4] ,[2,5], [3,6]]`
 - `$zip([1,2,3],[4,5],[7,8,9])` => `[[1,4,7], [2,5,8]]`
diff --git a/docs/comparison-operators.md b/docs/comparison-operators.md
index af91c5a..cd7a964 100644
--- a/docs/comparison-operators.md
+++ b/docs/comparison-operators.md
@@ -6,20 +6,20 @@ sidebar_label: Comparison Operators
 
 ## `=` (Equals)
 
-The equality operator returns Boolean `true` if both operands are the same (type and value).  Otherwise it returns `false`.
+The equality operator returns Boolean `true` if both operands are the same (type and value).  Arrays and objects are checked for deep equality.  Arrays must have the same values in the same order. Objects must have the same key/value pairs (order is not relevant).  Otherwise it returns `false`.
 
 __Examples__
 
-- `1+1 = 2` => `true`   
+- `1+1 = 2` => `true`
 - `"Hello" = "World"` => `false`
 
 ## `!=` (Not equals)
 
-The inequality operator returns Boolean `false` if both operands are the same (type and value).  Otherwise it returns `true`.
+The inequality operator returns Boolean `false` if both operands are the same (type and value, deep equality).  Otherwise it returns `true`.
 
 __Examples__
 
-- `1+1 != 3` => `true`   
+- `1+1 != 3` => `true`
 - `"Hello" != "World"` => `true`
 
 ## `>` (Greater than)
@@ -28,7 +28,7 @@ The 'greater than' operator returns Boolean `true` if the LHS is numerically gre
 
 __Examples__
 
-- `22 / 7 > 3` => `true`   
+- `22 / 7 > 3` => `true`
 - `5 > 5` => `false`
 
 ## `<` (Less than)
@@ -37,7 +37,7 @@ The 'less than' operator returns Boolean `true` if the LHS is numerically less t
 
 __Examples__
 
-- `22 / 7 < 3` => `false`   
+- `22 / 7 < 3` => `false`
 - `5 < 5` => `false`
 
 
@@ -47,7 +47,7 @@ The 'greater than or equals' operator returns Boolean `true` if the LHS is numer
 
 __Examples__
 
-- `22 / 7 >= 3` => `true`   
+- `22 / 7 >= 3` => `true`
 - `5 >= 5` => `true`
 
 
@@ -57,7 +57,7 @@ The 'less than or equals' operator returns Boolean `true` if the LHS is numerica
 
 __Examples__
 
-- `22 / 7 <= 3` => `false`   
+- `22 / 7 <= 3` => `false`
 - `5 <= 5` => `true`
 
 ## `in` (Inclusion)
@@ -66,7 +66,7 @@ The array (sequence) inclusion operator returns Boolean `true` if the value of t
 
 __Examples__
 
-- `"world" in ["hello", "world"]` => `true`   
-- `"hello" in "hello"` => `true`   
+- `"world" in ["hello", "world"]` => `true`
+- `"hello" in "hello"` => `true`
 - `library.books["Aho" in authors].title`
 
diff --git a/src/functions.js b/src/functions.js
index a2acf28..8e23648 100644
--- a/src/functions.js
+++ b/src/functions.js
@@ -13,10 +13,12 @@ const functions = (() => {
     var isArrayOfStrings = utils.isArrayOfStrings;
     var isArrayOfNumbers = utils.isArrayOfNumbers;
     var createSequence = utils.createSequence;
+    var isSequence = utils.isSequence;
     var isFunction = utils.isFunction;
     var isLambda = utils.isLambda;
     var isIterable = utils.isIterable;
     var getFunctionArity = utils.getFunctionArity;
+    var deepEquals = utils.isDeepEqual;
 
     /**
      * Sum function
@@ -1923,6 +1925,40 @@ const functions = (() => {
         return result;
     }
 
+    /**
+     * Returns the values that appear in a sequence, with duplicates eliminated.
+     * @param {Array} arr - An array or sequence of values
+     * @returns {Array} - sequence of distinct values
+     */
+    function distinct(arr) {
+        // undefined inputs always return undefined
+        if (typeof arr === 'undefined') {
+            return undefined;
+        }
+
+        if(!Array.isArray(arr) || arr.length <= 1) {
+            return arr;
+        }
+
+        var results = isSequence(arr) ? createSequence() : [];
+
+        for(var ii = 0; ii < arr.length; ii++) {
+            var value = arr[ii];
+            // is this value already in the result sequence?
+            var includes = false;
+            for(var jj = 0; jj < results.length; jj++) {
+                if (deepEquals(value, results[jj])) {
+                    includes = true;
+                    break;
+                }
+            }
+            if(!includes) {
+                results.push(value);
+            }
+        }
+        return results;
+    }
+
     /**
      * Applies a predicate function to each key/value pair in an object, and returns an object containing
      * only the key/value pairs that passed the predicate
@@ -1959,7 +1995,7 @@ const functions = (() => {
         formatNumber, formatBase, number, floor, ceil, round, abs, sqrt, power, random,
         boolean, not,
         map, zip, filter, single, foldLeft, sift,
-        keys, lookup, append, exists, spread, merge, reverse, each, error, sort, shuffle,
+        keys, lookup, append, exists, spread, merge, reverse, each, error, sort, shuffle, distinct,
         base64encode, base64decode,  encodeUrlComponent, encodeUrl, decodeUrlComponent, decodeUrl
     };
 })();
diff --git a/src/jsonata.js b/src/jsonata.js
index 4a2621f..290f807 100644
--- a/src/jsonata.js
+++ b/src/jsonata.js
@@ -33,6 +33,7 @@ var jsonata = (function() {
     var isLambda = utils.isLambda;
     var isIterable = utils.isIterable;
     var getFunctionArity = utils.getFunctionArity;
+    var isDeepEqual = utils.isDeepEqual;
 
     // Start of Evaluator code
 
@@ -602,10 +603,10 @@ var jsonata = (function() {
 
         switch (op) {
             case '=':
-                result = lhs === rhs;
+                result = isDeepEqual(lhs, rhs);
                 break;
             case '!=':
-                result = (lhs !== rhs);
+                result = !isDeepEqual(lhs, rhs);
                 break;
         }
         return result;
@@ -1703,6 +1704,7 @@ var jsonata = (function() {
     staticFrame.bind('error', defineFunction(fn.error, '<s?:x>'));
     staticFrame.bind('sort', defineFunction(fn.sort, '<af?:a>'));
     staticFrame.bind('shuffle', defineFunction(fn.shuffle, '<a:a>'));
+    staticFrame.bind('distinct', defineFunction(fn.distinct, '<x:x>'));
     staticFrame.bind('base64encode', defineFunction(fn.base64encode, '<s-:s>'));
     staticFrame.bind('base64decode', defineFunction(fn.base64decode, '<s-:s>'));
     staticFrame.bind('encodeUrlComponent', defineFunction(fn.encodeUrlComponent, '<s-:s>'));
diff --git a/src/utils.js b/src/utils.js
index 3c614d1..fea9059 100644
--- a/src/utils.js
+++ b/src/utils.js
@@ -125,6 +125,57 @@ const utils = (() => {
         );
     }
 
+    /**
+     * Compares two values for equality
+     * @param {*} lhs first value
+     * @param {*} rhs second value
+     * @returns {boolean} true if they are deep equal
+     */
+    function isDeepEqual(lhs, rhs) {
+        if (lhs === rhs) {
+            return true;
+        }
+        if(typeof lhs === 'object' && typeof rhs === 'object' && lhs !== null && rhs !== null) {
+            if(Array.isArray(lhs) && Array.isArray(rhs)) {
+                // both arrays (or sequences)
+                // must be the same length
+                if(lhs.length !== rhs.length) {
+                    return false;
+                }
+                // must contain same values in same order
+                for(var ii = 0; ii < lhs.length; ii++) {
+                    if(!isDeepEqual(lhs[ii], rhs[ii])) {
+                        return false;
+                    }
+                }
+                return true;
+            }
+            // both objects
+            // must have the same set of keys (in any order)
+            var lkeys = Object.getOwnPropertyNames(lhs);
+            var rkeys = Object.getOwnPropertyNames(rhs);
+            if(lkeys.length !== rkeys.length) {
+                return false;
+            }
+            lkeys = lkeys.sort();
+            rkeys = rkeys.sort();
+            for(ii=0; ii < lkeys.length; ii++) {
+                if(lkeys[ii] !== rkeys[ii]) {
+                    return false;
+                }
+            }
+            // must have the same values
+            for(ii=0; ii < lkeys.length; ii++) {
+                var key = lkeys[ii];
+                if(!isDeepEqual(lhs[key], rhs[key])) {
+                    return false;
+                }
+            }
+            return true;
+        }
+        return false;
+    }
+
     return {
         isNumeric,
         isArrayOfStrings,
@@ -134,7 +185,8 @@ const utils = (() => {
         isFunction,
         isLambda,
         isIterable,
-        getFunctionArity
+        getFunctionArity,
+        isDeepEqual
     };
 })();
 


Test patch:
diff --git a/test/test-suite/groups/comparison-operators/deep-equals.json b/test/test-suite/groups/comparison-operators/deep-equals.json
new file mode 100644
index 0000000..863b0f1
--- /dev/null
+++ b/test/test-suite/groups/comparison-operators/deep-equals.json
@@ -0,0 +1,117 @@
+[
+    {
+        "expr": "[1,2,3,4] = [1, 2, 3, 4]",
+        "data": null,
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "[1,2] != [1, 2]",
+        "data": null,
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "[1,2,[3],4] = [1, 2, [3], 4]",
+        "data": null,
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "[1,2,3,4] = [1, 2, [3], 4]",
+        "data": null,
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "[1,2,3,4] = [2, 3, 4]",
+        "data": null,
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "data = [1, 2, 3, 4]",
+        "data": { "data": [1,2,3,4]},
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "data[2] = 3",
+        "data": { "data": [1,2,3,4]},
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "{ 'one': 1, 'two': 2 } = { 'two': 2, 'one': 1 }",
+        "data": {},
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "{ 'one': 1, 'two': 2 } = { 'too': 2, 'one': 1 }",
+        "data": {},
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "{ 'one': 1, 'two': 3 } = { 'two': 2, 'one': 1 }",
+        "data": {},
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "{ 'one': 1, 'two': 2, 'three': 3 } = { 'two': 2, 'one': 1 }",
+        "data": {},
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "two = 2",
+        "data": { "one": 1, "two": 2 },
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "one + one = two",
+        "data": { "one": 1, "two": 2 },
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "null = null",
+        "data": {},
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "null != null",
+        "data": {},
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "null = nothing",
+        "data": {},
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "nothing = nothing",
+        "data": {},
+        "bindings": {},
+        "result": false
+    },
+    {
+        "expr": "Account.Order[0].Product[0].Description = Account.Order[1].Product[0].Description",
+        "dataset": "dataset5",
+        "bindings": {},
+        "result": true
+    },
+    {
+        "expr": "Account.Order[0].Product[0] = Account.Order[1].Product[0]",
+        "dataset": "dataset5",
+        "bindings": {},
+        "result": false
+    }
+
+]
diff --git a/test/test-suite/groups/function-distinct/distinct.json b/test/test-suite/groups/function-distinct/distinct.json
new file mode 100644
index 0000000..e256d09
--- /dev/null
+++ b/test/test-suite/groups/function-distinct/distinct.json
@@ -0,0 +1,81 @@
+[
+    {
+        "expr": "$distinct([1,2,3,4])",
+        "data": null,
+        "bindings": {},
+        "result": [1,2,3,4]
+    },
+    {
+        "expr": "$distinct([1])",
+        "data": null,
+        "bindings": {},
+        "result": [1]
+    },
+    {
+        "expr": "$distinct(nothing)",
+        "data": null,
+        "bindings": {},
+        "undefinedResult": true
+    },
+    {
+        "expr": "$distinct(null)",
+        "data": null,
+        "bindings": {},
+        "result": null
+    },
+    {
+        "expr": "$distinct('hello')",
+        "data": null,
+        "bindings": {},
+        "result": "hello"
+    },
+    {
+        "expr": "$distinct(Account.Order.Product.SKU)",
+        "dataset": "dataset5",
+        "bindings": {},
+        "result": [
+            "0406654608",
+            "0406634348",
+            "040657863",
+            "0406654603"
+        ]
+    },
+    {
+        "expr": "$distinct(Account.Order.Product.`Product Name`)",
+        "dataset": "dataset5",
+        "bindings": {},
+        "result": [
+            "Bowler Hat",
+            "Trilby hat",
+            "Cloak"
+        ]
+    },
+    {
+        "expr": "$distinct(Account.Order.Product.Description)",
+        "dataset": "dataset5",
+        "bindings": {},
+        "result": [
+            {
+                "Colour": "Purple",
+                "Width": 300,
+                "Height": 200,
+                "Depth": 210,
+                "Weight": 0.75
+            },
+            {
+                "Colour": "Orange",
+                "Width": 300,
+                "Height": 200,
+                "Depth": 210,
+                "Weight": 0.6
+            },
+            {
+                "Colour": "Black",
+                "Width": 30,
+                "Height": 20,
+                "Depth": 210,
+                "Weight": 2
+            }
+        ]
+    }
+]