Repository: jqlang/jq

Problem statement:
bsearch as C-coded builtin
As best I can tell, based on my timings on a Mac, a jq-coded binary search algorithm on a sorted array can't compete even with the C-coded linear search.  The jq version of bsearch that I used is appended.

Thus I would like to request a C-coded bsearch that would always terminate and that would return the index of the item if the array is sorted (as per `sort`).  I realize that such a function would be unusual, but one can say "it is what it is".

The only satisfactory alternative that I can think of (#517) has already been effectively rejected.

```
# binary search
# requires last/1
# WARNING: If the input is not sorted as per sort, bsearch will terminate but with
# irrelevant results.
#
def bsearch(target):
  if length == 0 then null
  elif length == 1 then (if .[0] == target then 0 else null end)
  else . as $in
    # state variable: [start, end, answer]
    # where start and end are the upper and lower offsets to use.
    | last( [0, length-1, null]
        | while ( .[0] <= .[1] ;
                  (if .[2] != null then (.[1] = -1) # i.e. break
                   else
                     ( ( (.[1] + .[0]) / 2 ) | floor ) as $mid
                     | $in[$mid] as $monkey
                     | if $monkey == target  then (.[2] = $mid)     # success
                       elif .[0] == .[1]     then (.[1] = -1)       # failure
                       elif $monkey < target then (.[0] = ($mid + 1))
                       else (.[1] = ($mid - 1))
                       end
                   end ))) | .[2]
    end;
```



Hints:
@pkoppstein No objections to a C-coded bsearch.  As long as we have a C-coded sort, a C-coded bsearch makes sense.

When we add a jq-coded sort that takes a comparator we'd have to add a jq-coded bsearch that takes a comparator.

I tried my hand at a jq-coded bsearch and it looked much like yours (also, mine isn't quite correct, but neither is yours):

```
def bsi(t):
    t as $t|
    [.,0] |
        last(while(true;
        .[1] as $left |
        .[0] | if length <= 1 then break else . end |
        (length / 2 | floor) as $i |
        .[$i] as $e |
        if $e == $t then    [.[$i:$i+1], $left + $i]
        elif $e < $t then   [.[$i:], $left + $i]
        else                [.[0:$i], $left]
        end)
        |.[1]);
```

This version works:

```
# Outputs the index of . where to find or insert t
def bsearch(t):
    t as $t|
    [.,0] |
        last(while(true;
        .[1] as $left | .[0] |
        if length == 0 then break else . end |
        if length == 1 then
            [[], if .[0] >= $t then $left else $left + 1 end]
        else
        (length / 2 | floor) as $i |
        .[$i] as $e |
        if $e == $t then    [.[$i:$i+1], $left + $i]
        elif $e < $t then   [.[$i:], $left + $i]
        else                [.[0:$i], $left]
        end end)
        |.[1]);
```

Inserting is easy then:

```
def set_contains(t):
    t as $t | bsearch($t) as $i | .[$i] == $t;

def set_add(t):
    reduce t as $t (.;
        if set_contains($t) then .
        else bsearch(t) as $i | .[0:$i] + [$t] + .[$i:]
        end);
```

And there you go.  Sort external arrays to start with, keep them sorted with `set_add/1`, and you have a set.

How fast is this?  Probably not that fast (`set_add/` took 417 jq opcode executions to (not) insert an existing element in a sorted array of size 10, and 1424 to insert a missing element in the same array).  It could be improved, no doubt, but it already has the advantage of being O(log N): for large enough arrays this should be plenty fast :)

EDIT: Add smiley.

@nicowilliams - Your `set_add` raises several interesting points.

For clarity, I'll define a "regular" variant of set_add as follows: 

```
def set_add_regular(t):  # produce |s| * |t| items
  bsearch(t) as $i
  | if .[$i] == t then .
    else .[0:$i] + [t] + .[$i:]
    end;
```

And I'll also refer to your version as "set_add_special", since it normally produces just one result for each array input, no matter how many items there may be in the "t" stream.

### NAMING

The ambiguity between the "|s|_|t|" and the "|s|_1" interpretations of
"add this value" highlights, at least for me, the need for a
convention to make it clear that the set_add_special function is
special.

Elsewhere I've made the case for using a special character (such as
"!") in the name of special functions (as a matter of convention,
going forward), but in this case at least there is a clear
alternative: "set_add_each".

Perhaps the suffix "_each" would be applicable in other cases as well.

As you pointed out elsewhere, it is really the arguments of functions
that are regular or special, but it's clearly too late to require
markers at that level of detail. However, as you also pointed out,
markers could be useful for eye-catching documentation.

As @stedolan stated, "Everything should be a cartesian product unless
there's a really good reason not to.", so we really only need
a way to highlight special arguments.

Two possibilities are illustrated by the following:

`def set_add_special((t)): ...;`

`def set_add_special(*t): ...;`

### SET-ORIENTATION

For computing the union of two sets, one would of course like to take
advantage of the fact that both arrays are sorted, resulting in
at worst O(m+n) operations.  (I.e. merging rather than inserting.)

(Because of the insertions, set_add_special requires O(m*n) operations.)

I don't know whether you envision any version of "set_add" making its
way into builtins, but there's something to be said for having an
efficient "merge" function, especially since add_set_special is (nearly) trivial to write.

### IMPROVEMENT:

set_add_special can easily be tweaked so that only one binary search is needed:

```
def set_add_each(t):
  reduce t as $t (.;
    bsearch($t) as $i
    | if .[$i] == $t then .
      else .[0:$i] + [$t] + .[$i:]
      end) ;
```

## bsearch specification

One final question: To avoid the post-mortem check against the input array that your bsearch in practice requires, when I have the option, I require the generalized bsearch to return -(1 + n) instead of n when the item is not found. Would you have any object to using the -(1+n) convention?

@pkoppstein I think regularity isn't always desired.  It takes fewer characters to use my `set_add` to produce N outputs for each input, with each output having just one of the new elements added, than it takes to reduce your variant's N outputs into one with all N new elements added.  Utility counts for something, and here I think adding all N elements to the same input is a lot more useful than producing N outputs each with only one addition.  This is a bit of a subjective call, of course.

```
t as $t | set_add_special($t)
```

vs

```
reduce t as $t (.; set_add_regular($t))
```

:)

BTW, my `set_add` can be used to implement an insertion sort:

```
. as $source | [] | set_add($source[])
```

Also, my `set_add` works even on non-sorted arrays, only failing to avoid adding duplicates in that case!

Adding a sort/bsearch/set_add with a jq-coded comparator wouldn't be difficult at all.

@nicowilliams wrote:

> Utility counts for something

Of course, and I was not suggesting that add_set_regular was useful at all. On the contrary!!! 

However, for the two reasons I gave, I do think that (assuming jq is the way it is now), it would be very unfortunate if "add_set_special" were simply named "add_set".

(For your reference, the two reasons stem from: (1) the fact that there are "regular" and "special" interpretations of "add set"; and (2) the fact that there are two possibilities as to whether both inputs are sorted.)

@pkoppstein  Examples of other utility defs where a common general naming convention would help that also applies to `set_add` would help me agree with you.

Again, I don't think "regular" is always better than "special".  I think one of the brilliant things about jq is that it is possible to write defs that have the sort of power that a Lisp macro system brings, but without a macro system being necessary.  The key being that all function arguments are closures.  Sometimes we should use this to produce "special" forms (e.g., `first/1`, `limit/2`, `while/2`, `any/1`, `all/1`, `sort_by/1`, and so on), and other times we shouldn't, or not for every argument (e.g., the `n` argument of `limit/2` must be a "regular" argument though the other one clearly must not be).

So far I've not felt the need for a naming convention to distinguish purely-regular from special or somewhat special defs.  Indeed, my feeling so far is that such a convention would do more harm than good.  Names can only denote so much without becoming burdensome; we must seek a decent trade-off of name vs. docs complexity.

@nicowilliams wrote:

> Examples of other utility defs where a common general naming convention would help ...

set_minus would be an exact analog :-)

Also, a similar situation arises with respect to your recent suggestion to add sigma/2 (which I'll henceforth call `summation` to address @stedolan's concern) as a companion to summation(exp), because there are various other variations (e.g. to deal with "missing values").  I have to run in a minute, so I won't have time just now to think of others, but let me say that I completely agree that a naming convention would not be the _ideal_ way to deal with these issues.  However, we have to deal with jq as it is at 1.4, and it's too late to introduce some new mandatory syntax.

One thought does occur to me, though. I don't think you'll like it, but perhaps it will inspire you to think of something better. In brief, if jq provided a way to add "special forms" (such as "reduce" and "foreach") as easily as it is to add "def"-style functions, then the situation could be alleviated by letting all new (post 1.4) builtin "def"-style functions be "regular".


Patch:
diff --git a/src/builtin.c b/src/builtin.c
index d9dfdf427c..717a07528a 100644
--- a/src/builtin.c
+++ b/src/builtin.c
@@ -835,6 +835,40 @@ static jv f_sort_by_impl(jq_state *jq, jv input, jv keys) {
   }
 }
 
+/*
+ * Assuming the input array is sorted, bsearch/1 returns
+ * the index of the target if the target is in the input array; and otherwise
+ * (-1 - ix), where ix is the insertion point that would leave the array sorted.
+ * If the input is not sorted, bsearch will terminate but with irrelevant results.
+ */
+static jv f_bsearch(jq_state *jq, jv input, jv target) {
+  if (jv_get_kind(input) != JV_KIND_ARRAY) {
+    jv_free(target);
+    return type_error(input, "cannot be searched from");
+  }
+  int start = 0;
+  int end = jv_array_length(jv_copy(input));
+  jv answer = jv_invalid();
+  while (start < end) {
+    int mid = start + (end - start) / 2;
+    int result = jv_cmp(jv_copy(target), jv_array_get(jv_copy(input), mid));
+    if (result == 0) {
+      answer = jv_number(mid);
+      break;
+    } else if (result < 0) {
+      end = mid;
+    } else {
+      start = mid + 1;
+    }
+  }
+  if (!jv_is_valid(answer)) {
+    answer = jv_number(-1 - start);
+  }
+  jv_free(input);
+  jv_free(target);
+  return answer;
+}
+
 static jv f_group_by_impl(jq_state *jq, jv input, jv keys) {
   if (jv_get_kind(input) == JV_KIND_ARRAY &&
       jv_get_kind(keys) == JV_KIND_ARRAY &&
@@ -1844,6 +1878,7 @@ BINOPS
   CFUNC(f_sort, "sort", 1),
   CFUNC(f_sort_by_impl, "_sort_by_impl", 2),
   CFUNC(f_group_by_impl, "_group_by_impl", 2),
+  CFUNC(f_bsearch, "bsearch", 2),
   CFUNC(f_min, "min", 1),
   CFUNC(f_max, "max", 1),
   CFUNC(f_min_by_impl, "_min_by_impl", 2),
diff --git a/src/builtin.jq b/src/builtin.jq
index 0e12bd5389..bb32c515cc 100644
--- a/src/builtin.jq
+++ b/src/builtin.jq
@@ -219,37 +219,6 @@ def tostream:
   getpath($p) |
   reduce path(.[]?) as $q ([$p, .]; [$p+$q]);
 
-# Assuming the input array is sorted, bsearch/1 returns
-# the index of the target if the target is in the input array; and otherwise
-#  (-1 - ix), where ix is the insertion point that would leave the array sorted.
-# If the input is not sorted, bsearch will terminate but with irrelevant results.
-def bsearch($target):
-  if length == 0 then -1
-  elif length == 1 then
-     if $target == .[0] then 0 elif $target < .[0] then -1 else -2 end
-  else . as $in
-    # state variable: [start, end, answer]
-    # where start and end are the upper and lower offsets to use.
-    | [0, length-1, null]
-    | until( .[0] > .[1] ;
-             if .[2] != null then (.[1] = -1)               # i.e. break
-             else
-               ( ( (.[1] + .[0]) / 2 ) | floor ) as $mid
-               | $in[$mid] as $monkey
-               | if $monkey == $target  then (.[2] = $mid)   # success
-                 elif .[0] == .[1]     then (.[1] = -1)     # failure
-                 elif $monkey < $target then (.[0] = ($mid + 1))
-                 else (.[1] = ($mid - 1))
-                 end
-             end )
-    | if .[2] == null then          # compute the insertion point
-         if $in[ .[0] ] < $target then (-2 -.[0])
-         else (-1 -.[0])
-         end
-      else .[2]
-      end
-  end;
-
 # Apply f to composite entities recursively, and to atoms
 def walk(f):
   def w:


Test patch:
diff --git a/tests/jq.test b/tests/jq.test
index b430348ab2..b9b1f8d2c4 100644
--- a/tests/jq.test
+++ b/tests/jq.test
@@ -1642,12 +1642,22 @@ ascii_upcase
 "useful but not for é"
 "USEFUL BUT NOT FOR é"
 
-bsearch(0,2,4)
+bsearch(0,1,2,3,4)
 [1,2,3]
 -1
+0
 1
+2
 -4
 
+bsearch({x:1})
+[{ "x": 0 },{ "x": 1 },{ "x": 2 }]
+1
+
+try ["OK", bsearch(0)] catch ["KO",.]
+"aa"
+["KO","string (\"aa\") cannot be searched from"]
+
 # strptime tests are in optional.test
 
 strftime("%Y-%m-%dT%H:%M:%SZ")