Repository: jqlang/jq

Problem statement:
Disable some tests when building without regex support

Skip regex tests when built without regexp support
This PR catches the exception of regex library absence, and gracefully skip the test when built without oniguruma. This resolves #2292, but is better running all the manual tests except for regex ones.


Hints:

[![Coverage Status](https://coveralls.io/builds/38318328/badge)](https://coveralls.io/builds/38318328)

Coverage decreased (-2.03%) to 82.1% when pulling **f6a69a6e52b68a92b816a28eb20719a3d0cb51ae on KAction:no-regex** into **80052e5275ae8c45b20411eecdd49c945a64a412 on stedolan:master**.

I understand the tests fail without regex support, but I'm against skipping the manual test entirely. This is written in README.md as `note that jq's tests require regexp support to pass`. Closing as unacceptable patch, but if you have any other solution, please comment below.
Just to be sure I understand correctly, with my distribution maintainer hat on -- if I am building  `jq` version without regex support, I am not supposed to run tests, correct?
Depends on what you want to achieve. I found Homebrew recipe checks a minimal test case that the executable can be invoked ([ref](https://github.com/Homebrew/homebrew-core/blob/1d647932d0c46f64891d6494a6191791d8e51326/Formula/jq.rb#L57)). This allows end users to install the command even when some bugs or flakiness were introduced to bundled test cases. But if you still want to run the entire tests, we need a better solution than skipping man.test entirely. Maybe splitting the man.test into two, or improving `--run-tests` to skip regex tests if not available.
Then I am lost. This pull request is supposed doing exactly that -- improving tests so regex tests are skipped when regex is not built.
I created a PR #2644 to the solution `improving --run-tests to skip regex tests if not available`.
Could we use the fact that (nearly?) all regexp related test are in onig.test? if so maybe skip could be done in the build system instead of in the run tests code?

The macOS build failure looks like something temporary:
```
error: Could not read ffac597fdb6ce8236b5e2da77ab6fbae3d9d9cb7
fatal: revision walk setup failed
error: https://github.com/Homebrew/homebrew-cask did not send all necessary objects

Error: Fetching /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask failed!
```
But we need this code to make man.test pass without regexp support after all.
> But we need this code to make man.test pass without regexp support after all.

Aha i see 👍 

Maybe if we end up with more conditional tests in the future the .test format could be extended with some kind of feature gating? 
Hmm, let me reconsider merging and explore about improving the .test format. How about adding `%%REGEX` sign to indicate the test case requires the regex support, including the `regex: true` flag in the manual source file, and skip them unless `HAVE_LIBONIG`?
Maybe use the config vars and have `%%HAVE_LIBOING` or just `%%LIBOING` would make it similar to what we do in c code. But  i like `%%REGEX` also, more feature name than implementation detail... but i guess jq is not changing regex implementation anytime soon :)

Patch:
diff --git a/.gitattributes b/.gitattributes
index 8e8a2e8a86..20508cf3d6 100644
--- a/.gitattributes
+++ b/.gitattributes
@@ -9,5 +9,6 @@ src/decNumber/** linguist-vendored
 src/lexer.[ch] linguist-generated=true
 src/parser.[ch] linguist-generated=true
 tests/man.test linguist-generated=true
+tests/manonig.test linguist-generated=true
 jq.1.prebuilt linguist-generated=true
 docs/Pipfile.lock linguist-generated=true
diff --git a/.github/workflows/manpage.yml b/.github/workflows/manpage.yml
index 6ce2fdcfe8..41d7a1dda1 100644
--- a/.github/workflows/manpage.yml
+++ b/.github/workflows/manpage.yml
@@ -1,16 +1,18 @@
-name: Building man page and man.test
+name: Building man page, man.test, manonig.test
 on:
   push:
     paths:
       - '.github/workflows/manpage.yml'
       - 'docs/**'
       - 'tests/man.test'
+      - 'tests/manonig.test'
       - 'jq.1.prebuilt'
   pull_request:
     paths:
       - '.github/workflows/manpage.yml'
       - 'docs/**'
       - 'tests/man.test'
+      - 'tests/manonig.test'
       - 'jq.1.prebuilt'
 
 jobs:
@@ -33,14 +35,14 @@ jobs:
         run: |
           autoreconf -i
           ./configure --enable-docs
-      - name: Build man page and man.test
+      - name: Build man page, man.test, manonig.test
         run: |
           mv jq.1.prebuilt jq.1.old
-          rm -f tests/man.test
-          make jq.1.prebuilt tests/man.test
-      - name: Make sure that jq.1.prebuilt and man.test are up to date
+          rm -f tests/man.test manonig.test
+          make jq.1.prebuilt tests/man.test tests/manonig.test
+      - name: Make sure that jq.1.prebuilt, man.test, manonig.test are up to date
         run: |
-          git diff --exit-code tests/man.test
+          git diff --exit-code tests/man.test tests/manonig.test
           # skip build date in jq.1.prebuilt
           test -s jq.1.prebuilt
           diff -- <(tail -n +3 jq.1.old) <(tail -n +3 jq.1.prebuilt)
diff --git a/Makefile.am b/Makefile.am
index 662d294bc9..b8cd58d3bb 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -134,7 +134,7 @@ endif
 
 ### Tests (make check)
 
-TESTS = tests/optionaltest tests/mantest tests/jqtest tests/onigtest tests/shtest tests/utf8test tests/base64test
+TESTS = tests/optionaltest tests/mantest tests/jqtest tests/shtest tests/utf8test tests/base64test
 AM_TESTS_ENVIRONMENT = JQ=$(abs_builddir)/jq NO_VALGRIND=$(NO_VALGRIND)
 
 # This is a magic make variable that causes it to treat tests/man.test as a
@@ -150,9 +150,9 @@ check_DATA = tests/man.test
 # Making changes to the manpage without having the python deps means your
 # tests won't run. If you aren't making changes to the examples, you probably
 # don't care. But if you are, then you need to run the tests anyway.
-tests/man.test: $(srcdir)/docs/content/manual/manual.yml
+tests/man.test tests/manonig.test: $(srcdir)/docs/content/manual/manual.yml
 if ENABLE_DOCS
-	$(AM_V_GEN) ( cd ${abs_srcdir}/docs; $(PIPENV) run python build_mantests.py ) > $@
+	$(AM_V_GEN) ( cd ${abs_srcdir}/docs; $(PIPENV) run python build_mantests.py )
 else
 	@echo Changes to the manual.yml require docs to be enabled to update the manual test.
 	@echo As a result, the manual test is out of date.
@@ -186,6 +186,10 @@ endif
 
 AM_CFLAGS += $(onig_CFLAGS)
 
+if WITH_ONIGURUMA
+TESTS += tests/onigtest tests/manonigtest
+endif
+
 ### Packaging
 
 docs/site.yml: configure.ac
@@ -216,9 +220,9 @@ EXTRA_DIST = $(DOC_FILES) $(man_MANS) $(TESTS) $(TEST_LOG_COMPILER)     \
         tests/modules/test_bind_order2.jq                               \
         tests/onig.supp tests/local.supp                                \
         tests/setup tests/torture/input0.json                           \
-        tests/optional.test tests/man.test tests/jq.test                \
-        tests/onig.test tests/base64.test tests/utf8-truncate.jq        \
-        tests/jq-f-test.sh
+        tests/optional.test tests/man.test tests/manonig.test           \
+        tests/jq.test tests/onig.test tests/base64.test                 \
+        tests/utf8-truncate.jq tests/jq-f-test.sh
 
 AM_DISTCHECK_CONFIGURE_FLAGS=--disable-maintainer-mode --with-oniguruma=builtin
 
diff --git a/README.md b/README.md
index 2680244d77..7871ca8438 100644
--- a/README.md
+++ b/README.md
@@ -20,8 +20,7 @@ homepage and on the github release page, https://github.com/jqlang/jq/releases
 If you're building directly from the latest git, you'll need flex,
 bison (3.0 or newer), libtool, make, automake, and autoconf installed.
 To get regexp support you'll also need to install Oniguruma or clone it as a
-git submodule as per the instructions below.
-(note that jq's tests require regexp support to pass).  To build, run:
+git submodule as per the instructions below. To build, run:
 
     git submodule update --init # if building from git to get oniguruma
     autoreconf -i               # if building from git
diff --git a/configure.ac b/configure.ac
index d4c601af15..a100a55eb5 100644
--- a/configure.ac
+++ b/configure.ac
@@ -288,6 +288,7 @@ AC_SUBST(onig_CFLAGS)
 AC_SUBST(onig_LDFLAGS)
 
 AM_CONDITIONAL([BUILD_ONIGURUMA], [test "x$build_oniguruma" = xyes])
+AM_CONDITIONAL([WITH_ONIGURUMA], [test "x$with_oniguruma" != xno])
 AC_SUBST([BUNDLER], ["$bundle_cmd"])
 
 AC_CONFIG_MACRO_DIRS([config/m4 m4])
diff --git a/docs/README.md b/docs/README.md
index 541390535a..52fdd59b82 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -5,22 +5,22 @@ The jq website, manpages and some of the tests are generated from this
 directory. The manual is a YAML file in `content/manual`.
 
 To build the documentation (including building the jq manpage), you'll
-need python3 and pipenv. You can install pipenv like so:
+need `python3` and `pipenv`. You can install `pipenv` like so:
 
     pip install pipenv
 
-Though, you may need to say pip3 instead, depending on your system. Once
-you have pipenv installed, you can install the dependencies by running
-`pipenv install` from the `docs` directory.
+Though, you may need to say `pip3` instead, depending on your system. Once
+you have `pipenv` installed, you can install the dependencies by running
+`pipenv sync` from the `docs` directory.
 
 Also, you may need to run `virtualenv -p /usr/bin/python3 venv/` and
-then `source venv/bin/activate`, and only then `pipenv install`.
+then `source venv/bin/activate`, and only then `pipenv sync`.
 
 Once this is done, rerun `./configure` in the jq root directory and then
 the Makefile will be able to generate the jq manpage.  You can also just
 run `pipenv run build_manpage.py` in the `docs` directory to build the
 `jq.1` page manually, and `pipenv run build_mantests.py` to build the
-contents of `tests/man.test`.
+contents of `tests/man.test` and `tests/manonig.test`.
 
 To build the website, run `pipenv run ./build_website.py` from inside
 the `docs` directory.


Test patch:
diff --git a/docs/build_mantests.py b/docs/build_mantests.py
index 5973604776..925e14488b 100755
--- a/docs/build_mantests.py
+++ b/docs/build_mantests.py
@@ -1,13 +1,21 @@
 #!/usr/bin/env python3
 import yaml
+import re
 
-with open("content/manual/manual.yml") as f:
-    manual = yaml.safe_load(f)
+regex_program_pattern = re.compile(
+    r'\b(?:test|match|capture|scan|split|splits|sub|gsub)\s*\(')
+
+with open('content/manual/manual.yml') as source, \
+        open('../tests/man.test', 'w') as man, \
+        open('../tests/manonig.test', 'w') as manonig:
+    manual = yaml.safe_load(source)
     for section in manual.get('sections', []):
         for entry in section.get('entries', []):
             for example in entry.get('examples', []):
-                print(example.get('program', '').replace('\n', ' '))
-                print(example.get('input', ''))
+                program = example.get('program', '').replace('\n', ' ')
+                out = manonig if regex_program_pattern.search(program) else man
+                print(program, file=out)
+                print(example.get('input', ''), file=out)
                 for s in example.get('output', []):
-                    print(s)
-                print('')
+                    print(s, file=out)
+                print('', file=out)
diff --git a/tests/man.test b/tests/man.test
index 426d0d4bb7..25aeb9ddc0 100644
--- a/tests/man.test
+++ b/tests/man.test
@@ -564,10 +564,6 @@ implode
 [65, 66, 67]
 "ABC"
 
-split(", ")
-"a, b,c,d, e, "
-["a","b,c,d","e",""]
-
 join(", ")
 ["a","b,c,d","e"]
 "a, b,c,d, e"
@@ -612,10 +608,6 @@ walk(if type == "array" then sort else . end)
 [[4, 1, 7], [8, 5, 2], [3, 6, 9]]
 [[1,4,7],[2,5,8],[3,6,9]]
 
-walk( if type == "object" then with_entries( .key |= sub( "^_+"; "") ) else . end )
-[ { "_a": { "__b": 2 } } ]
-[{"a":{"b":2}}]
-
 $ENV.PAGER
 null
 "less"
@@ -755,42 +747,6 @@ true
 ["1", "invalid", "3", 4]
 [1, 3, 4]
 
-test("foo")
-"foo"
-true
-
-.[] | test("a b c # spaces are ignored"; "ix")
-["xabcd", "ABC"]
-true
-true
-
-match("(abc)+"; "g")
-"abc abc"
-{"offset": 0, "length": 3, "string": "abc", "captures": [{"offset": 0, "length": 3, "string": "abc", "name": null}]}
-{"offset": 4, "length": 3, "string": "abc", "captures": [{"offset": 4, "length": 3, "string": "abc", "name": null}]}
-
-match("foo")
-"foo bar foo"
-{"offset": 0, "length": 3, "string": "foo", "captures": []}
-
-match(["foo", "ig"])
-"foo bar FOO"
-{"offset": 0, "length": 3, "string": "foo", "captures": []}
-{"offset": 8, "length": 3, "string": "FOO", "captures": []}
-
-match("foo (?<bar123>bar)? foo"; "ig")
-"foo bar foo foo  foo"
-{"offset": 0, "length": 11, "string": "foo bar foo", "captures": [{"offset": 4, "length": 3, "string": "bar", "name": "bar123"}]}
-{"offset": 12, "length": 8, "string": "foo  foo", "captures": [{"offset": -1, "length": 0, "string": null, "name": "bar123"}]}
-
-[ match("."; "g")] | length
-"abc"
-3
-
-capture("(?<a>[a-z]+)-(?<n>[0-9]+)")
-"xyzzy-14"
-{ "a": "xyzzy", "n": "14" }
-
 .bar as $x | .foo | . + $x
 {"foo":10, "bar":200}
 210
diff --git a/tests/manonig.test b/tests/manonig.test
new file mode 100644
index 0000000000..36f3b091f0
--- /dev/null
+++ b/tests/manonig.test
@@ -0,0 +1,44 @@
+split(", ")
+"a, b,c,d, e, "
+["a","b,c,d","e",""]
+
+walk( if type == "object" then with_entries( .key |= sub( "^_+"; "") ) else . end )
+[ { "_a": { "__b": 2 } } ]
+[{"a":{"b":2}}]
+
+test("foo")
+"foo"
+true
+
+.[] | test("a b c # spaces are ignored"; "ix")
+["xabcd", "ABC"]
+true
+true
+
+match("(abc)+"; "g")
+"abc abc"
+{"offset": 0, "length": 3, "string": "abc", "captures": [{"offset": 0, "length": 3, "string": "abc", "name": null}]}
+{"offset": 4, "length": 3, "string": "abc", "captures": [{"offset": 4, "length": 3, "string": "abc", "name": null}]}
+
+match("foo")
+"foo bar foo"
+{"offset": 0, "length": 3, "string": "foo", "captures": []}
+
+match(["foo", "ig"])
+"foo bar FOO"
+{"offset": 0, "length": 3, "string": "foo", "captures": []}
+{"offset": 8, "length": 3, "string": "FOO", "captures": []}
+
+match("foo (?<bar123>bar)? foo"; "ig")
+"foo bar foo foo  foo"
+{"offset": 0, "length": 11, "string": "foo bar foo", "captures": [{"offset": 4, "length": 3, "string": "bar", "name": "bar123"}]}
+{"offset": 12, "length": 8, "string": "foo  foo", "captures": [{"offset": -1, "length": 0, "string": null, "name": "bar123"}]}
+
+[ match("."; "g")] | length
+"abc"
+3
+
+capture("(?<a>[a-z]+)-(?<n>[0-9]+)")
+"xyzzy-14"
+{ "a": "xyzzy", "n": "14" }
+
diff --git a/tests/manonigtest b/tests/manonigtest
new file mode 100755
index 0000000000..cc06f42e7b
--- /dev/null
+++ b/tests/manonigtest
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+. "${0%/*}/setup" "$@"
+
+$VALGRIND $Q $JQ -L "$mods" --run-tests $JQBASEDIR/tests/manonig.test
diff --git a/tests/mantest b/tests/mantest
index 2e429eeade..e9b56e0973 100755
--- a/tests/mantest
+++ b/tests/mantest
@@ -3,4 +3,4 @@
 . "${0%/*}/setup" "$@"
 
 # We set PAGER because there's a mantest for `env` that uses it.
-env PAGER=less $VALGRIND $Q $JQ -L "$mods" --run-tests < $JQBASEDIR/tests/man.test
+env PAGER=less $VALGRIND $Q $JQ -L "$mods" --run-tests $JQBASEDIR/tests/man.test