Repository: nf-core/tools

Issue:
PyInquirer conda conflict for nf-core 1.10.2 and python 3.7
Hi,
when running `conda create --name nf-core-test python=3.7 nf-core=1.10.2` I get 

```
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package python conflicts for:
nf-core=1.10.2 -> python
nf-core=1.10.2 -> click -> python[version='2.7.*|3.5.*|3.6.*|3.4.*|>=3|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.8,<3.9.0a0|>=3.4']
python=3.7
```

Without specifying the python version it works using `python 3.6.11`.


Discussion:
Good to know, thanks! Not sure that there’s much that we can do about this, other than recommending Python 3.8 instead of 3.7 (I guess that also works, based on the error message?)
Somehow `conda create --name nf-core-test python=3.8 nf-core=1.10.2` doesn't work either. I guess otherwise it would also install a newer version, if one is compatible, when not specifying the python version.
Don't know, the conda conflict messages do not always help somehow.
I just got the same problem for py3.8 and nf-core 1.10.2 in conda - also with no explanation in the errors. Installing from Pip (in the same conda environment) worked fine...?!
I have found `mamba` to be quite helpful in determining the cause of this so tried this out in a fresh environment :-) 

```bash
$ mamba install -c bioconda -c conda-forge python=3.8 nf-core=1.10.2 -k

                  __    __    __    __
                 /  \  /  \  /  \  /  \
                /    \/    \/    \/    \
███████████████/  /██/  /██/  /██/  /████████████████████████
              /  / \   / \   / \   / \  \____
             /  /   \_/   \_/   \_/   \    o \__,
            / _/                       \_____/  `
            |/
        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗
        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗
        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║
        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║
        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║
        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝

        mamba (0.6.1) supported by @QuantStack

        GitHub:  https://github.com/mamba-org/mamba
        Twitter: https://twitter.com/QuantStack

█████████████████████████████████████████████████████████████

bioconda/noarch          [====================] (00m:00s) Done
pkgs/main/linux-64       [====================] (00m:00s) Done
pkgs/r/noarch            [====================] (00m:00s) Done
bioconda/linux-64        [====================] (00m:00s) Done
pkgs/main/noarch         [====================] (00m:00s) Done
conda-forge/noarch       [====================] (00m:00s) Done
pkgs/r/linux-64          [====================] (00m:00s) Done
conda-forge/linux-64     [====================] (00m:01s) Done

Looking for: ['python=3.8', 'nf-core=1.10.2']

Encountered problems while solving.
Problem: package nf-core-1.10.2-pyh864c0ab_0 requires pyinquirer 1.0.2.*, but none of the providers can be installed
```

So it looks like its directly related to pyinquirer :-( 
PyInquirer!

![shakefist](https://media.giphy.com/media/kOkvavM1PxZao/giphy.gif)
[Questionary](https://github.com/tmbo/questionary) now has a `file` selector type which is quite nice, and should handle full validation I think. Needs thought and work, but at the moment I think that this is the best / easiest swap for PyInquirer.

Difficult to tell whether it will be better than PyInquirer with its dependencies as it uses Poetry. But might be worth a play with that first to check that it actually solves the installation issues.

Could also think about going old school and just doing it homebrew without any packages. Won't be as slick but might simplify things.
![](https://github.trello.services/images/mini-trello-icon.png) [nf-core tools](https://trello.com/c/7IF6AMOo/2-nf-core-tools)
Started to play a bit with Questionary - it looks like we can really just swap it.
It has a `prompt` command that takes the same input as PyInquirer, and it has a `Separator()`.
I swapped it in all places and the code works and takes questions as expected. Haven't tried all use cases yet, though.

You can't install it alongside PyInquirer because that depends on an old version of the prompt-toolkit (version 1.-something, while Questionary uses 3.0.8). In my book that speaks for Questionary ... 

I will test around a bit more - still need to fully understand the nf-core/tools code :-)
I've had great experiences with Questionary. The devs are also very responsive, somewhat contrary to the PyInquirer devs.
I've recently found similar [issues](https://github.com/ENCODE-DCC/atac-seq-pipeline/pull/282#issuecomment-704582247) caused by outdated bioconda libraries conflicting with updated dependencies on conda-forge. You can resolved it like this.

```
conda install -n ENV_NAME --strict-channel-priority --override-channels --c conda-forge -c bioconda PKG1 PKG2 PK3...
```
Yeah, Questionary looks good but be aware that _"it has the same two problems that PyInquirer has"_ - https://github.com/nf-core/tools/pull/684#issue-454516402

Unfortunately past-me didn't care to elaborate on what those two problems were 🤦🏻 
Ah, you can see the reason in the code comments when I switched back to the stable(!) release of PyInquirer:

```diff
- #
- # NOTE: WE ARE USING A PRE-RELEASE VERSION OF PYINQUIRER
- #
- # This is so that we can capture keyboard interruptions in a nicer way
- # with the raise_keyboard_interrupt=True argument in the prompt.prompt() calls
- # It also allows list selections to have a default set.
- #
- # Waiting for a release of version of >1.0.3 of PyInquirer.
- # See https://github.com/CITGuru/PyInquirer/issues/90
- #
- # When available, update setup.py to use regular pip version

+ # NOTE: When PyInquirer 1.0.3 is released we can capture keyboard interruptions
+ # in a nicer way # with the raise_keyboard_interrupt=True argument in the PyInquirer.prompt() calls
+ # It also allows list selections to have a default set.
+ #
+ # Until then we have workarounds:
+ # * Default list item is moved to the top of the list
+ # * We manually raise a KeyboardInterrupt if we get None back from a question
+ #
```

https://github.com/nf-core/tools/pull/684/files#diff-31477effcfb136e692dd57c7d0abf196e9cb8962d49c1bb43cee0f2a7ad8c93bL24-L33

So the two problems I was hitting were:

* Being able to capture `ctrl-c` to exit
* Having a default value when using a `select` list of options

And I added slightly unsatisfactory workarounds to both problems.
1) questionary has a `unsafe_ask()` method, which does not catch any keyboard interrupts. (https://github.com/tmbo/questionary/blob/master/questionary/question.py#L55).
I just wrote a new function, which wraps all questionary functions (I get what I want with getattr), catches all keyboard interrupts and does what I want with it. 
2) well kinda similar story to earlier with PyInquirer. There's an open PR which is pretty much done, but the author didn't have time to clean it. But honestly, having the default on top is not toooo bad. 
So what to do? :-) 
Questionary and PyInquirer are pretty much compatible, partly because the former is based on the latter (at least he mentions that its based on that). But Questionary is much more frequently updated and maintained.
So exchanging the two is just a swap of the import and some mentions in the code - and it would solve the issue with the python version..

On top of that, we could implement the `path` method from Questionary. For this we would need to know which input parameter actually is a path. Could be done by adding a boolean `path` to the schema, which we could then query in the `single_param_to_pyinquirer` function, and simply set the question type to path.

What do you think?

Edit: or directly change the `type` to `path` in the schema, of course.

Edit2: hmm okay I'm now realising that you can't just declare the `type`as `path` in JSON schema ... 

Resolving patch:
diff --git a/nf_core/launch.py b/nf_core/launch.py
--- a/nf_core/launch.py
+++ b/nf_core/launch.py
@@ -10,7 +10,8 @@
 import json
 import logging
 import os
-import PyInquirer
+import prompt_toolkit
+import questionary
 import re
 import subprocess
 import textwrap
@@ -20,15 +21,21 @@
 
 log = logging.getLogger(__name__)
 
-#
-# NOTE: When PyInquirer 1.0.3 is released we can capture keyboard interruptions
-# in a nicer way # with the raise_keyboard_interrupt=True argument in the PyInquirer.prompt() calls
-# It also allows list selections to have a default set.
-#
-# Until then we have workarounds:
-# * Default list item is moved to the top of the list
-# * We manually raise a KeyboardInterrupt if we get None back from a question
-#
+# Custom style for questionary
+nfcore_question_style = prompt_toolkit.styles.Style(
+    [
+        ("qmark", "fg:ansiblue bold"),  # token in front of the question
+        ("question", "bold"),  # question text
+        ("answer", "fg:ansigreen nobold"),  # submitted answer text behind the question
+        ("pointer", "fg:ansiyellow bold"),  # pointer used in select and checkbox prompts
+        ("highlighted", "fg:ansiblue bold"),  # pointed-at choice in select and checkbox prompts
+        ("selected", "fg:ansigreen noreverse"),  # style for a selected item of a checkbox
+        ("separator", "fg:ansiblack"),  # separator in lists
+        ("instruction", ""),  # user instructions for select, rawselect, checkbox
+        ("text", ""),  # plain text
+        ("disabled", "fg:gray italic"),  # disabled choices for select and checkbox prompts
+    ]
+)
 
 
 class Launch(object):
@@ -256,11 +263,9 @@ def prompt_web_gui(self):
             "name": "use_web_gui",
             "message": "Choose launch method",
             "choices": ["Web based", "Command line"],
+            "default": "Web based",
         }
-        answer = PyInquirer.prompt([question])
-        # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released
-        if answer == {}:
-            raise KeyboardInterrupt
+        answer = questionary.unsafe_prompt([question], style=nfcore_question_style)
         return answer["use_web_gui"] == "Web based"
 
     def launch_web_gui(self):
@@ -347,14 +352,14 @@ def sanitise_web_response(self):
         The web builder returns everything as strings.
         Use the functions defined in the cli wizard to convert to the correct types.
         """
-        # Collect pyinquirer objects for each defined input_param
-        pyinquirer_objects = {}
+        # Collect questionary objects for each defined input_param
+        questionary_objects = {}
         for param_id, param_obj in self.schema_obj.schema.get("properties", {}).items():
-            pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False)
+            questionary_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False)
 
         for d_key, definition in self.schema_obj.schema.get("definitions", {}).items():
             for param_id, param_obj in definition.get("properties", {}).items():
-                pyinquirer_objects[param_id] = self.single_param_to_pyinquirer(param_id, param_obj, print_help=False)
+                questionary_objects[param_id] = self.single_param_to_questionary(param_id, param_obj, print_help=False)
 
         # Go through input params and sanitise
         for params in [self.nxf_flags, self.schema_obj.input_params]:
@@ -364,7 +369,7 @@ def sanitise_web_response(self):
                     del params[param_id]
                     continue
                 # Run filter function on value
-                filter_func = pyinquirer_objects.get(param_id, {}).get("filter")
+                filter_func = questionary_objects.get(param_id, {}).get("filter")
                 if filter_func is not None:
                     params[param_id] = filter_func(params[param_id])
 
@@ -396,19 +401,13 @@ def prompt_param(self, param_id, param_obj, is_required, answers):
         """Prompt for a single parameter"""
 
         # Print the question
-        question = self.single_param_to_pyinquirer(param_id, param_obj, answers)
-        answer = PyInquirer.prompt([question])
-        # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released
-        if answer == {}:
-            raise KeyboardInterrupt
+        question = self.single_param_to_questionary(param_id, param_obj, answers)
+        answer = questionary.unsafe_prompt([question], style=nfcore_question_style)
 
         # If required and got an empty reponse, ask again
         while type(answer[param_id]) is str and answer[param_id].strip() == "" and is_required:
             log.error("'–-{}' is required".format(param_id))
-            answer = PyInquirer.prompt([question])
-            # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released
-            if answer == {}:
-                raise KeyboardInterrupt
+            answer = questionary.unsafe_prompt([question], style=nfcore_question_style)
 
         # Don't return empty answers
         if answer[param_id] == "":
@@ -426,29 +425,31 @@ def prompt_group(self, group_id, group_obj):
         Returns:
           Dict of param_id:val answers
         """
-        question = {
-            "type": "list",
-            "name": group_id,
-            "message": group_obj.get("title", group_id),
-            "choices": ["Continue >>", PyInquirer.Separator()],
-        }
-
-        for param_id, param in group_obj["properties"].items():
-            if not param.get("hidden", False) or self.show_hidden:
-                question["choices"].append(param_id)
-
-        # Skip if all questions hidden
-        if len(question["choices"]) == 2:
-            return {}
-
         while_break = False
         answers = {}
         while not while_break:
+            question = {
+                "type": "list",
+                "name": group_id,
+                "message": group_obj.get("title", group_id),
+                "choices": ["Continue >>", questionary.Separator()],
+            }
+
+            for param_id, param in group_obj["properties"].items():
+                if not param.get("hidden", False) or self.show_hidden:
+                    q_title = param_id
+                    if param_id in answers:
+                        q_title += "  [{}]".format(answers[param_id])
+                    elif "default" in param:
+                        q_title += "  [{}]".format(param["default"])
+                    question["choices"].append(questionary.Choice(title=q_title, value=param_id))
+
+            # Skip if all questions hidden
+            if len(question["choices"]) == 2:
+                return {}
+
             self.print_param_header(group_id, group_obj)
-            answer = PyInquirer.prompt([question])
-            # TODO: use raise_keyboard_interrupt=True when PyInquirer 1.0.3 is released
-            if answer == {}:
-                raise KeyboardInterrupt
+            answer = questionary.unsafe_prompt([question], style=nfcore_question_style)
             if answer[group_id] == "Continue >>":
                 while_break = True
                 # Check if there are any required parameters that don't have answers
@@ -456,7 +457,7 @@ def prompt_group(self, group_id, group_obj):
                     req_default = self.schema_obj.input_params.get(p_required, "")
                     req_answer = answers.get(p_required, "")
                     if req_default == "" and req_answer == "":
-                        log.error("'{}' is required.".format(p_required))
+                        log.error("'--{}' is required.".format(p_required))
                         while_break = False
             else:
                 param_id = answer[group_id]
@@ -465,8 +466,8 @@ def prompt_group(self, group_id, group_obj):
 
         return answers
 
-    def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_help=True):
-        """Convert a JSONSchema param to a PyInquirer question
+    def single_param_to_questionary(self, param_id, param_obj, answers=None, print_help=True):
+        """Convert a JSONSchema param to a Questionary question
 
         Args:
           param_id: Parameter ID (string)
@@ -475,7 +476,7 @@ def single_param_to_pyinquirer(self, param_id, param_obj, answers=None, print_he
           print_help: If description and help_text should be printed (bool)
 
         Returns:
-          Single PyInquirer dict, to be appended to questions list
+          Single Questionary dict, to be appended to questions list
         """
         if answers is None:
             answers = {}
@@ -598,16 +599,6 @@ def filter_range(val):
             question["type"] = "list"
             question["choices"] = param_obj["enum"]
 
-            # Validate enum from schema
-            def validate_enum(val):
-                if val == "":
-                    return True
-                if val in param_obj["enum"]:
-                    return True
-                return "Must be one of: {}".format(", ".join(param_obj["enum"]))
-
-            question["validate"] = validate_enum
-
         # Validate pattern from schema
         if "pattern" in param_obj:
 
@@ -620,21 +611,6 @@ def validate_pattern(val):
 
             question["validate"] = validate_pattern
 
-        # WORKAROUND - PyInquirer <1.0.3 cannot have a default position in a list
-        # For now, move the default option to the top.
-        # TODO: Delete this code when PyInquirer >=1.0.3 is released.
-        if question["type"] == "list" and "default" in question:
-            try:
-                question["choices"].remove(question["default"])
-                question["choices"].insert(0, question["default"])
-            except ValueError:
-                log.warning(
-                    "Default value `{}` not found in list of choices: {}".format(
-                        question["default"], ", ".join(question["choices"])
-                    )
-                )
-        ### End of workaround code
-
         return question
 
     def print_param_header(self, param_id, param_obj):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -36,7 +36,8 @@
         "GitPython",
         "jinja2",
         "jsonschema",
-        "PyInquirer==1.0.2",
+        "questionary>=1.8.0",
+        "prompt_toolkit",
         "pyyaml",
         "requests",
         "requests_cache",


Test patch:
diff --git a/tests/test_launch.py b/tests/test_launch.py
--- a/tests/test_launch.py
+++ b/tests/test_launch.py
@@ -93,21 +93,21 @@ def test_nf_merge_schema(self):
         assert self.launcher.schema_obj.schema["allOf"][0] == {"$ref": "#/definitions/coreNextflow"}
         assert "-resume" in self.launcher.schema_obj.schema["definitions"]["coreNextflow"]["properties"]
 
-    def test_ob_to_pyinquirer_string(self):
+    def test_ob_to_questionary_string(self):
         """ Check converting a python dict to a pyenquirer format - simple strings """
         sc_obj = {
             "type": "string",
             "default": "data/*{1,2}.fastq.gz",
         }
-        result = self.launcher.single_param_to_pyinquirer("input", sc_obj)
+        result = self.launcher.single_param_to_questionary("input", sc_obj)
         assert result == {"type": "input", "name": "input", "message": "input", "default": "data/*{1,2}.fastq.gz"}
 
-    @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Web based"}])
+    @mock.patch("questionary.unsafe_prompt", side_effect=[{"use_web_gui": "Web based"}])
     def test_prompt_web_gui_true(self, mock_prompt):
         """ Check the prompt to launch the web schema or use the cli """
         assert self.launcher.prompt_web_gui() == True
 
-    @mock.patch("PyInquirer.prompt", side_effect=[{"use_web_gui": "Command line"}])
+    @mock.patch("questionary.unsafe_prompt", side_effect=[{"use_web_gui": "Command line"}])
     def test_prompt_web_gui_false(self, mock_prompt):
         """ Check the prompt to launch the web schema or use the cli """
         assert self.launcher.prompt_web_gui() == False
@@ -198,13 +198,13 @@ def test_sanitise_web_response(self):
         assert self.launcher.schema_obj.input_params["single_end"] == True
         assert self.launcher.schema_obj.input_params["max_cpus"] == 12
 
-    def test_ob_to_pyinquirer_bool(self):
+    def test_ob_to_questionary_bool(self):
         """ Check converting a python dict to a pyenquirer format - booleans """
         sc_obj = {
             "type": "boolean",
             "default": "True",
         }
-        result = self.launcher.single_param_to_pyinquirer("single_end", sc_obj)
+        result = self.launcher.single_param_to_questionary("single_end", sc_obj)
         assert result["type"] == "list"
         assert result["name"] == "single_end"
         assert result["message"] == "single_end"
@@ -218,10 +218,10 @@ def test_ob_to_pyinquirer_bool(self):
         assert result["filter"]("false") == False
         assert result["filter"](False) == False
 
-    def test_ob_to_pyinquirer_number(self):
+    def test_ob_to_questionary_number(self):
         """ Check converting a python dict to a pyenquirer format - with enum """
         sc_obj = {"type": "number", "default": 0.1}
-        result = self.launcher.single_param_to_pyinquirer("min_reps_consensus", sc_obj)
+        result = self.launcher.single_param_to_questionary("min_reps_consensus", sc_obj)
         assert result["type"] == "input"
         assert result["default"] == "0.1"
         assert result["validate"]("123") is True
@@ -232,10 +232,10 @@ def test_ob_to_pyinquirer_number(self):
         assert result["filter"]("123.456") == float(123.456)
         assert result["filter"]("") == ""
 
-    def test_ob_to_pyinquirer_integer(self):
+    def test_ob_to_questionary_integer(self):
         """ Check converting a python dict to a pyenquirer format - with enum """
         sc_obj = {"type": "integer", "default": 1}
-        result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj)
+        result = self.launcher.single_param_to_questionary("broad_cutoff", sc_obj)
         assert result["type"] == "input"
         assert result["default"] == "1"
         assert result["validate"]("123") is True
@@ -246,10 +246,10 @@ def test_ob_to_pyinquirer_integer(self):
         assert result["filter"]("123") == int(123)
         assert result["filter"]("") == ""
 
-    def test_ob_to_pyinquirer_range(self):
+    def test_ob_to_questionary_range(self):
         """ Check converting a python dict to a pyenquirer format - with enum """
         sc_obj = {"type": "range", "minimum": "10", "maximum": "20", "default": 15}
-        result = self.launcher.single_param_to_pyinquirer("broad_cutoff", sc_obj)
+        result = self.launcher.single_param_to_questionary("broad_cutoff", sc_obj)
         assert result["type"] == "input"
         assert result["default"] == "15"
         assert result["validate"]("20") is True
@@ -260,21 +260,18 @@ def test_ob_to_pyinquirer_range(self):
         assert result["filter"]("20") == float(20)
         assert result["filter"]("") == ""
 
-    def test_ob_to_pyinquirer_enum(self):
-        """ Check converting a python dict to a pyenquirer format - with enum """
+    def test_ob_to_questionary_enum(self):
+        """ Check converting a python dict to a questionary format - with enum """
         sc_obj = {"type": "string", "default": "copy", "enum": ["symlink", "rellink"]}
-        result = self.launcher.single_param_to_pyinquirer("publish_dir_mode", sc_obj)
+        result = self.launcher.single_param_to_questionary("publish_dir_mode", sc_obj)
         assert result["type"] == "list"
         assert result["default"] == "copy"
         assert result["choices"] == ["symlink", "rellink"]
-        assert result["validate"]("symlink") is True
-        assert result["validate"]("") is True
-        assert result["validate"]("not_allowed") == "Must be one of: symlink, rellink"
 
-    def test_ob_to_pyinquirer_pattern(self):
-        """ Check converting a python dict to a pyenquirer format - with pattern """
+    def test_ob_to_questionary_pattern(self):
+        """ Check converting a python dict to a questionary format - with pattern """
         sc_obj = {"type": "string", "pattern": "^([a-zA-Z0-9_\\-\\.]+)@([a-zA-Z0-9_\\-\\.]+)\\.([a-zA-Z]{2,5})$"}
-        result = self.launcher.single_param_to_pyinquirer("email", sc_obj)
+        result = self.launcher.single_param_to_questionary("email", sc_obj)
         assert result["type"] == "input"
         assert result["validate"]("test@email.com") is True
         assert result["validate"]("") is True