Repository: HypothesisWorks/hypothesis

Issue:
Add a CLI option for verbosity to the pytest plugin
The `HYPOTHESIS_VERBOSITY_LEVEL` environment variable is now deprecated (from #1211).  An easy way to control verbosity is still useful though, so we would like to support this as a command-line flag.

This would be implemented in [`hypothesis.extra.pytestplugin`](https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/extra/pytestplugin.py), similarly to [the deprecated version here](https://github.com/HypothesisWorks/hypothesis/blob/3c5f3906a7339af8bf2448281377abe903575245/hypothesis-python/src/hypothesis/_settings.py#L626-L629).  The new ``--hypothesis-verbosity`` option should be applied *after* loading the profile specified by ``--hypothesis-profile`` (if given).

Finally, the new option should be listed in `docs/details.rst`, including that the verbosity option is applied after loading a profile.

*If you would like to work on this issue, feel free to comment and I will help you get started!*


Discussion:
Hi Zac. I think I would be interested in helping out with this issue.
From a quick glance, looks like I would need to add a constant in pytestplugin.py and add some code to the pytest_addoption and pytest_configure functions to handle verbosity.

I presume the deprecated code in _settings.py would need removed? However, I am new to this project and not very familiar with how the Verbosity class is used. Do the class methods need to be retained?

Thanks in advance!
> Hi Zac. I think I would be interested in helping out with this issue.

Awesome!  We (and I) are always delighted to meet new contributors 😄  
(and again, just let me know if you have any questions or want a hand with something)

> I presume the deprecated code in _settings.py would need removed?

We'll actually keep it around for a while - Semantic Versioning means that it will not be an error before Hypothesis 4.0 (later this year), and we *also* have a stability policy that says we should wait six months before removing a deprecated feature, which might push the final removal to 5.0.  That can feel like a *very long time* as a dev, but it's great as a user!

> From a quick glance, looks like I would need to add a constant in pytestplugin.py and add some code to the pytest_addoption and pytest_configure functions to handle verbosity.

Exactly right!  I linked to `_settings.py` as the example code to copy for `pytest_addoption`, so that should be easy enough.  
It would be great to document these options as part of the same pull request, but if you'd rather not that can happen later.

> [you didn't ask for general advice, but here it is:]

- I suggest opening a pull request while it's still a work-in-progress - mine *always* are, even when I didn't plan it that way
- Read our [documentation guide](https://github.com/HypothesisWorks/hypothesis/blob/master/guides/documentation.rst), especially the changelog section
- Add yourself to the list of contributors in `CONTRIBUTING.rst`




Good to know about the semantic versioning policy used in this project. I will leave the code in _settings.py untouched. Thanks for the advice too, I will look over the documentation guide before making my contribution. I will probably need to brush up on github real quick too, haha.

> Exactly right! I linked to _settings.py as the example code to copy for pytest_addoption, so that should be easy enough.

Cool! Yes this part appears pretty straightforward.

I would not be opposed to writing some documentation. If I understand correctly, the available parameters for this option for this are:
    quiet = 0
    normal = 1
    verbose = 2
    debug = 3

A few quick questions. 

- Do you have any recommendations for where to look that would give me a better idea of what each respective option does? 
- Should there be an action parameter passed to the group.addoption call for the verbosity option? 
- If a default option is provided (I'm guessing one should be), is any particular naming convention preferred (e.g. default= 'normal' or default='1')?
Valid options for the *setting* are the members of the `Verbosity` enum.  Valid options for the *pytest flag* are the names, e.g. `--hypothesis-verbosity=debug` (because we'll then use the string to look up the enum member).  The [`enum` module docs](https://docs.python.org/3/library/enum.html) are a good reference if you're curious.

Some docs on verbosity [here](https://hypothesis.readthedocs.io/en/latest/settings.html#seeing-intermediate-result), though having found them they're incomplete (also controls debug logging, deprecation warnings, etc.).  I suggest linking to the settings page instead of re-describing the behaviour of each option, and I'll fix that end!

Yes, use `action='store'` to save the string for later use.

There should not be a default option - instead, treat it like `SEED` in `pytest_configure` and only set the verbosity if the option was passed.
I'm having a look at this one @ pyconAU sprintday


Resolving patch:
diff --git a/hypothesis-python/src/hypothesis/extra/pytestplugin.py b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
--- a/hypothesis-python/src/hypothesis/extra/pytestplugin.py
+++ b/hypothesis-python/src/hypothesis/extra/pytestplugin.py
@@ -19,7 +19,7 @@
 
 import pytest
 
-from hypothesis import core, settings
+from hypothesis import Verbosity, core, settings
 from hypothesis.reporting import default as default_reporter
 from hypothesis.reporting import with_reporter
 from hypothesis.statistics import collector
@@ -27,6 +27,7 @@
 from hypothesis.internal.detection import is_hypothesis_test
 
 LOAD_PROFILE_OPTION = '--hypothesis-profile'
+VERBOSITY_OPTION = '--hypothesis-verbosity'
 PRINT_STATISTICS_OPTION = '--hypothesis-show-statistics'
 SEED_OPTION = '--hypothesis-seed'
 
@@ -52,6 +53,12 @@ def pytest_addoption(parser):
         action='store',
         help='Load in a registered hypothesis.settings profile'
     )
+    group.addoption(
+        VERBOSITY_OPTION,
+        action='store',
+        choices=[opt.name for opt in Verbosity],
+        help='Override profile with verbosity setting specified'
+    )
     group.addoption(
         PRINT_STATISTICS_OPTION,
         action='store_true',
@@ -68,7 +75,7 @@ def pytest_addoption(parser):
 def pytest_report_header(config):
     profile = config.getoption(LOAD_PROFILE_OPTION)
     if not profile:
-        profile = 'default'
+        profile = settings._current_profile
     settings_str = settings.get_profile(profile).show_changed()
     if settings_str != '':
         settings_str = ' -> %s' % (settings_str)
@@ -80,6 +87,16 @@ def pytest_configure(config):
     profile = config.getoption(LOAD_PROFILE_OPTION)
     if profile:
         settings.load_profile(profile)
+    verbosity_name = config.getoption(VERBOSITY_OPTION)
+    if verbosity_name:
+        verbosity_value = Verbosity[verbosity_name]
+        profile_name = '%s-with-%s-verbosity' % (
+            settings._current_profile, verbosity_name
+        )
+        # register_profile creates a new profile, exactly like the current one,
+        # with the extra values given (in this case 'verbosity')
+        settings.register_profile(profile_name, verbosity=verbosity_value)
+        settings.load_profile(profile_name)
     seed = config.getoption(SEED_OPTION)
     if seed is not None:
         try:


Test patch: