Repository: mkdocs/mkdocs

Issue:
Improve error/warning handling in Config
While working on moving the markdown extension handling to the config, I noticed a few oddities with how errors and warnings are handled within the config. And taking a closer look at the code I see even more. I could just do a PR, but wanted some feedback on where you want to take this.
1.  I noticed that you need to hardcode the config key into any warnings (see this [example](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/config/config_options.py#L224)) . This is bad as the same ConfigOption class could be used for more than one key (for example, if the Type or URL types raised a warning). `base.Config._validate` does the right thing with errors and uses the current key (see [L51](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/config/base.py#L51)). The same thing should happen for errors two lines above.
2. Why don't warnings actually use warnings? Why not follow the pattern used for errors and define a `ValidationWarning` which subclasses [`warnings.Warning`](https://docs.python.org/2/library/exceptions.html#exceptions.Warning). Then within the code call `raise ValidationWarning('some message...')` and in `base.Config._validate` add a catch for warnings in addition to the current catch for errors. Using the same API for both warnings and errors seems more sensible to me (and easier to remember).
3. Why are the errors being obscured? Only the message from the error is saved (see [L51](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/config/base.py#L51)) and then later a new error is raised (see [L148](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/config/base.py#L148)). Wouldn't it be better if the errors themselves were saved and re-raised? It would certainly help with debugging. Yes, I realize that if multiple errors were raised, only one could get re-raised. Not sure what the best solution is here for errors. Maybe log all the errors and then raise the first one? At least that way the user can work through them one at a time if the traceback is needed to figure out what went wrong.



Discussion:
> 1, I noticed that you need to hardcode the config key into any warnings

Yeah, that's bad. I was planning on changing to have the same method signature as `post_validation` which would make the key available.

> 2 Why don't warnings actually use warnings?

Because how do you add multiple warnings with `warnings.Warning`?

> 3 Why are the errors being obscured? 

Because I don't want config errors to be the concern of anything outside of `mkdocs.config` and I also want to make sure we list all errors, I hate tools that show you one error at a time and you need to keep running it and fixing them one by one.

> > 2 Why don't warnings actually use warnings?
> 
> Because how do you add multiple warnings with warnings.Warnings?

Ah, sorry, you want to use `warnings.warn` not `raise Warning`. Try running this code:

``` python
import warnings
print 'Before warnings'
warnings.warn('A first warning')
print 'Between warnings'
warnings.warn('A second warning')
print 'Atfer warnings'
```

The output will be:

```
$ python foo.py
Before warnings
foo.py:4: UserWarning: A first warning
  warnings.warn('A first warning')
Between warnings
foo.py:6: UserWarning: A second warning
  warnings.warn('A second warning')
Atfer warnings
```

You can have as many warnings as you want and code execution never stops.

Of course, they don't need to be used. I was just thinking a consistent API for both warnings and errors would be nice. But as we can't "raise" warnings without stopping code execution, I guess it doesn't really matter for this use case.

> > 3 Why are the errors being obscured?
> 
> Because I don't want config errors to be the concern of anything outside of mkdocs.config and I also want to make sure we list all errors, I hate tools that show you one error at a time and you need to keep running it and fixing them one by one.

Yes, but I loose the original traceback when debugging, which, in my opinion, is much more important. That is why I suggest passing the errors themselves (not just the messages). Then, while you can still list all errors, you can re-raise at least one of them (rather than a new error) and get the traceback. Worst case scenario: I fix all errors listed expect the one I can't make sense of, rerun and get the traceback for that one outstanding error.

Perhaps if the --verbose flag is provided we can error on the first problem and show the traceback?

I forgot to add, I've been doing some further work on configs for #144, and it covers point 1. that you mentioned here.

It is still a work in progress. https://github.com/d0ugal/mkdocs/tree/deprecated-config

I've been thinking about this some more and I continue to believe that the error instance needs to be passed  through (not just the message string). Note that the logging module actually supports recording the error along with the message (see the explanation for the `exec_info` kwarg on [Logger.debug](https://docs.python.org/2/library/logging.html#logging.Logger.debug)). Then with increased verbosity, perhaps the additional info could be output for each item.

Also, the new ConfigurationError that is raised [here](https://github.com/mkdocs/mkdocs/blob/master/mkdocs/config/base.py#L148) and its traceback is confusing. It appears after the list of warnings and errors that actually matter. And in fact, the user may need to scroll up to see the warnings and errors that matter. Why not just exit with exit code 0 (maybe use [`click.Abort`](http://click.pocoo.org/4/api/#click.Abort))? The point it, the current trace_back is misleading and masking the actual problems that need to be resolved. As it serves no useful purpose, I would remove it.

Sure, I'd be happy to see this be improved. So long as it doesn't change anything behaviourally and we display all errors in one go. Go for it.

... it's just relatively low in my priorities. There are far uglier areas than the config now.

Additionally, the ConfigurationError which is raised when `config['strict']`==True` for warnings [here][1], prevents the errors from ever being logged. Of course, any errors will always exit, so the warnings need to be logged first. Perhaps L141 should be changed as follows:

```
-         if config['strict']:
+         if config['strict'] and len(errors) == 0: 
```

Again, the raised error should be replaced with a simple exit with exit code 0 for the reasons stated in my previous comment.

Although, it occurs to me that another solution would be to log all warnings and errors when they happen, rather than gathering them and logging them later. In other words, each individual Option could log the warnings in their validate/run_validation method rather than calling `self.warnings.append`. The errors could be a little trickier as then stop execution when raised, but its not impossible and would provide a more consistent API for errors and warnings.

That's a good catch with regards to strict. I'll fix that. Thanks!

> Although, it occurs to me that another solution would be to log all warnings and errors when they happen[...]

I started with this approach early on, but didn't really like it. I don't think the change is worth it for a consistent API. In some ways, I like that they feel different because the use case is different enough.

> > Although, it occurs to me that another solution would be to log all warnings and errors when they happen[...]
> 
> I started with this approach early on, but didn't really like it. I don't think the change is worth it for a consistent API. In some ways, I like that they feel different because the use case is different enough.

Fair point. It occurred to me there might be a reason for it. As you started with that route first and abandoned it, I'll take your word for it.

> Sure, I'd be happy to see this be improved. So long as it doesn't change anything behaviourally and we display all errors in one go. Go for it.

I'll see what I can work up when I find the time. Although it will probably wait till after #539 hits.


Resolving patch:
diff --git a/mkdocs/cli.py b/mkdocs/cli.py
--- a/mkdocs/cli.py
+++ b/mkdocs/cli.py
@@ -10,6 +10,7 @@
 from mkdocs import new
 from mkdocs import serve
 from mkdocs import utils
+from mkdocs import exceptions
 from mkdocs.config import load_config
 
 log = logging.getLogger(__name__)
@@ -71,13 +72,17 @@ def serve_command(dev_addr, config_file, strict, theme, livereload):
 
     logging.getLogger('tornado').setLevel(logging.WARNING)
 
-    serve.serve(
-        config_file=config_file,
-        dev_addr=dev_addr,
-        strict=strict,
-        theme=theme,
-        livereload=livereload,
-    )
+    try:
+        serve.serve(
+            config_file=config_file,
+            dev_addr=dev_addr,
+            strict=strict,
+            theme=theme,
+            livereload=livereload,
+        )
+    except exceptions.ConfigurationError as e:
+        # Avoid ugly, unhelpful traceback
+        raise SystemExit('\n' + str(e))
 
 
 @cli.command(name="build")
@@ -88,12 +93,16 @@ def serve_command(dev_addr, config_file, strict, theme, livereload):
 @click.option('--site-dir', type=click.Path(), help=site_dir_help)
 def build_command(clean, config_file, strict, theme, site_dir):
     """Build the MkDocs documentation"""
-    build.build(load_config(
-        config_file=config_file,
-        strict=strict,
-        theme=theme,
-        site_dir=site_dir
-    ), clean_site_dir=clean)
+    try:
+        build.build(load_config(
+            config_file=config_file,
+            strict=strict,
+            theme=theme,
+            site_dir=site_dir
+        ), clean_site_dir=clean)
+    except exceptions.ConfigurationError as e:
+        # Avoid ugly, unhelpful traceback
+        raise SystemExit('\n' + str(e))
 
 
 @cli.command(name="json")
@@ -114,11 +123,15 @@ def json_command(clean, config_file, strict, site_dir):
                 "MkDocs release. For details on updating: "
                 "http://www.mkdocs.org/about/release-notes/")
 
-    build.build(load_config(
-        config_file=config_file,
-        strict=strict,
-        site_dir=site_dir
-    ), dump_json=True, clean_site_dir=clean)
+    try:
+        build.build(load_config(
+            config_file=config_file,
+            strict=strict,
+            site_dir=site_dir
+        ), dump_json=True, clean_site_dir=clean)
+    except exceptions.ConfigurationError as e:
+        # Avoid ugly, unhelpful traceback
+        raise SystemExit('\n' + str(e))
 
 
 @cli.command(name="gh-deploy")
@@ -128,12 +141,16 @@ def json_command(clean, config_file, strict, site_dir):
 @click.option('--remote-branch', '-b', help=remote_branch_help)
 def gh_deploy_command(config_file, clean, message, remote_branch):
     """Deply your documentation to GitHub Pages"""
-    config = load_config(
-        config_file=config_file,
-        remote_branch=remote_branch
-    )
-    build.build(config, clean_site_dir=clean)
-    gh_deploy.gh_deploy(config, message=message)
+    try:
+        config = load_config(
+            config_file=config_file,
+            remote_branch=remote_branch
+        )
+        build.build(config, clean_site_dir=clean)
+        gh_deploy.gh_deploy(config, message=message)
+    except exceptions.ConfigurationError as e:
+        # Avoid ugly, unhelpful traceback
+        raise SystemExit('\n' + str(e))
 
 
 @cli.command(name="new")
diff --git a/mkdocs/config/base.py b/mkdocs/config/base.py
--- a/mkdocs/config/base.py
+++ b/mkdocs/config/base.py
@@ -53,7 +53,7 @@ def _validate(self):
                 self[key] = config_option.validate(value)
                 warnings.extend([(key, w) for w in config_option.warnings])
             except ValidationError as e:
-                failed.append((key, str(e)))
+                failed.append((key, e))
 
         for key in (set(self.keys()) - self._schema_keys):
             warnings.append((
@@ -147,20 +147,22 @@ def load_config(config_file=None, **kwargs):
 
     errors, warnings = cfg.validate()
 
-    if len(warnings) > 0:
-        for config_name, warning in warnings:
-            log.warning("Config value: %s. Warning: %s", config_name, warning)
-        if cfg['strict']:
-            raise exceptions.ConfigurationError(
-                "Warnings found in the config file")
+    for config_name, warning in warnings:
+        log.warning("Config value: '%s'. Warning: %s", config_name, warning)
 
-    if len(errors) > 0:
-        for config_name, error in errors:
-            log.error("Config value: %s. Error: %s", config_name, error)
-        raise exceptions.ConfigurationError("Errors found in the config file.")
+    for config_name, error in errors:
+        log.error("Config value: '%s'. Error: %s", config_name, error)
 
     for key, value in cfg.items():
+        log.debug("Config value: '%s' = %r", key, value)
 
-        log.debug("Config value: %s = %r", key, value)
+    if len(errors) > 0:
+        raise exceptions.ConfigurationError(
+            "Aborted with {0} Configuration Errors!".format(len(errors))
+        )
+    elif cfg['strict'] and len(warnings) > 0:
+        raise exceptions.ConfigurationError(
+            "Aborted with {0} Configuration Warnings in 'strict' mode!".format(len(warnings))
+        )
 
     return cfg


Test patch:
diff --git a/mkdocs/tests/config/base_tests.py b/mkdocs/tests/config/base_tests.py
--- a/mkdocs/tests/config/base_tests.py
+++ b/mkdocs/tests/config/base_tests.py
@@ -26,13 +26,13 @@ def test_missing_required(self):
 
         c = base.Config(schema=defaults.DEFAULT_SCHEMA)
 
-        failed, warnings = c.validate()
+        errors, warnings = c.validate()
 
-        self.assertEqual(failed, [
-            ('site_name', 'Required configuration not provided.')
-        ])
+        self.assertEqual(len(errors), 1)
+        self.assertEqual(errors[0][0], 'site_name')
+        self.assertEqual(str(errors[0][1]), 'Required configuration not provided.')
 
-        self.assertEqual(warnings, [])
+        self.assertEqual(len(warnings), 0)
 
     def test_load_missing_required(self):
         """
diff --git a/mkdocs/tests/config/config_tests.py b/mkdocs/tests/config/config_tests.py
--- a/mkdocs/tests/config/config_tests.py
+++ b/mkdocs/tests/config/config_tests.py
@@ -28,11 +28,12 @@ def load_missing_config():
     def test_missing_site_name(self):
         c = config.Config(schema=config.DEFAULT_SCHEMA)
         c.load_dict({})
-        errors, warings = c.validate()
-        self.assertEqual([
-            ('site_name', 'Required configuration not provided.')
-        ], errors)
-        self.assertEqual([], warings)
+        errors, warnings = c.validate()
+        self.assertEqual(len(errors), 1)
+        self.assertEqual(errors[0][0], 'site_name')
+        self.assertEqual(str(errors[0][1]), 'Required configuration not provided.')
+
+        self.assertEqual(len(warnings), 0)
 
     def test_empty_config(self):
         def load_empty_config():