Repository: conan-io/conan
Pull request: Add `core:warnings_as_errors` configuration option
Changelog: Feature: Add `core:warnings_as_errors` configuration option to make Conan raise on warnings and errors.
Docs: https://github.com/conan-io/docs/pull/3484

First try for `core:warnings_as_errors`. It worries me that people might want to have Conan stop processing on errors in this case (We mostly do that now when we print proper errors as they are next to raised exceptions), but I'm not quite sure if I should just raise here too. Any opinions are welcomed :)

Closes #14909


Problem statement:
[feature] Add "warnings as errors" mode
### What is your suggestion?

For example, sometimes when i'm developing a package I get: "Warning: No files in package"...I would like that to show up as an error. Of course, I guess that also implies the need to suppress warnings - so perhaps it's not so simple.

### Have you read the CONTRIBUTING guide?

- [X] I've read the CONTRIBUTING guide

Hints from issue discussion:
Thanks for the suggestion @rconde01 

There is already some capability of suppressing warnings in Conan 2.0 with ``core:skip_warnings``
I think this is a valid feature request and something we would be interesting to provide. It is true that ``core:skip_warnings`` is not capable of suppressing all warnings, only those that are tagged now (only a few are, with "deprecation"), but that shouldn't be a stopper to provide this, lets discuss it with the team.
For additional context, 2 other places i've seen warnings are patches and/or replace_in_file not applying correctly. I would say as a rule one should architect such that these apply, or are errors. Seems pretty risky otherwise and could be difficult to debug downstream.
> For additional context, 2 other places i've seen warnings are patches and/or replace_in_file not applying correctly. I would say as a rule one should architect such that these apply, or are errors. Seems pretty risky otherwise and could be difficult to debug downstream.

I think these would deserve their own separate ticket, as it is a different one.
The ``replace_in_file`` by default will raise an error, not a warning, if not successful. It has an opt-out ``strict=False``, but otherwise it will raise an error. Same as ``patch()``. But if they dont, I agree it would be better if they raised an error (not that it could be changed now without breaking)
hmm...but if the feature is "warnings as errors" then it's a bit weird if they don't cover these cases. But I see that indeed it's an edge case. You don't want to break any recipes, though you could make the policy to not allow `strict=False` in conan center for updated recipes. I guess the choice to show a warning is weird here. Either 

1. You didn't select `strict=False` and you want an error on failure.
2. You specified `strict=False` implying "hey - i know what i'm doing here and this behavior is expected"...warnings seem like noise then. 

Could change these cases to something like: `Info: replace_in_file didn't replace anything.` if it's really important to see it.

Commit message: Add core:warnings_as_errors configuration option

Patch to review:
diff --git a/conan/api/output.py b/conan/api/output.py
index 179d983cb13..a01eedfa1c7 100644
--- a/conan/api/output.py
+++ b/conan/api/output.py
@@ -51,6 +51,7 @@ class ConanOutput:
     # Singleton
     _conan_output_level = LEVEL_STATUS
     _silent_warn_tags = []
+    _warnings_as_errors = False
 
     def __init__(self, scope=""):
         self.stream = sys.stderr
@@ -63,6 +64,10 @@ def __init__(self, scope=""):
     def define_silence_warnings(cls, warnings):
         cls._silent_warn_tags = warnings or []
 
+    @classmethod
+    def set_warnings_as_errors(cls, value):
+        cls._warnings_as_errors = value
+
     @classmethod
     def define_log_level(cls, v):
         """
@@ -204,11 +209,16 @@ def success(self, msg):
         return self
 
     def warning(self, msg, warn_tag=None):
-        if self._conan_output_level <= LEVEL_WARNING:
-            if warn_tag is not None and warn_tag in self._silent_warn_tags:
+        def _skip_warning():
+            return (warn_tag is not None and warn_tag in self._silent_warn_tags
+                    or warn_tag is None and "unknown" in self._silent_warn_tags)
+        if self._conan_output_level <= LEVEL_WARNING or (self._warnings_as_errors and self._conan_output_level <= LEVEL_ERROR):
+            if _skip_warning():
                 return self
             warn_tag_msg = "" if warn_tag is None else f"{warn_tag}: "
-            self._write_message(f"WARN: {warn_tag_msg}{msg}", Color.YELLOW)
+            message_type = "ERROR" if self._warnings_as_errors else "WARN"
+            message_color = Color.RED if self._warnings_as_errors else Color.YELLOW
+            self._write_message(f"{message_type}: {warn_tag_msg}{msg}", message_color)
         return self
 
     def error(self, msg):
diff --git a/conan/internal/conan_app.py b/conan/internal/conan_app.py
index b42e16a61fc..dc2190f4d8c 100644
--- a/conan/internal/conan_app.py
+++ b/conan/internal/conan_app.py
@@ -44,8 +44,12 @@ def __init__(self, cache_folder, global_conf):
 
         home_paths = HomePaths(self.cache_folder)
         self.hook_manager = HookManager(home_paths.hooks_path)
-        # Wraps an http_requester to inject proxies, certs, etc
+
+        ConanOutput.set_warnings_as_errors(global_conf.get("core:warnings_as_errors",
+                                                           default=False, check_type=bool))
         ConanOutput.define_silence_warnings(global_conf.get("core:skip_warnings", check_type=list))
+
+        # Wraps an http_requester to inject proxies, certs, etc
         self.requester = ConanRequester(global_conf, cache_folder)
         # To handle remote connections
         rest_client_factory = RestApiClientFactory(self.requester, global_conf)
diff --git a/conans/model/conf.py b/conans/model/conf.py
index 2c6803a35bf..780f93844ef 100644
--- a/conans/model/conf.py
+++ b/conans/model/conf.py
@@ -12,6 +12,7 @@
 BUILT_IN_CONFS = {
     "core:required_conan_version": "Raise if current version does not match the defined range.",
     "core:non_interactive": "Disable interactive user input, raises error if input necessary",
+    "core:warnings_as_errors": "Treat warnings as errors",
     "core:skip_warnings": "Do not show warnings in this list",
     "core:default_profile": "Defines the default host profile ('default' by default)",
     "core:default_build_profile": "Defines the default build profile ('default' by default)",
diff --git a/conans/test/integration/command_v2/test_output.py b/conans/test/integration/command_v2/test_output.py
new file mode 100644
index 00000000000..153192d54b9
--- /dev/null
+++ b/conans/test/integration/command_v2/test_output.py
@@ -0,0 +1,197 @@
+import json
+import textwrap
+
+from conans.test.assets.genconanfile import GenConanfile
+from conans.test.utils.tools import TestClient
+
+
+class TestOutputLevel:
+    def test_invalid_output_level(self):
+        t = TestClient()
+        t.save({"conanfile.py": GenConanfile("foo", "1.0")})
+        t.run("create . -vfooling", assert_error=True)
+        assert "Invalid argument '-vfooling'"
+
+    def test_output_level(self):
+        lines = ("self.output.trace('This is a trace')",
+                 "self.output.debug('This is a debug')",
+                 "self.output.verbose('This is a verbose')",
+                 "self.output.info('This is a info')",
+                 "self.output.highlight('This is a highlight')",
+                 "self.output.success('This is a success')",
+                 "self.output.warning('This is a warning')",
+                 "self.output.error('This is a error')",
+                 )
+
+        t = TestClient()
+        t.save({"conanfile.py": GenConanfile("foo", "1.0").with_package(*lines)})
+
+        # By default, it prints > info
+        t.run("create .")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" not in t.out
+        assert "This is a verbose" not in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # Check if -v argument is equal to VERBOSE level
+        t.run("create . -v")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" not in t.out
+        assert "This is a verbose" in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # Print also verbose traces
+        t.run("create . -vverbose")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" not in t.out
+        assert "This is a verbose" in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # Print also debug traces
+        t.run("create . -vv")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" in t.out
+        assert "This is a verbose" in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+        t.run("create . -vdebug")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" in t.out
+        assert "This is a verbose" in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # Print also "trace" traces
+        t.run("create . -vvv")
+        assert "This is a trace" in t.out
+        assert "This is a debug" in t.out
+        assert "This is a verbose" in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+        t.run("create . -vtrace")
+        assert "This is a trace" in t.out
+        assert "This is a debug" in t.out
+        assert "This is a verbose" in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # With notice
+        t.run("create . -vstatus")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" not in t.out
+        assert "This is a verbose" not in t.out
+        assert "This is a info" in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # With notice
+        t.run("create . -vnotice")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" not in t.out
+        assert "This is a verbose" not in t.out
+        assert "This is a info" not in t.out
+        assert "This is a highlight" in t.out
+        assert "This is a success" in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # With warnings
+        t.run("create . -vwarning")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" not in t.out
+        assert "This is a verbose" not in t.out
+        assert "This is a info" not in t.out
+        assert "This is a highlight" not in t.out
+        assert "This is a success" not in t.out
+        assert "This is a warning" in t.out
+        assert "This is a error" in t.out
+
+        # With errors
+        t.run("create . -verror")
+        assert "This is a trace" not in t.out
+        assert "This is a debug" not in t.out
+        assert "This is a verbose" not in t.out
+        assert "This is a info" not in t.out
+        assert "This is a highlight" not in t.out
+        assert "This is a success" not in t.out
+        assert "This is a warning" not in t.out
+        assert "This is a error" in t.out
+
+
+class TestWarningHandling:
+    lines = ("self.output.warning('Untagged warning')",
+            "self.output.warning('Tagged warning', warn_tag='tag')")
+
+    def test_warning_as_error(self):
+        t = TestClient()
+        t.save({"conanfile.py": GenConanfile("foo", "1.0").with_package(*self.lines)})
+
+        t.save_home({"global.conf": "core:warnings_as_errors=False"})
+        t.run("create . -vwarning")
+        assert "WARN: Untagged warning" in t.out
+        assert "WARN: tag: Tagged warning" in t.out
+
+        t.save_home({"global.conf": "core:warnings_as_errors=True"})
+        t.run("create . -vwarning")
+        assert "ERROR: Untagged warning" in t.out
+        assert "ERROR: tag: Tagged warning" in t.out
+
+        t.save_home({"global.conf": """core:warnings_as_errors=True\ncore:skip_warnings=["tag"]"""})
+        t.run("create . -verror")
+        assert "ERROR: Untagged warning" in t.out
+        assert "ERROR: tag: Tagged warning" not in t.out
+
+        t.save_home({"global.conf": "core:warnings_as_errors=False"})
+        t.run("create . -verror")
+        assert "ERROR: Untagged warning" not in t.out
+        assert "ERROR: tag: Tagged warning" not in t.out
+
+    def test_skip_warnings(self):
+        t = TestClient()
+        t.save({"conanfile.py": GenConanfile("foo", "1.0").with_package(*self.lines)})
+
+        t.save_home({"global.conf": "core:skip_warnings=[]"})
+        t.run("create . -vwarning")
+        assert "WARN: Untagged warning" in t.out
+        assert "WARN: tag: Tagged warning" in t.out
+
+        t.save_home({"global.conf": "core:skip_warnings=['tag']"})
+        t.run("create . -vwarning")
+        assert "WARN: Untagged warning" in t.out
+        assert "WARN: tag: Tagged warning" not in t.out
+
+        t.save_home({"global.conf": "core:skip_warnings=['unknown']"})
+        t.run("create . -vwarning")
+        assert "WARN: Untagged warning" not in t.out
+        assert "WARN: tag: Tagged warning" in t.out
+
+        t.save_home({"global.conf": "core:skip_warnings=['unknown', 'tag']"})
+        t.run("create . -vwarning")
+        assert "WARN: Untagged warning" not in t.out
+        assert "WARN: tag: Tagged warning" not in t.out
diff --git a/conans/test/integration/command_v2/test_output_level.py b/conans/test/integration/command_v2/test_output_level.py
deleted file mode 100644
index f485424924e..00000000000
--- a/conans/test/integration/command_v2/test_output_level.py
+++ /dev/null
@@ -1,144 +0,0 @@
-import json
-
-from conans.test.assets.genconanfile import GenConanfile
-from conans.test.utils.tools import TestClient
-
-
-def test_invalid_output_level():
-    t = TestClient()
-    t.save({"conanfile.py": GenConanfile()})
-    t.run("create . --name foo --version 1.0 -vfooling", assert_error=True)
-    assert "Invalid argument '-vfooling'"
-
-
-def test_output_level():
-
-    lines = ("self.output.trace('This is a trace')",
-             "self.output.debug('This is a debug')",
-             "self.output.verbose('This is a verbose')",
-             "self.output.info('This is a info')",
-             "self.output.highlight('This is a highlight')",
-             "self.output.success('This is a success')",
-             "self.output.warning('This is a warning')",
-             "self.output.error('This is a error')",
-             )
-
-    t = TestClient()
-    t.save({"conanfile.py": GenConanfile().with_package(*lines)})
-
-    # By default, it prints > info
-    t.run("create . --name foo --version 1.0")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" not in t.out
-    assert "This is a verbose" not in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # Check if -v argument is equal to VERBOSE level
-    t.run("create . --name foo --version 1.0 -v")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" not in t.out
-    assert "This is a verbose" in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # Print also verbose traces
-    t.run("create . --name foo --version 1.0 -vverbose")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" not in t.out
-    assert "This is a verbose" in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # Print also debug traces
-    t.run("create . --name foo --version 1.0 -vv")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" in t.out
-    assert "This is a verbose" in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-    t.run("create . --name foo --version 1.0 -vdebug")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" in t.out
-    assert "This is a verbose" in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # Print also "trace" traces
-    t.run("create . --name foo --version 1.0 -vvv")
-    assert "This is a trace" in t.out
-    assert "This is a debug" in t.out
-    assert "This is a verbose" in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-    t.run("create . --name foo --version 1.0 -vtrace")
-    assert "This is a trace" in t.out
-    assert "This is a debug" in t.out
-    assert "This is a verbose" in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # With notice
-    t.run("create . --name foo --version 1.0 -vstatus")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" not in t.out
-    assert "This is a verbose" not in t.out
-    assert "This is a info" in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # With notice
-    t.run("create . --name foo --version 1.0 -vnotice")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" not in t.out
-    assert "This is a verbose" not in t.out
-    assert "This is a info" not in t.out
-    assert "This is a highlight" in t.out
-    assert "This is a success" in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # With warnings
-    t.run("create . --name foo --version 1.0 -vwarning")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" not in t.out
-    assert "This is a verbose" not in t.out
-    assert "This is a info" not in t.out
-    assert "This is a highlight" not in t.out
-    assert "This is a success" not in t.out
-    assert "This is a warning" in t.out
-    assert "This is a error" in t.out
-
-    # With errors
-    t.run("create . --name foo --version 1.0 -verror")
-    assert "This is a trace" not in t.out
-    assert "This is a debug" not in t.out
-    assert "This is a verbose" not in t.out
-    assert "This is a info" not in t.out
-    assert "This is a highlight" not in t.out
-    assert "This is a success" not in t.out
-    assert "This is a warning" not in t.out
-    assert "This is a error" in t.out


Code review comments:
diff_hunk: @@ -204,11 +209,16 @@ def success(self, msg):
         return self
 
     def warning(self, msg, warn_tag=None):
-        if self._conan_output_level <= LEVEL_WARNING:
-            if warn_tag is not None and warn_tag in self._silent_warn_tags:
+        def _skip_warning():
+            return (warn_tag is not None and warn_tag in self._silent_warn_tags
+                    or warn_tag is None and "unknown" in self._silent_warn_tags)
+        if self._conan_output_level <= LEVEL_WARNING or (self._warnings_as_errors and self._conan_output_level <= LEVEL_ERROR):
+            if _skip_warning():
                 return self
             warn_tag_msg = "" if warn_tag is None else f"{warn_tag}: "
-            self._write_message(f"WARN: {warn_tag_msg}{msg}", Color.YELLOW)
+            message_type = "ERROR" if self._warnings_as_errors else "WARN"
+            message_color = Color.RED if self._warnings_as_errors else Color.YELLOW
+            self._write_message(f"{message_type}: {warn_tag_msg}{msg}", message_color)
path: conan/api/output.py
text: @user1:
Not raising?

@author:
We need to check this in more depth. If we raise here, it begs the question as to why normal `error`s do not raise in some cases. That's the main reason I didn't do that in the first place.

Maybe we need to come up with the idea of `core:raise_on_all_errors`, or knowing which errors should raise on print or something? let's look into it


@user1:
``error()`` messages should be there mostly for 2 reasons:
- They will end up raising an exception, but we want to print a bunch of error messages before raising the exception instead of creating a huge exception message, sometimes a bit more difficult to read
- They are legacy, something was bad, but we couldn't really raise because of risk of breaking users, but we still wanted to print a red error message.

So maybe we can assume that module the legacy, error messages are only printed before exceptions?

diff_hunk: @@ -44,8 +44,12 @@ def __init__(self, cache_folder, global_conf):
 
         home_paths = HomePaths(self.cache_folder)
         self.hook_manager = HookManager(home_paths.hooks_path)
-        # Wraps an http_requester to inject proxies, certs, etc
+
+        ConanOutput.set_warnings_as_errors(global_conf.get("core:warnings_as_errors",
+                                                           default=False, check_type=bool))
path: conan/internal/conan_app.py
text: @user1:
Maybe unify ``ConanOutput.configure(global_conf)``?