Repository: PyCQA/bandit (python)

Issue:
Replace `toml` dependency with `tomli`
**Is your feature request related to a problem? Please describe.**
Support for defining bandit options in `pyproject.toml` was added in https://github.com/PyCQA/bandit/pull/401, using `toml` library.

`toml` contributions [have been stagnating](https://github.com/uiri/toml/graphs/contributors), and the last commit dates back (at the time of writing) to [November 1st, 2020](https://github.com/uiri/toml/commit/3f637dba5f68db63d4b30967fedda51c82459471).

`tomli` library is another library that has multiple advantages over `toml`:
- it is well maintained, as can be seen in the [contributions graph](https://github.com/hukkin/tomli/graphs/contributors)
- it supports version `1.0.0` of the TOML specification (`toml` only supports `0.5.0`, as can be seen [here](https://github.com/uiri/toml/blob/3f637dba5f68db63d4b30967fedda51c82459471/RELEASE.rst) and in https://github.com/uiri/toml/issues/300)
- [PEP 680](https://www.python.org/dev/peps/pep-0680/), which will add a TOML parser to Python 3.11 stdlib, has been accepted, and the code will be based on `tomli` library, as specified [here](https://www.python.org/dev/peps/pep-0680/#rationale), so this would be easier to conditionally import either `tomli` or `tomllib` based on the Python version, as APIs should be identical
- a lot of widely used Python projects have switched to it ([black](https://github.com/psf/black/blob/main/setup.py#L102), [mypy](https://github.com/python/mypy/blob/b0c44617426acaaba759b9ad9d3c03d9a237d2af/setup.py#L201), [pytest](https://github.com/pytest-dev/pytest/blob/5599c5ad4589994c7816364139d6bedae4c5fd46/setup.cfg#L49), [pip](https://github.com/pypa/pip/tree/e68ac94595aa3af9c8981479ce305fdb4143c8b6/src/pip/_vendor/tomli), ...), for some of the reasons mentioned above, so this would also make the ecosystem more consistent, especially for people that use features of TOML 1.0 in their `pyproject.toml`

**Describe the solution you'd like**
Relying on `tomli` library for parsing options defined in `pyproject.toml` instead of `toml` library.

**Describe alternatives you've considered**
Keep relying on `toml` for parsing options defined in `pyproject.toml`.

**Additional context**
n/a

Pull request description:
Replace `toml` with `tomli`
Resolves #828 

Replace `toml` with `tomli` for parsing bandit options defined in `pyproject.toml`.

The reason for requiring at least `1.1.0` is because support for text files has been [deprecated in 1.2.0](https://github.com/hukkin/tomli/blob/d21aa80b441fb46361386c44f92985b4921b9b8b/CHANGELOG.md#120) then [removed in 2.0.0](https://github.com/hukkin/tomli/blob/d21aa80b441fb46361386c44f92985b4921b9b8b/CHANGELOG.md#200), and support for binary files has only been [added in 1.1.0](https://github.com/hukkin/tomli/blob/d21aa80b441fb46361386c44f92985b4921b9b8b/CHANGELOG.md#110), so this ensures that users have `1.1.0+`, to reliably be able to read binary files.

As `tomli` only reads binary files, it is required to open the file using `rb` flags, which also impacts the format passed to `safe_load` for YAML files, but `PyYAML` accepts both binary and text files, so this still works as expected, though this may slow down a bit parsing time.

Patch:
diff --git a/bandit/blacklists/calls.py b/bandit/blacklists/calls.py
index 65e471d..c81e1cd 100644
--- a/bandit/blacklists/calls.py
+++ b/bandit/blacklists/calls.py
@@ -137,6 +137,8 @@ be reviewed.
 B309: httpsconnection
 ---------------------
 
+The check for this call has been removed.
+
 Use of HTTPSConnection on older versions of Python prior to 2.7.9 and 3.4.3 do
 not provide security, see https://wiki.openstack.org/wiki/OSSN/OSSN-0033
 
@@ -297,6 +299,8 @@ behavior that does not validate certificates or perform hostname checks.
 B325: tempnam
 --------------
 
+The check for this call has been removed.
+
 Use of os.tempnam() and os.tmpnam() is vulnerable to symlink attacks. Consider
 using tmpfile() instead.
 
@@ -479,21 +483,7 @@ def gen_blacklist():
         )
     )
 
-    sets.append(
-        utils.build_conf_dict(
-            "httpsconnection",
-            "B309",
-            issue.Cwe.CLEARTEXT_TRANSMISSION,
-            [
-                "httplib.HTTPSConnection",
-                "http.client.HTTPSConnection",
-                "six.moves.http_client.HTTPSConnection",
-            ],
-            "Use of HTTPSConnection on older versions of Python prior to 2.7.9"
-            " and 3.4.3 do not provide security, see "
-            "https://wiki.openstack.org/wiki/OSSN/OSSN-0033",
-        )
-    )
+    # skipped B309 as the check for a call to httpsconnection has been removed
 
     sets.append(
         utils.build_conf_dict(
@@ -696,15 +686,7 @@ def gen_blacklist():
 
     # skipped B324 (used in bandit/plugins/hashlib_new_insecure_functions.py)
 
-    sets.append(
-        utils.build_conf_dict(
-            "tempnam",
-            "B325",
-            issue.Cwe.INSECURE_TEMP_FILE,
-            ["os.tempnam", "os.tmpnam"],
-            "Use of os.tempnam() and os.tmpnam() is vulnerable to symlink "
-            "attacks. Consider using tmpfile() instead.",
-        )
-    )
+    # skipped B325 as the check for a call to os.tempnam and os.tmpnam have
+    # been removed
 
     return {"Call": sets}
diff --git a/bandit/core/config.py b/bandit/core/config.py
index 7c259bc..236f357 100644
--- a/bandit/core/config.py
+++ b/bandit/core/config.py
@@ -3,13 +3,17 @@
 #
 # SPDX-License-Identifier: Apache-2.0
 import logging
+import sys
 
 import yaml
 
-try:
-    import toml
-except ImportError:
-    toml = None
+if sys.version_info >= (3, 11):
+    import tomllib
+else:
+    try:
+        import tomli as tomllib
+    except ImportError:
+        tomllib = None
 
 from bandit.core import constants
 from bandit.core import extension_loader
@@ -34,14 +38,14 @@ class BanditConfig:
 
         if config_file:
             try:
-                f = open(config_file)
+                f = open(config_file, "rb")
             except OSError:
                 raise utils.ConfigError(
                     "Could not read config file.", config_file
                 )
 
             if config_file.endswith(".toml"):
-                if toml is None:
+                if tomllib is None:
                     raise utils.ConfigError(
                         "toml parser not available, reinstall with toml extra",
                         config_file,
@@ -49,8 +53,8 @@ class BanditConfig:
 
                 try:
                     with f:
-                        self._config = toml.load(f)["tool"]["bandit"]
-                except toml.TomlDecodeError as err:
+                        self._config = tomllib.load(f)["tool"]["bandit"]
+                except tomllib.TOMLDecodeError as err:
                     LOG.error(err)
                     raise utils.ConfigError("Error parsing file.", config_file)
             else:
diff --git a/bandit/core/constants.py b/bandit/core/constants.py
index 4019cab..dd8ddeb 100644
--- a/bandit/core/constants.py
+++ b/bandit/core/constants.py
@@ -5,9 +5,6 @@
 # default plugin name pattern
 plugin_name_pattern = "*.py"
 
-# default progress increment
-progress_increment = 50
-
 RANKING = ["UNDEFINED", "LOW", "MEDIUM", "HIGH"]
 RANKING_VALUES = {"UNDEFINED": 1, "LOW": 3, "MEDIUM": 5, "HIGH": 10}
 CRITERIA = [("SEVERITY", "UNDEFINED"), ("CONFIDENCE", "UNDEFINED")]
diff --git a/bandit/core/manager.py b/bandit/core/manager.py
index 8a4bdc6..4e07517 100644
--- a/bandit/core/manager.py
+++ b/bandit/core/manager.py
@@ -13,6 +13,8 @@ import sys
 import tokenize
 import traceback
 
+from rich import progress
+
 from bandit.core import constants as b_constants
 from bandit.core import extension_loader
 from bandit.core import issue
@@ -25,6 +27,7 @@ from bandit.core import test_set as b_test_set
 LOG = logging.getLogger(__name__)
 NOSEC_COMMENT = re.compile(r"#\s*nosec:?\s*(?P<tests>[^#]+)?#?")
 NOSEC_COMMENT_TESTS = re.compile(r"(?:(B\d+|[a-z_]+),?)+", re.IGNORECASE)
+PROGRESS_THRESHOLD = 50
 
 
 class BanditManager:
@@ -69,9 +72,6 @@ class BanditManager:
         self.agg_type = agg_type
         self.metrics = metrics.Metrics()
         self.b_ts = b_test_set.BanditTestSet(config, profile)
-
-        # set the increment of after how many files to show progress
-        self.progress = b_constants.progress_increment
         self.scores = []
 
     def get_skipped(self):
@@ -263,19 +263,20 @@ class BanditManager:
 
         :return: -
         """
-        self._show_progress("%s [" % len(self.files_list))
-
         # if we have problems with a file, we'll remove it from the files_list
         # and add it to the skipped list instead
         new_files_list = list(self.files_list)
+        if (
+            len(self.files_list) > PROGRESS_THRESHOLD
+            and LOG.getEffectiveLevel() <= logging.INFO
+        ):
+            files = progress.track(self.files_list)
+        else:
+            files = self.files_list
 
-        for count, fname in enumerate(self.files_list):
+        for count, fname in enumerate(files):
             LOG.debug("working on file : %s", fname)
 
-            if len(self.files_list) > self.progress:
-                # is it time to update the progress indicator?
-                if count % self.progress == 0:
-                    self._show_progress("%s.. " % count, flush=True)
             try:
                 if fname == "-":
                     open_fd = os.fdopen(sys.stdin.fileno(), "rb", 0)
@@ -291,32 +292,12 @@ class BanditManager:
                 self.skipped.append((fname, e.strerror))
                 new_files_list.remove(fname)
 
-        self._show_progress("]\n", flush=True)
-
         # reflect any files which may have been skipped
         self.files_list = new_files_list
 
         # do final aggregation of metrics
         self.metrics.aggregate()
 
-    def _show_progress(self, message, flush=False):
-        """Show progress on stderr
-
-        Write progress message to stderr, if number of files warrants it and
-        log level is high enough.
-
-        :param message: The message to write to stderr
-        :param flush: Whether to flush stderr after writing the message
-        :return:
-        """
-        if (
-            len(self.files_list) > self.progress
-            and LOG.getEffectiveLevel() <= logging.INFO
-        ):
-            sys.stderr.write(message)
-            if flush:
-                sys.stderr.flush()
-
     def _parse_file(self, fname, fdata, new_files_list):
         try:
             # parse the current file
@@ -340,11 +321,7 @@ class BanditManager:
                 pass
             score = self._execute_ast_visitor(fname, fdata, data, nosec_lines)
             self.scores.append(score)
-            self.metrics.count_issues(
-                [
-                    score,
-                ]
-            )
+            self.metrics.count_issues([score])
         except KeyboardInterrupt:
             sys.exit(2)
         except SyntaxError:
diff --git a/bandit/plugins/crypto_request_no_cert_validation.py b/bandit/plugins/crypto_request_no_cert_validation.py
index 5121d74..223d421 100644
--- a/bandit/plugins/crypto_request_no_cert_validation.py
+++ b/bandit/plugins/crypto_request_no_cert_validation.py
@@ -13,17 +13,17 @@ identity of the party you are communicating with.  This is accomplished by one
 or both parties presenting trusted certificates during the connection
 initialization phase of TLS.
 
-When request methods are used certificates are validated automatically which is
-the desired behavior.  If certificate validation is explicitly turned off
-Bandit will return a HIGH severity error.
+When HTTPS request methods are used, certificates are validated automatically
+which is the desired behavior.  If certificate validation is explicitly turned
+off Bandit will return a HIGH severity error.
 
 
 :Example:
 
 .. code-block:: none
 
-    >> Issue: [request_with_no_cert_validation] Requests call with verify=False
-    disabling SSL certificate checks, security issue.
+    >> Issue: [request_with_no_cert_validation] Call to requests with
+    verify=False disabling SSL certificate checks, security issue.
        Severity: High   Confidence: High
        CWE: CWE-295 (https://cwe.mitre.org/data/definitions/295.html)
        Location: examples/requests-ssl-verify-disabled.py:4
@@ -42,6 +42,9 @@ Bandit will return a HIGH severity error.
 .. versionchanged:: 1.7.3
     CWE information added
 
+.. versionchanged:: 1.7.5
+    Added check for httpx module
+
 """
 import bandit
 from bandit.core import issue
@@ -51,17 +54,22 @@ from bandit.core import test_properties as test
 @test.checks("Call")
 @test.test_id("B501")
 def request_with_no_cert_validation(context):
-    http_verbs = ("get", "options", "head", "post", "put", "patch", "delete")
+    HTTP_VERBS = ("get", "options", "head", "post", "put", "patch", "delete")
+    HTTPX_ATTRS = ("request", "stream", "Client", "AsyncClient") + HTTP_VERBS
+    qualname = context.call_function_name_qual.split(".")[0]
+
     if (
-        "requests" in context.call_function_name_qual
-        and context.call_function_name in http_verbs
+        qualname == "requests"
+        and context.call_function_name in HTTP_VERBS
+        or qualname == "httpx"
+        and context.call_function_name in HTTPX_ATTRS
     ):
         if context.check_call_arg_value("verify", "False"):
             return bandit.Issue(
                 severity=bandit.HIGH,
                 confidence=bandit.HIGH,
                 cwe=issue.Cwe.IMPROPER_CERT_VALIDATION,
-                text="Requests call with verify=False disabling SSL "
+                text=f"Call to {qualname} with verify=False disabling SSL "
                 "certificate checks, security issue.",
                 lineno=context.get_lineno_for_call_arg("verify"),
             )
diff --git a/bandit/plugins/hashlib_insecure_functions.py b/bandit/plugins/hashlib_insecure_functions.py
index 669642c..a60bd1e 100644
--- a/bandit/plugins/hashlib_insecure_functions.py
+++ b/bandit/plugins/hashlib_insecure_functions.py
@@ -63,8 +63,8 @@ def _hashlib_func(context):
                         severity=bandit.HIGH,
                         confidence=bandit.HIGH,
                         cwe=issue.Cwe.BROKEN_CRYPTO,
-                        text="Use of weak MD4, MD5, or SHA1 hash for "
-                        "security. Consider usedforsecurity=False",
+                        text=f"Use of weak {func.upper()} hash for security. "
+                        "Consider usedforsecurity=False",
                         lineno=context.node.lineno,
                     )
             elif func == "new":
@@ -76,7 +76,7 @@ def _hashlib_func(context):
                             severity=bandit.HIGH,
                             confidence=bandit.HIGH,
                             cwe=issue.Cwe.BROKEN_CRYPTO,
-                            text="Use of weak MD4, MD5, or SHA1 hash for "
+                            text=f"Use of weak {name.upper()} hash for "
                             "security. Consider usedforsecurity=False",
                             lineno=context.node.lineno,
                         )
@@ -96,7 +96,7 @@ def _hashlib_new(context):
                     severity=bandit.MEDIUM,
                     confidence=bandit.HIGH,
                     cwe=issue.Cwe.BROKEN_CRYPTO,
-                    text="Use of insecure MD4, MD5, or SHA1 hash function.",
+                    text=f"Use of insecure {name.upper()} hash function.",
                     lineno=context.node.lineno,
                 )
 
diff --git a/doc/source/plugins/b324_hashlib.rst b/doc/source/plugins/b324_hashlib.rst
new file mode 100644
index 0000000..9210ee5
--- /dev/null
+++ b/doc/source/plugins/b324_hashlib.rst
@@ -0,0 +1,5 @@
+-------------
+B324: hashlib
+-------------
+
+.. automodule:: bandit.plugins.hashlib_insecure_functions
diff --git a/examples/httplib_https.py b/examples/httplib_https.py
deleted file mode 100644
index f5de3f9..0000000
--- a/examples/httplib_https.py
+++ /dev/null
@@ -1,8 +0,0 @@
-import httplib
-c = httplib.HTTPSConnection("example.com")
-
-import http.client
-c = http.client.HTTPSConnection("example.com")
-
-import six
-six.moves.http_client.HTTPSConnection("example.com")
diff --git a/examples/requests-ssl-verify-disabled.py b/examples/requests-ssl-verify-disabled.py
index c2f641a..e314036 100644
--- a/examples/requests-ssl-verify-disabled.py
+++ b/examples/requests-ssl-verify-disabled.py
@@ -1,5 +1,7 @@
+import httpx
 import requests
 
+
 requests.get('https://gmail.com', verify=True)
 requests.get('https://gmail.com', verify=False)
 requests.post('https://gmail.com', verify=True)
@@ -14,3 +16,26 @@ requests.options('https://gmail.com', verify=True)
 requests.options('https://gmail.com', verify=False)
 requests.head('https://gmail.com', verify=True)
 requests.head('https://gmail.com', verify=False)
+
+httpx.request('GET', 'https://gmail.com', verify=True)
+httpx.request('GET', 'https://gmail.com', verify=False)
+httpx.get('https://gmail.com', verify=True)
+httpx.get('https://gmail.com', verify=False)
+httpx.options('https://gmail.com', verify=True)
+httpx.options('https://gmail.com', verify=False)
+httpx.head('https://gmail.com', verify=True)
+httpx.head('https://gmail.com', verify=False)
+httpx.post('https://gmail.com', verify=True)
+httpx.post('https://gmail.com', verify=False)
+httpx.put('https://gmail.com', verify=True)
+httpx.put('https://gmail.com', verify=False)
+httpx.patch('https://gmail.com', verify=True)
+httpx.patch('https://gmail.com', verify=False)
+httpx.delete('https://gmail.com', verify=True)
+httpx.delete('https://gmail.com', verify=False)
+httpx.stream('https://gmail.com', verify=True)
+httpx.stream('https://gmail.com', verify=False)
+httpx.Client()
+httpx.Client(verify=False)
+httpx.AsyncClient()
+httpx.AsyncClient(verify=False)
diff --git a/examples/tempnam.py b/examples/tempnam.py
deleted file mode 100644
index bc92f03..0000000
--- a/examples/tempnam.py
+++ /dev/null
@@ -1,13 +0,0 @@
-from os import tempnam
-from os import tmpnam
-import os
-
-os.tmpnam()
-
-tmpnam()
-
-os.tempnam('dir1')
-os.tempnam('dir1', 'prefix1')
-
-tempnam('dir1')
-tempnam('dir1', 'prefix1')
diff --git a/requirements.txt b/requirements.txt
index 749ba56..9947620 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -5,3 +5,4 @@ GitPython>=1.0.1 # BSD License (3 clause)
 PyYAML>=5.3.1 # MIT
 stevedore>=1.20.0 # Apache-2.0
 colorama>=0.3.9;platform_system=="Windows" # BSD License (3 clause)
+rich # MIT
diff --git a/setup.cfg b/setup.cfg
index f00237e..7449f15 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -22,6 +22,7 @@ classifier =
     Programming Language :: Python :: 3.8
     Programming Language :: Python :: 3.9
     Programming Language :: Python :: 3.10
+    Programming Language :: Python :: 3 :: Only
     Topic :: Security
 project_urls =
     Release notes = https://github.com/PyCQA/bandit/releases
@@ -30,7 +31,7 @@ project_urls =
 yaml =
     PyYAML
 toml =
-    toml
+    tomli>=1.1.0; python_version < "3.11"
 
 [entry_points]
 console_scripts =


Test patch:
diff --git a/test-requirements.txt b/test-requirements.txt
index b2e3c37..6c8c57e 100644
--- a/test-requirements.txt
+++ b/test-requirements.txt
@@ -7,6 +7,6 @@ flake8>=4.0.0 # Apache-2.0
 stestr>=2.5.0 # Apache-2.0
 testscenarios>=0.5.0 # Apache-2.0/BSD
 testtools>=2.3.0 # MIT
-toml # MIT
+tomli>=1.1.0;python_version<"3.11" # MIT
 beautifulsoup4>=4.8.0 # MIT
 pylint==1.9.4 # GPLv2
diff --git a/tests/functional/test_functional.py b/tests/functional/test_functional.py
index c45afa5..29eb9b9 100644
--- a/tests/functional/test_functional.py
+++ b/tests/functional/test_functional.py
@@ -196,14 +196,6 @@ class FunctionalTests(testtools.TestCase):
         }
         self.check_example("hardcoded-tmp.py", expect)
 
-    def test_httplib_https(self):
-        """Test for `httplib.HTTPSConnection`."""
-        expect = {
-            "SEVERITY": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 3, "HIGH": 0},
-            "CONFIDENCE": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 0, "HIGH": 3},
-        }
-        self.check_example("httplib_https.py", expect)
-
     def test_imports_aliases(self):
         """Test the `import X as Y` syntax."""
         if sys.version_info >= (3, 9):
@@ -284,14 +276,6 @@ class FunctionalTests(testtools.TestCase):
         }
         self.check_example("mktemp.py", expect)
 
-    def test_tempnam(self):
-        """Test for `os.tempnam` / `os.tmpnam`."""
-        expect = {
-            "SEVERITY": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 6, "HIGH": 0},
-            "CONFIDENCE": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 0, "HIGH": 6},
-        }
-        self.check_example("tempnam.py", expect)
-
     def test_nonsense(self):
         """Test that a syntactically invalid module is skipped."""
         self.run_example("nonsense.py")
@@ -404,8 +388,8 @@ class FunctionalTests(testtools.TestCase):
     def test_requests_ssl_verify_disabled(self):
         """Test for the `requests` library skipping verification."""
         expect = {
-            "SEVERITY": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 0, "HIGH": 7},
-            "CONFIDENCE": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 0, "HIGH": 7},
+            "SEVERITY": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 0, "HIGH": 18},
+            "CONFIDENCE": {"UNDEFINED": 0, "LOW": 0, "MEDIUM": 0, "HIGH": 18},
         }
         self.check_example("requests-ssl-verify-disabled.py", expect)