Repository: ckan/ckan

Issue:
Replacing MD5 hashing algorithm with SHA512
In file: common_middleware.py, method: __call__, the used hashing algorithm is no longer considered secure because it is possible to have collisions. This can lead to brute force attempt to find two or more inputs that produce the same hash. iCR suggested that safer alternative hash algorithms, such as SHA-256, SHA-512, SHA-3 are used. 

In the file, MD5 is used to generate a key based on several parameters and inserted into the database as `user_key`. In that case, it's recommended to use a more secure, less collision prone hash function such as- SHA256 or SHA512.


### Sponsorship and Support:

This work is done by the security researchers from OpenRefactory and is supported by the [Open Source Security Foundation (OpenSSF)](https://openssf.org/): [Project Alpha-Omega](https://alpha-omega.dev/). Alpha-Omega is a project partnering with open source software project maintainers to systematically find new, as-yet-undiscovered vulnerabilities in open source code - and get them fixed – to improve global software supply chain security.

The bug is found by running the Intelligent Code Repair (iCR) tool by OpenRefactory and then manually triaging the results.


Discussion:
Sorry for leaving it without context. Let me explain myself.

Initially, I replaced the `md5` method used in file [common_middleware.py](https://github.com/ckan/ckan/blob/master/ckan/config/middleware/common_middleware.py#L59) with `sha512` for reducing the probability of hash collision, since Md5 is long deprecated.

However, after the first CI run, tests were failing because the database has a field limit of 100 characters for the `user_key` field. As a result, I changed `md5` to `sha3_256` which can be contained within 100 characters limit. However, this time the tests related to the changes passed but some failed (something related to HTTP 500, where 409 was expected).

Please suggest what can be done from my end.

Thanks 
I don't think that the ckanext/datastore/tests/test_create.py::TestDatastoreCreate::test_create_alias_twice test is related -- there was a recent patch for an ordering dependent issue on that flakey test. 

In terms of the issue, I have a few notes:

1) Hash collisions here aren't dangerous, and could arguably be privacy enhancing. A collision would lead to undercounting distinct views.  The code is creating an opaque, somewhat stable identifier that's not PII that can be used to dedup counts of views of datasets. We could replace this with something like truncated md5 with no change in any security stance. 

2) Changing this hash function will invalidate any existing mappings of user tuple->count, but only to the extent that a browser update cycle will. This tends to overcount views.

3) There's a performance difference to sha3, though it's unclear if there's enough of a difference to be material. 

4) Perhaps for python 3.9+, we could pass in `usedforsecurity=False` in the hash constructor to indicate that usage.
Thank you for giving such a detailed explanation.

The use of `md5` to generate `user_key` to track view counts of a dataset- makes sense. Also, that answers my question. 

I think your point #4 can be implemented to indicate the usage. I think in that case, the project would need to change it's requirement (Python 3.9+). In that case, are you planning to add a PR for it?
I wouldn't bump to 3.9 for this, as it's essentially a no-op in the functional space, it's only an indication that we've considered the issue and we explicitly aren't considering this to be a security context. In this case, either we select based on python version, or leave explicit code comments to that effect to make the change when the version gets bumped. 

We're likely to bump the minimum requirement to 3.9 around EOY2024, as that's when 3.8 goes out of security support upstream. 
Well, since this isn't a security threat as you mentioned earlier, I think the `usedforsecurity=False` option can be added after you bump the min requirement for **Python to 3.9 around end of 2024.**

Resolving patch:
diff --git a/ckanext/tracking/middleware.py b/ckanext/tracking/middleware.py
--- a/ckanext/tracking/middleware.py
+++ b/ckanext/tracking/middleware.py
@@ -27,7 +27,9 @@ def track_request(response: Response) -> Response:
             request.environ.get('HTTP_ACCEPT_LANGUAGE', ''),
             request.environ.get('HTTP_ACCEPT_ENCODING', ''),
         ])
-        key = hashlib.md5(key.encode()).hexdigest()
+        # raises a type error on python<3.9
+        h = hashlib.new('md5', usedforsecurity=False)  # type: ignore
+        key = h.update(key.encode()).hexdigest()
         # store key/data here
         sql = '''INSERT INTO tracking_raw
                     (user_key, url, tracking_type)


Test patch: