Repository: certbot/certbot

Issue:
Use sha256 for account ids 
md5 is an artifact from days long gone - so much so that it isn't supported at all by web crypto.

I'm working on a [node.js (and browser) clone of letsencrypt](https://git.daplie.com/Daplie/node-greenlock) that works on all platforms - including OS X and Windows (complete) and the browser (incomplete) and I don't know what every possible security consideration of an account id might be, but I'd like to stay 100% compatible and I'd also rather be safe than sorry.



Discussion:
Our use of `md5` is explained [here](https://github.com/letsencrypt/letsencrypt/pull/1013#issuecomment-149983479).

> Point about md5 is that it is short (it's a filename, accessible by users); collisions are very unlikely to happen anyway and we don't need any other crypto property for it. @kuba

I don't see that as valid reasoning. How is being short an advantage? And how is "currently we don't think it will be a security issue, so where's it's 6 of one and half a dozen of the other, let's err on the side where if our current thought processes and current perceived use cases are incorrect, it could become a big deal" a sound practice?

Let me rephrase that in the speak of days of yore:

> We can just use unauthenticated plain text. It's just between local systems on a trusted network, so we don't really need authentication or encryption.

You would have thought the people designing email would have considered the future consequences of their actions, but they didn't. They were only thinking about their current use cases. It wasn't a big deal... until it was (as it often is with security).

Being that there are only advantages and no disadvantages, I side with @jsha:

> That said, if we're going to break compatibility, can we also switch the hash to something other than md5? I realize this isn't a security-critical hash, **but it's nice to not have to think about it**.

I don't see anything else in the thread about md5.

Perhaps the next version could simply hash both md5 and sha256, try sha256 first and fall back to md5 (and rewrite as sha256 at that time), keep that code in for the next year or so, and then drop md5 altogether.

It's been a while since I've written python, but I feel very confident I could put together a pull request that makes that 5-10 line change.

Also, users don't use the **account id**s, machines do.

Users use their **email address** and their **domain name**.

I peeked at the python client here and there to learn a few things, so I'm not sure exactly how you guys do it, but in my case the renewal path is `:config/renewal/`**:hostname**`.conf` which stores the **account id** and then is looked up in `:config/accounts/:acme_discovery_url/:account_id`.

Furthermore, if a user did have to interact with it, it's not anyone is sitting next to their coworker and saying "Hey Bob, it's alpha 8 alpha 1 7 0 charlie delta 6 ...." so the difference between 32 and 64 characters that have to be emailed or communicated by other copy/paste means is of insignificant concern to a user.

Neither is accessible. Both have to be selected and copied. Both would be mapped to a domain and or email from any user interface. The interface designer has 0 extra burden for the length of the id (unless they happen to show it but I would imagine that, like github, they would give it a user-defined name or hostname instead).

There are people that I've talked to (outside of my own company) that are building shared hosting systems with letsencrypt. I'm very willing to bet that some of them will be unwittingly use hashes of private keys in some strange way for authentication. (touting security as a marketing move and providing free ssl does not a security expert make). The wild is wild.

For more context on why `md5` was picked. We considered with @jdkasten  scenarios where users would move account data between machines - that's where incremental integer fails. On the other hand I didn't want to trust CA to provide unique registration IDs. That's why hash was picked. For longer hashes please consider that some filesystems might have character limits for filenames :P Nevertheless, it's very hard for me to believe that `md5` is not available in JavaScript...

@kuba 

> For longer hashes please consider that some filesystems might have character limits for filenames

MD5 creates 32 hexadecimal character long filenames and if we assume SHA-256 is safe enough, we would end up with a 64 character long hexadecimal file name. If we look at a [list of file system limits](https://en.wikipedia.org/wiki/Comparison_of_file_systems#Limits), we can conclude the ONLY two file systems affected by this change would be (assuming this list is complete ánd correct):
- FATX;
- Minix V3 FS.

All other file systems in that list either wouldn't fit MD5 anyway ór would fit a SHA-256 hash.

So unless someone is trying to install Let's Encrypt on a XBox or a system with the Minix V3 file system (never heard of it :stuck_out_tongue:), you're good to go with SHA-256.

I get an error when I tried to run certbot on a FIPS enabled system (CentOS 7.3 on AWS). 

```
[root@ip-10-100-140-13 ~]# certbot certonly --dry-run --webroot -w /var/www/hiawatha/ -d my.domain.com 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel):engops@bluemedora.com
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org

-------------------------------------------------------------------------------
Please read the Terms of Service at
https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf. You must agree
in order to register with the ACME server at
https://acme-staging.api.letsencrypt.org/directory
-------------------------------------------------------------------------------
(A)gree/(C)ancel: A
An unexpected error occurred:
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Please see the logfiles in /var/log/letsencrypt for more details.
[root@ip-10-100-140-13 ~]# cat /var/log/letsencrypt/letsencrypt.log
2017-06-22 14:08:18,659:DEBUG:certbot.main:certbot version: 0.14.1
2017-06-22 14:08:18,659:DEBUG:certbot.main:Arguments: ['--dry-run', '--webroot', '-w', '/var/www/hiawatha/', '-d', 'my.domain.com']
2017-06-22 14:08:18,659:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2017-06-22 14:08:18,673:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer <certbot.cli._Default object at 0x196d810>
2017-06-22 14:08:18,673:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x196aa90>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x196a850>, apache=<certbot.cli._Default object at 0x196db10>, authenticator='webroot', break_my_certs=<certbot.cli._Default object at 0x1965910>, cert_path=<certbot.cli._Default object at 0x196afd0>, certname=<certbot.cli._Default object at 0x1967c50>, chain_path=<certbot.cli._Default object at 0x196d310>, checkpoints=<certbot.cli._Default object at 0x196a310>, config_dir=<certbot.cli._Default object at 0x196d410>, config_file=None, configurator=<certbot.cli._Default object at 0x196d810>, csr=<certbot.cli._Default object at 0x196a810>, debug=<certbot.cli._Default object at 0x1967510>, debug_challenges=<certbot.cli._Default object at 0x1965fd0>, dialog=None, domains='keycloak.bm-engops.com', dry_run=True, duplicate=<certbot.cli._Default object at 0x196ab90>, eff_email=<certbot.cli._Default object at 0x196a190>, email=<certbot.cli._Default object at 0x196a090>, expand=<certbot.cli._Default object at 0x196a490>, force_interactive=<certbot.cli._Default object at 0x1967a50>, fullchain_path=<certbot.cli._Default object at 0x196d210>, func=<function certonly at 0x18e7758>, hsts=<certbot.cli._Default object at 0x1965190>, http01_port=<certbot.cli._Default object at 0x1965a90>, ifaces=<certbot.cli._Default object at 0x196add0>, init=<certbot.cli._Default object at 0x196a110>, installer=<certbot.cli._Default object at 0x196d810>, key_path=<certbot.cli._Default object at 0x196d110>, logs_dir=<certbot.cli._Default object at 0x196d610>, manual=<certbot.cli._Default object at 0x196de10>, manual_auth_hook=<certbot.cli._Default object at 0x1972090>, manual_cleanup_hook=<certbot.cli._Default object at 0x1972210>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x1972310>, must_staple=<certbot.cli._Default object at 0x1965610>, nginx=<certbot.cli._Default object at 0x196dc10>, no_bootstrap=<certbot.cli._Default object at 0x1967c10>, no_self_upgrade=<certbot.cli._Default object at 0x1967e10>, no_verify_ssl=<certbot.cli._Default object at 0x1965e50>, noninteractive_mode=<certbot.cli._Default object at 0x1967950>, num=<certbot.cli._Default object at 0x191ce90>, os_packages_only=<certbot.cli._Default object at 0x1967fd0>, post_hook=<certbot.cli._Default object at 0x191c510>, pre_hook=<certbot.cli._Default object at 0x191c290>, pref_challs=<certbot.cli._Default object at 0x196fcd0>, prepare=<certbot.cli._Default object at 0x196acd0>, quiet=<certbot.cli._Default object at 0x1967a10>, reason=<certbot.cli._Default object at 0x196a510>, redirect=<certbot.cli._Default object at 0x1965490>, register_unsafely_without_email=<certbot.cli._Default object at 0x1967e50>, reinstall=<certbot.cli._Default object at 0x196a390>, renew_by_default=<certbot.cli._Default object at 0x196a650>, renew_hook=<certbot.cli._Default object at 0x191c750>, renew_with_new_domains=<certbot.cli._Default object at 0x196a750>, rsa_key_size=<certbot.cli._Default object at 0x1965790>, server=<certbot.cli._Default object at 0x196d710>, staging=<certbot.cli._Default object at 0x1967810>, standalone=<certbot.cli._Default object at 0x196dd10>, standalone_supported_challenges=<certbot.cli._Default object at 0x1972450>, staple=<certbot.cli._Default object at 0x196f510>, strict_permissions=<certbot.cli._Default object at 0x196fa90>, text_mode=<certbot.cli._Default object at 0x1967850>, tls_sni_01_port=<certbot.cli._Default object at 0x1965c90>, tos=<certbot.cli._Default object at 0x196a990>, uir=<certbot.cli._Default object at 0x196f050>, update_registration=<certbot.cli._Default object at 0x1967f50>, user_agent=<certbot.cli._Default object at 0x196a910>, validate_hooks=<certbot.cli._Default object at 0x191c990>, verb='certonly', verbose_count=<certbot.cli._Default object at 0x1967750>, webroot=True, webroot_map=<certbot.cli._Default object at 0x1972650>, webroot_path='/var/www/hiawatha/', work_dir=<certbot.cli._Default object at 0x196d510>)
```

Replacing `hashlib.md5` with `hashlib.sha256` in account.py fixes it. 

see a similar issue with saltstack [here](https://github.com/saltstack/salt/issues/35862)
Sorry to hear about the problems here on a FIPS enabled system.

We should fix this, either by switching to `sha256` or another approach entirely. Whoever works on this should be careful that the new names work on every system and don't break things for existing installations where `md5` is used.
Another approach would be to skip hashing altogether. Is there ever a case where we have just a public key and need to look up an account based on that?

Given that the vast majority of configs have just a single account, I think it would make sense to just assign accounts sequentially increasing integer ids, "1", "2", etc.
> Another approach would be to skip hashing altogether. Is there ever a case where we have just a public key and need to look up an account based on that?

No Certbot doesn't currently support this.

> Given that the vast majority of configs have just a single account, I think it would make sense to just assign accounts sequentially increasing integer ids, "1", "2", etc.

Kuba briefly touched on the reason he didn't use increasing integer ids [here](https://github.com/certbot/certbot/issues/1948#issuecomment-166883774).
Thanks for the link!

> For more context on why md5 was picked. We considered with @jdkasten scenarios where users would move account data between machines - that's where incremental integer fails. On the other hand I didn't want to trust CA to provide unique registration IDs. That's why hash was picked. For longer hashes please consider that some filesystems might have character limits for filenames :P Nevertheless, it's very hard for me to believe that md5 is not available in JavaScript...

I think this only applies when you are copying an account to a new machine that already has an account, and you want both accounts to coexist. I think in practice this hasn't turned out to be a common practice, and it's certainly not something we officially support. If someone *does* want to do this they can simply rename the directory.
Yes. You are correct that it's only when both machines have an existing account.

I also agree it's not very common, but it is something people do. Here's one example https://community.letsencrypt.org/t/moving-and-merging-certs-from-server-a-to-b/19015. Using an incrementing counter would mean there'd be more path conflicts and if we kept the `account_id` fields in renewal conf files working the same way, they'd have to properly update them to keep the same behavior.

If we decide merging two `/etc/letsencrypt` directories is something we never want to support, I'd be fine with an incrementing counter, but I think there are probably other solutions to this problem that don't force us to make this decision.
Another approach would be to abandon md5 and just generate a random string of characters as a unique id.
What about using an [UUID](https://en.wikipedia.org/wiki/Universally_unique_identifier)? No use reinventing the wheel :wink: 
I'm going to open a PR for this and hope it gets in. I tested it on several RHEL CentOS flavors with no adverse effects.
I've just been replacing `hashlib.md5( ... )` with `hashlib.new('md5', ... , usedforsecurity=False)`, which preserves the original MD5 behavior. It can be put into a try/except block as a fallback for FIPS boxes.

Example: https://github.com/Cisco-Talos/clamav-faq/pull/86/commits/6537e61e427543856905bc5a48acbfda3a9c8040

For new installations, it still might make more sense to just use SHA256, to make it more future-proof.
Surprised this important issue is not fixed. Sounds pretty straightforward to switch to sha256. Meanwhile we need to unFIPS, reboot, certbot, reFIPS, reboot. :(
We've had a couple of conversations in Mattermost about this.

Summary is that https://github.com/certbot/certbot/pull/7416 landed in our 0.40.0 release which should prevent problems for people downgrading their Certbot version if we change this value, however, it's probably worth looking through the code again to verify this.

Despite 0.40.0 being released a couple months ago, I personally think we should probably hold off on changing the value here for at least a couple months. The reason for this is we're currently working towards migrating our users off of certbot-auto which is currently still using Python 2.7 which is no longer maintained by the CPython team. Until we've finished this migration, a lot of downgrades to Certbot versions much older than 0.40.0 (such as 0.31.0 in Ubuntu) are possible so I think we should avoid making this change until we know that it will not negatively affect a large number of Certbot users.
Could we instead make this value configurable? This way existing installations can continue to use md5 and ones that have the issue (FIPS ) can change this value to whatever is supported by FIPS at that moment in time. 
We could, but unless someone has a FIPS system where the suggestion at https://github.com/certbot/certbot/issues/1948#issuecomment-493074434 doesn't work, I'd personally prefer to see a PR for that solution until we can unconditionally change the value.

Resolving patch:
diff --git a/certbot/certbot/_internal/account.py b/certbot/certbot/_internal/account.py
--- a/certbot/certbot/_internal/account.py
+++ b/certbot/certbot/_internal/account.py
@@ -56,11 +56,18 @@ def __init__(self, regr, key, meta=None):
                 tz=pytz.UTC).replace(microsecond=0),
             creation_host=socket.getfqdn()) if meta is None else meta
 
-        self.id = hashlib.md5(
-            self.key.key.public_key().public_bytes(
-                encoding=serialization.Encoding.PEM,
-                format=serialization.PublicFormat.SubjectPublicKeyInfo)
-        ).hexdigest()
+        # try MD5, else use MD5 in non-security mode (e.g. for FIPS systems / RHEL)
+        try:
+            hasher = hashlib.md5()
+        except ValueError:
+            hasher = hashlib.new('md5', usedforsecurity=False) # type: ignore
+
+        hasher.update(self.key.key.public_key().public_bytes(
+            encoding=serialization.Encoding.PEM,
+            format=serialization.PublicFormat.SubjectPublicKeyInfo)
+        )
+
+        self.id = hasher.hexdigest()
         # Implementation note: Email? Multiple accounts can have the
         # same email address. Registration URI? Assigned by the
         # server, not guaranteed to be stable over time, nor


Test patch: