Repository: certbot/certbot

Issue:
Add random sleep inside `certbot renew`
If you use a packaged certbot, it comes with a crontab that includes a random sleep. However, a lot of people are configuring their own crontabs, either with certbot or certbot-auto. They usually run at midnight or the top of some other hour. These create very significant traffic spikes to Let's Encrypt.

I'd like Certbot to build in a random sleep interval when run with `renew`. Even an interval with a min of 1 minute and a max of 4 minutes would probably help a lot, though longer is better. I think there's no need to coordinate with the packaged sleep; if we have one external sleep and one internal sleep, that's fine.

@ohemorange points out that we need to make sure this runs before the hooks, so people with hooks that take down their web server don't have unnecessarily long outages.

Also, if we want to get fancy, we could insert the random sleep only when not being run from a terminal.


Discussion:
@jsha, do you have some log data indicating what fraction of the load spikes are attributable to Certbot as opposed to other clients? What does the top-of-the-hour-renewal client distribution look like?
I don't have it on hand, but I did some investigation a while ago, and found that it was pretty heavily Certbot. Emailing a few of the users, I found that they had all installed a crontab by hand.
> Also, if we want to get fancy, we could insert the random sleep only when not being run from a terminal.

This is evidently as easy as

`if os.isatty(0):`

I tried

```
#!/usr/bin/env python

import os

open("/tmp/log", "w").write(str(os.isatty(0)) + "\n")
```

Running this program from the command line produced a `/tmp/log` containing the string `True`; creating a crontab to run it caused the contents to change to `False` as soon as the job had run.
@ohemorange tells me we already do this detection somewhere; we should reuse that.
I think it's in `log.py` and `reporter.py` where we check `sys.stderr.isatty()` and `sys.stdout.isatty()`, but the result isn't necessarily exported in a convenient way for other code to use. However, we could use exactly the same method that those tests use.
OK, I have a proof of concept of this. Two other questions:

* should we announce in the logs that this sleep is happening?
* should OS packagers disable this feature in their OS packages so that they don't have the duplicative sleeps?

Resolving patch:
diff --git a/certbot/main.py b/certbot/main.py
--- a/certbot/main.py
+++ b/certbot/main.py
@@ -4,7 +4,9 @@
 import functools
 import logging.handlers
 import os
+import random
 import sys
+import time
 
 import configobj
 import josepy as jose
@@ -1243,6 +1245,16 @@ def renew(config, unused_plugins):
     :rtype: None
 
     """
+    if not sys.stdin.isatty():
+        # Noninteractive renewals include a random delay in order to spread
+        # out the load on the certificate authority servers, even if many
+        # users all pick the same time for renewals.  This delay precedes
+        # running any hooks, so that side effects of the hooks (such as
+        # shutting down a web service) aren't prolonged unnecessarily.
+        sleep_time = random.randint(1, 60*8)
+        logger.info("Non-interactive renewal: random delay of %s seconds", sleep_time)
+        time.sleep(sleep_time)
+
     try:
         renewal.handle_renewal_request(config)
     finally:


Test patch:
diff --git a/certbot/tests/main_test.py b/certbot/tests/main_test.py
--- a/certbot/tests/main_test.py
+++ b/certbot/tests/main_test.py
@@ -520,6 +520,8 @@ def setUp(self):
                               '--work-dir', self.config.work_dir,
                               '--logs-dir', self.config.logs_dir, '--text']
 
+        self.mock_sleep = mock.patch('time.sleep').start()
+
     def tearDown(self):
         # Reset globals in cli
         reload_module(cli)
@@ -1092,6 +1094,26 @@ def test_reuse_key_no_dry_run(self, unused_save_successor):
         args = ["renew", "--reuse-key"]
         self._test_renewal_common(True, [], args=args, should_renew=True, reuse_key=True)
 
+    @mock.patch('sys.stdin')
+    def test_noninteractive_renewal_delay(self, stdin):
+        stdin.isatty.return_value = False
+        test_util.make_lineage(self.config.config_dir, 'sample-renewal.conf')
+        args = ["renew", "--dry-run", "-tvv"]
+        self._test_renewal_common(True, [], args=args, should_renew=True)
+        self.assertEqual(self.mock_sleep.call_count, 1)
+        # in main.py:
+        #     sleep_time = random.randint(1, 60*8)
+        sleep_call_arg = self.mock_sleep.call_args[0][0]
+        self.assertTrue(1 <= sleep_call_arg <= 60*8)
+
+    @mock.patch('sys.stdin')
+    def test_interactive_no_renewal_delay(self, stdin):
+        stdin.isatty.return_value = True
+        test_util.make_lineage(self.config.config_dir, 'sample-renewal.conf')
+        args = ["renew", "--dry-run", "-tvv"]
+        self._test_renewal_common(True, [], args=args, should_renew=True)
+        self.assertEqual(self.mock_sleep.call_count, 0)
+
     @mock.patch('certbot.renewal.should_renew')
     def test_renew_skips_recent_certs(self, should_renew):
         should_renew.return_value = False