Repository: getsentry/sentry

Issue:
SIGTERM doesn't kill Sentry
I'm using the hosted version of Sentry, and I've set it up with `systemd`.
By default, when stopping, `systemd` uses `SIGTERM` to inform Sentry that it should pack up and "call it a day".

However, `sentry` starts killing itself off, then decides that it has a little more work to do, and starts right back up. Note the section `...brutally killing workers`:

![1xzqkso](https://cloud.githubusercontent.com/assets/7784737/13779222/390b7e1a-eaba-11e5-82b3-8ed9f361d86f.png)

```
$ /opt/sentry/bin/sentry --config=/etc/sentry/sentry.conf.py start'
Running service: 'http'
*** Starting uWSGI 2.0.12 (64bit) on [Tue Mar 15 13:25:08 2016] ***
compiled with version: 5.3.0 on 13 March 2016 22:11:11
os: Linux-4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016
nodename: ***`
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /home/sentry
detected binary path: /opt/sentry/bin/uwsgi
your processes number limit is 31932
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 bound to TCP address 0.0.0.0:9000 fd 3
Python version: 2.7.11 (default, Mar  3 2016, 11:00:04)  [GCC 5.3.0]
Set PythonHome to /opt/sentry
Python main interpreter initialized at 0x1c3fef0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
setting request body buffering size to 65536 bytes
mapped 1922048 bytes (1877 KB) for 12 cores
*** Operational MODE: preforking+threaded ***
spawned uWSGI master process (pid: 1657)
spawned uWSGI worker 1 (pid: 1665, cores: 4)
spawned uWSGI worker 2 (pid: 1666, cores: 4)
spawned uWSGI worker 3 (pid: 1667, cores: 4)
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1c3fef0 pid: 1665 (default app)
WSGI app 0 (mountpoint='') ready in 4 seconds on interpreter 0x1c3fef0 pid: 1667 (default app)
WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x1c3fef0 pid: 1666 (default app)

// Right here, SIGTERM sent to sentry

...brutally killing workers...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
binary reloading uWSGI...
chdir() to /home/sentry
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (0.0.0.0:9000)
running /opt/sentry/bin/uwsgi
*** Starting uWSGI 2.0.12 (64bit) on [Tue Mar 15 13:25:29 2016] ***
compiled with version: 5.3.0 on 13 March 2016 22:11:11
os: Linux-4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016
nodename: ***
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /home/sentry
detected binary path: /opt/sentry/bin/uwsgi
your processes number limit is 31932
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: enabled
uwsgi socket 0 inherited INET address 0.0.0.0:9000 fd 3
Python version: 2.7.11 (default, Mar  3 2016, 11:00:04)  [GCC 5.3.0]
Set PythonHome to /opt/sentry
Python main interpreter initialized at 0x1bbf930
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
setting request body buffering size to 65536 bytes
mapped 1922048 bytes (1877 KB) for 12 cores
*** Operational MODE: preforking+threaded ***
gracefully (RE)spawned uWSGI master process (pid: 1657)
spawned uWSGI worker 1 (pid: 1702, cores: 4)
spawned uWSGI worker 2 (pid: 1703, cores: 4)
spawned uWSGI worker 3 (pid: 1704, cores: 4)

```



Discussion:
Workaround: In the `systemd` service declaration, add the line:

```
KillSignal=SIGINT
```

I wonder if that is actually sentry or uwsgi. In either case I guess we could handle SIGKILL and propagate a SIGINT.

Aha, looks like a feature, totally not a bug: http://uwsgi-docs.readthedocs.org/en/latest/Management.html#signals-for-controlling-uwsgi
Notably:

> (use --die-on-term to respect the convention of shutting down the instance)

Oh, looks like it's [fixed in `uWSGI` 2.1](https://github.com/getsentry/sentry):

> Til uWSGI 2.1, by default, sending the SIGTERM signal to uWSGI means “brutally reload the stack” while the convention is to shut an application down on SIGTERM. To shutdown uWSGI use SIGINT or SIGQUIT instead. If you absolutely can not live with uWSGI being so disrespectful towards SIGTERM, by all means enable the die-on-term option. Fortunately, this bad choice has been fixed in uWSGI 2.1


Resolving patch:
diff --git a/src/sentry/services/http.py b/src/sentry/services/http.py
--- a/src/sentry/services/http.py
+++ b/src/sentry/services/http.py
@@ -65,6 +65,7 @@ def __init__(self, host=None, port=None, debug=False, workers=None,
         options.setdefault('ignore-write-errors', True)
         options.setdefault('disable-write-exception', True)
         options.setdefault('virtualenv', sys.prefix)
+        options.setdefault('die-on-term', True)
         options.setdefault('log-format', '%(addr) - %(user) [%(ltime)] "%(method) %(uri) %(proto)" %(status) %(size) "%(referer)" "%(uagent)"')
 
         options.setdefault('%s-socket' % options['protocol'], '%s:%s' % (host, port))


Test patch: