Repository: pypi/warehouse

Issue:
Figure out uWSGI Configuration Options
We're currently serving Warehouse on Heroku using gunicorn with the eventlet workers. I started using gunicorn because it's simple, and eventually switched from sync to eventlet workers so that we weren't trivially DoSable by having someone hold open connections (since Heroku's routers don't buffer the request body, only the request headers). In looking at the options, it appears that the following options are sane-ish:
- Continue using gunicorn + eventlet.
- Switch to Waitress which is another Python WSGI server that has an async master process to buffer requests.
- Use nginx or something in front of gunicorn to buffer requests.
- Use uWSGI instead of gunicorn.

Looking at the options, I think uWSGI might be our best option, however we need to figure out a good set of configuration knobs to tune (since uWSGI supports so many of them). We currently have one situation:
- There is no buffering proxy to prevent slowloris or similar attacks.
- We want uploads to go back to the app unbuffered so the app can dynamically choose when to kill an upload (e.g. UserA gets 10MB and UserB gets 50MB).

However, in the future we might be splitting out the upload service to it's own app, in which case it'd be useful to also figure out the configuration options for the additional situation:
- There is no buffering proxy to prevent slowloris or similar attacks.
- We don't care if the app gets requests unbuffered because requests are small.

/cc @mattrobenolt

Add basic uwsgi config
Fixes GH-659



Discussion:
Have you looked into using twisted's wsgi container? I believe it supports
py3k these days.

On Sun, Sep 6, 2015 at 4:55 PM, Donald Stufft notifications@github.com
wrote:

> We're currently serving Warehouse on Heroku using gunicorn with the
> eventlet workers. I started using gunicorn because it's simple, and
> eventually switched from sync to eventlet workers so that we weren't
> trivially DoSable by having someone hold open connections (since Heroku's
> routers don't buffer the request body, only the request headers). In
> looking at the options, it appears that the following options are sane-ish:
> - Continue using gunicorn + eventlet.
> - Switch to Waitress which is another Python WSGI server that has an
>   async master process to buffer requests.
> - Use nginx or something in front of gunicorn to buffer requests.
> - Use uWSGI instead of gunicorn.
> 
> Looking at the options, I think uWSGI might be our best option, however we
> need to figure out a good set of configuration knobs to tune (since uWSGI
> supports so many of them). We currently have one situation:
> - There is no buffering proxy to prevent slowloris or similar attacks.
> - We want uploads to go back to the app unbuffered so the app can
>   dynamically choose when to kill an upload (e.g. UserA gets 10MB and UserB
>   gets 50MB).
> 
> However, in the future we might be splitting out the upload service to
> it's own app, in which case it'd be useful to also figure out the
> configuration options for the additional situation:
> - There is no buffering proxy to prevent slowloris or similar attacks.
> - We don't care if the app gets requests unbuffered because requests
>   are small.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pypa/warehouse/issues/659.

## 

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

I have not, but a quick test it looks like `twistd` doesn't get installed on Python 3:

``` console
$ pip list | grep Twisted
Twisted (15.4.0)
$ twistd -n web --wsgi warehouse.wsgi
zsh: command not found: twistd
```

Would you mind trying with twisted from master, I might have gotten ahead
of myself.

On Sun, Sep 6, 2015 at 5:00 PM, Donald Stufft notifications@github.com
wrote:

> I have not, but a quick test it looks like twistd doesn't get installed
> on Python 3:
> 
> $ pip list | grep TwistedTwisted (15.4.0)
> $ twistd -n web --wsgi warehouse.wsgizsh: command not found: twistd
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pypa/warehouse/issues/659#issuecomment-138125046.

## 

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

That gets it closer to working, but it still doesn't seem to work on Python 3 yet:

``` console
$ twistd -n web --wsgi warehouse.wsgi
Usage: twistd [options]
Options:
      --savestats      save the Stats object rather than the text output of the
                       profiler.
  -o, --no_save        do not save state on shutdown
  -e, --encrypted      The specified tap/aos file is encrypted.
  -n, --nodaemon       don't daemonize, don't use default umask of 0077
      --originalname   Don't try to change the process name
      --syslog         Log to syslog, not to file
      --euid           Set only effective user-id rather than real user-id.
                       (This option has no effect unless the server is running
                       as root, in which case it means not to shed all
                       privileges after binding ports, retaining the option to
                       regain privileges in cases such as spawning processes.
                       Use with caution.)
  -l, --logfile=       log to a specified file, - for stdout
      --logger=        A fully-qualified name to a log observer factory to use
                       for the initial log observer.  Takes precedence over
                       --logfile and --syslog (when available).
  -p, --profile=       Run in profile mode, dumping results to specified file.
      --profiler=      Name of the profiler to use (profile, cprofile).
                       [default: cprofile]
  -f, --file=          read the given .tap file [default: twistd.tap]
  -y, --python=        read an application from within a Python file (implies
                       -o)
  -s, --source=        Read an application from a .tas file (AOT format).
  -d, --rundir=        Change to a supplied directory before running [default:
                       .]
      --prefix=        use the given prefix when syslogging [default: twisted]
      --pidfile=       Name of the pidfile [default: twistd.pid]
      --chroot=        Chroot to a supplied directory before running
  -u, --uid=           The uid to run as.
  -g, --gid=           The gid to run as.
      --umask=         The (octal) file creation mask to apply.
      --version        Print version information and exit.
      --help           Display this help and exit.
      --spew           Print an insanely verbose log of everything that happens.
                       Useful when debugging freezes or locks in complex code.
      --help-reactors  Display a list of possibly available reactor names.
  -r, --reactor=       Which reactor to use (see --help-reactors for a list of
                       possibilities)
  -b, --debug          Run the application in the Python Debugger (implies
                       nodaemon),         sending SIGUSR2 will drop into
                       debugger

twistd reads a twisted.application.service.Application out of a file and runs
it.
Commands:

/Users/dstufft/.virtualenvs/tmp-e3a3921e99052af/bin/twistd: Unknown command: web
```

Ok. THanks for trying!

On Sun, Sep 6, 2015 at 5:11 PM, Donald Stufft notifications@github.com
wrote:

> That gets it closer to working, but it still doesn't seem to work on
> Python 3 yet:
> 
> $ twistd -n web --wsgi warehouse.wsgiUsage: twistd [options]Options:      --savestats      save the Stats object rather than the text output of the                       profiler.  -o, --no_save        do not save state on shutdown  -e, --encrypted      The specified tap/aos file is encrypted.  -n, --nodaemon       don't daemonize, don't use default umask of 0077      --originalname   Don't try to change the process name      --syslog         Log to syslog, not to file      --euid           Set only effective user-id rather than real user-id.                       (This option has no effect unless the server is running                       as root, in which case it means not to shed all                       privileges after binding ports, retaining the option to                       regain privileges in cases such as spawning processes.                       Use with caution.)  -l, --logfile=       log to a specified file, - for stdout      --logger=        A fully-qualified name to a log observer factory to use                       for the initial log observer.  Takes precedence over                       --logfile and --syslog (when available).  -p, --profile=       Run in profile mode, dumping results to specified file.      --profiler=      Name of the profiler to use (profile, cprofile).                       [default: cprofile]  -f, --file=          read the given .tap file [default: twistd.tap]  -y, --python=        read an application from within a Python file (implies                       -o)  -s, --source=        Read an application from a .tas file (AOT format).  -d, --rundir=        Change to a supplied directory before running [default:                       .]      --prefix=        use the given prefix when syslogging [default: twisted]      --pidfile=       Name of the pidfile [default: twistd.pid]      --chroot=        Chroot to a supplied directory before running  -u, --uid=           The uid to run as.  -g, --gid=           The gid to run as.      --umask=         The (octal) file creation mask to apply.      --version        Print version information and exit.      --help           Display this help and exit.      --spew           Print an insanely verbose log of everything that happens.                       Useful when debugging freezes or locks in complex code.      --help-reactors  Display a list of possibly available reactor names.  -r, --reactor=       Which reactor to use (see --help-reactors for a list of                       possibilities)  -b, --debug          Run the application in the Python Debugger (implies                       nodaemon),         sending SIGUSR2 will drop into                       debugger
> twistd reads a twisted.application.service.Application out of a file and runsit.Commands:
> /Users/dstufft/.virtualenvs/tmp-e3a3921e99052af/bin/twistd: Unknown command: web
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/pypa/warehouse/issues/659#issuecomment-138125462.

## 

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084



Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -73,8 +73,6 @@
         "boto3",
         "celery>=3.1",
         "click",
-        "eventlet",
-        "gunicorn",
         "hiredis",
         "html5lib",
         "itsdangerous",
@@ -94,7 +92,6 @@
         "requests",
         "redis>=2.8.0",
         "rfc3986",
-        "setproctitle",
         "setuptools",
         "sqlalchemy>=0.9",
         "sqlalchemy-citext",
diff --git a/tasks/pip.py b/tasks/pip.py
--- a/tasks/pip.py
+++ b/tasks/pip.py
@@ -20,6 +20,7 @@
 -e .
 
 newrelic
+uwsgi
 """
 
 


Test patch: