Package: wsgi-shim
wsgi-shimThis repository contains files to set up and configure a Python-based
website using the WSGI protocol for infrastructure that uses Phusion Passenger.
While the anticipated audience are folks using "project webspace" in the
Department of Computer Science at Princeton University, this package may have
broader utility.This package provides a shim between Phusion Passenger and the WSGI application
that performs some file system checks at start-up.  If any checks fail, a page
request to the website will return an error page with feedback rather than a
generic message from the webserver with error details going to a system log.
The benefit is that in some system configurations, developers do not have
direct real-time access to server logs.Additionally, this package provides awsgi-shimcommand that can set up
a correct directory structure.The following sections describe the package in general.  Additional details
specific to Princeton CS can be found atCS Guide: Python-based Websites
Using WSGI.Infrastructure AssumptionsDeveloper has no direct access to the webserver machine and no direct access to
webserver logs.Files are stored on a shared file systemDeveloper has shell access to a shared machine that can access the shared file systemMultiple virtual hosts are served by the webserver.  Files are segregated using
POSIX permissionsThe webserver runs Passenger Phusion and can run processes as youThe webserver initially runs under a private username and group⚠️ Security ConsiderationsFile PermissionsThe webserver will run your Python code with your UID/NetID (i.e., "as you") and
with a GID/group (to which you belong) of your choosing.  This has the benefit
that you can include secrets/credentials in files that are not world-readable.
A consideration to remember is that because the webserver will run "as you",
your running code will have access to any project space files (in any project)
that you have access.Note that before running "as you," the webserver is running as a user/group that
can only read certain configuration/control files if they are world-readable.
These files are noted in the documentation below.Environment VariablesIf your shell isbash, all the environment variables you set up in your.bashrcand.bash_profilewill be available to your running Python code.
(See: the default "on" configuration ofPassengerLoadShellEnvvars.)
This can obviously be
handy; however, there are some downsides.  First, from a security perspective,
environment variables containing secrets can be exposed to other running
processes on the shared webserver.  Second, from a configuration management
perspective, having some of the configuration for your site be in your personal.bashrcfile in your home directory and the rest of the configuration in the
project space for the site may lead to issues when you are ready to hand the
reins over to another person.When the webserver runs your.bashrcand.bash_profile, it will have set the
environment variableIN_PASSENGER.  If you want the webserver to effectively
skip your normal Bash configuration, you can put this at the top of your.bashrcand.bash_profilefiles:if [ -n "$IN_PASSENGER" ]; then
    umask 027  # no group-write or world-anything; adjust to taste
    return 0
fiAs an alternative, you can pass environment variables and secrets to the webserver
application, you can use the[environment]and[secret_files]sections in theconfig.tomlfile as described below.Requesting Resources / Initial Set-upThe step-by-step instructions in this section assume you are starting from scratch.
If you already have some of the resources that you wish to use, adapt
these instructions as appropriate.📝Note:You won't have direct access to web server logs for debugging.
Therefore, it's important to be methodical and test along the way.  Thewsgi-shimpackage can catch common configuration mistakes and give feedback
without access to the web server logs.1. Request "project disk space"From your sysadmin, request project disk space and a new POSIX group for the files.In these steps, we will assume the disk space is at/n/fs/myproject, the
POSIX group ismygroup, and your username ismynetid.2. Verify/set permissions on the project spaceAfter receiving notification that the disk space has been created, verify that it
exists with the correct group and permissions.Verify project space permissions:ls -ld /n/fs/myprojectThe output should look like something like this:drwxr-sr-x. 4 mynetid mygroup 59 Sep 20 13:24 /n/fs/myprojectIf the owner is notmynetid(i.e.,yourusername), contact support.The group shouldnotbe your default group (i.e., the one displayed when you
run theid -gncommand).  If it is your default group, contact support.If the permissions are notdrwxr-sr-x(note the lowercase "s"), change them withchmod 2755 /n/fs/myproject.  The lowercase "s" in this position is the SETGID bit
and ensures that files and subdirectories created under/n/fs/myprojectwill belong
to themygroupgroup by default.3. Build PythonBuild a local-to-your-website version of Python 3.11 or greater.  You will build
this in the same directory that is shared with the webserver.  Note that the
webserver machine and your access machine must be configured such that Python
built on the access machine will work on the webserver (same architecture, same OS,
same shared libraries, etc.).These are abbreviated instructions from theCS Guide: Building a Local Version of
Python.
To build Python 3.11.5 and install it in your project space at/n/fs/myproject/python-3.11.5use these commands (adjustingPROJECT_DIRandPYVERas appropriate):PROJECT_DIR="/n/fs/myproject"
PYVER="3.11.5"
cd $PROJECT_DIR
mkdir temp
cd temp
wget https://www.python.org/ftp/python/$PYVER/Python-$PYVER.tar.xz
tar xJf Python-$PYVER.tar.xz
cd Python-$PYVER
./configure --prefix=$PROJECT_DIR/python-$PYVER
make
make install
cd /n/fs/myproject
rm -rf tempTest the Python installation:/n/fs/myproject/python-3.11.5/bin/python3 --versionOutput should be:Python 3.11.54. Create Website Directory and Build Python Virtual EnvironmentCreate the directory to hold the files associated with your website.
This directory must be world-readable but not world-writeable:mkdir /n/fs/myproject/mywebsitefiles
chmod o=rx /n/fs/myproject/mywebsitefilesBuild the Python virtual environment for your website in, for example,venv.cd /n/fs/myproject/mywebsitefiles
/n/fs/myproject/python-3.11.5/bin/python3 -m venv venv📝Note:This is the Python virtual environment that will be
used for the running website.  This is, for example, where you
would "pip install" packages such as Flask or Django.5. Install wsgi-shim and Create Initial ConfigurationActivate your virtual environment, install thewsgi-shimpackage,
and create the initial directory structure:cd /n/fs/myproject/mywebsitefiles
source venv/bin/activate
pip install wsgi-shim
wsgi-shim install /n/fs/myproject/mywebsitefiles
deactivateAt this point, your project space directory structure will look like this:/n/fs/myproject/
├── python-3.11.5/
│   └── ...
└── mywebsitefiles/
    ├── venv/
    │   └── ...
    ├── www-approot/
    │   ├── config.toml
    │   ├── passenger_wsgi.py
    │   └── tmp/
    │       ├── maint.txt
    │       └── restart.txt
    └── www-docroot/6. Request "project web space"With the directory structure created, it is time to request that your sysadmins
configure the webserver to recognize your domain (e.g., mywebsite.example.com) as
a valid website and route its requests to your Python code via the providedpassenger_wsgi.pyfile.You will need to provide the key/value pairs in the[passenger]section of the/n/fs/myproject/mywebsitefiles/www-approot/config.tomlfile created in the previous step
to your sysadmins.7. Test Initial ConfigurationWhen you get verification from your sysadmins that the project web space is
set up, verify that the DNS name for the website is in place:host mywebsite.example.comIf you get a response indicating "not found," wait an hour and try again.
If at that point you are still getting a "not found" indication, follow-up
with your sysadmins.Open a browser and visithttps://mywebsite.example.comYou should see a "Maintenance" page.  Now, disablemaintenance modeand force a restart of your site:Run:rm /n/fs/myproject/mywebsitewsgi/www-approot/tmp/maint.txtRun:touch /n/fs/myproject/mywebsitewsgi/www-approot/tmp/restart.txtReload the site a few times (waiting several seconds between reloads)
until it shows updated content.You should see a "Hello, World" page.  If so, congratulations!  You're
ready to start installing and developing code as described in the next
section.If you are not getting the "Hello, World" page, carefully review your
steps and also refer to the Troubleshooting section, below.Install / Develop Site-Specific CodeDirectory StructureOther than being readable by the user/group specified in theconfig.tomlfile, the webserver doesn't put specific restrictions on where in project
space the code implementing your web application must go.  One possibility
is the following layout:/n/fs/myproject/
├── python-3.11.5/
│   └── ...
└── mywebsitefiles/
    ├── venv/
    │   └── ...
    ├── www-approot/
    │   ├── config.toml
    │   ├── passenger_wsgi.py
    │   └── tmp/
    │       └── restart.txt
    ├── www-docroot/
    ├── secrets/
    │   └── credentials.env
    ├── log/
    │   └── logfile
    └── myapp/
        ├── file1.py
        ├── file2.py
        └── ...In this example, thesecretsdirectory contains credentials and, as such,
should not be world-readable; themyappdirectory contains the Python code
making up the web application.  Themyappdirectory might reasonably be
a checked out Git repository of your code.The next section describes how to edit yourconfig.tomlfile to point
to your application inmyapp.Configuration SettingsTheconfig.tomlfile created by thewsgi-shim installcontains
the documentation on available settings.  Here is an example
using the directory structure above with theapplicationcallable
defined infile1.py:[passenger]
passenger_user = "mynetid"
passenger_group = "projectgroup"
passenger_app_root = "/n/fs/myproject/mywebsitefiles/www-approot"
passenger_python = "/n/fs/myproject/mywebsitefiles/venv/bin/python"
[wsgi]
module = "/n/fs/myproject/mywebsitefiles/myapp/file1.py"
app = "app"
[secret_files]
SECRETS = "/n/fs/myproject/mywebsitefiles/secrets/credentials.env"
[environment]
LOG_FILENAME = "/n/fs/myproject/mywebsitefiles/log/logfile"LoggingUnder the assumption that you don't have ready access to the webserver logs,
it is important to set up logging as early as possible in your application.Flask ExampleHere is an example Flask application that would work with the exampleconfig.tomlshown above.# /n/fs/myproject/mywebsitefiles/myapp/file1.py
import logging.config
import os

from flask import Flask

# Configure logging before instantiating Flask app
if LOG_FILENAME := os.environ.get('LOG_FILENAME'):
    logging.config.dictConfig({
        'version': 1,
        'formatters': {'default': {
            'format': '%(asctime)s %(levelname)s %(name)s '
                      '%(threadName)s : %(message)s',
        }},
        'handlers': {'file': {
            'class': 'logging.handlers.RotatingFileHandler',
            'formatter': 'default',
            'filename': LOG_FILENAME,
            'maxBytes': 40000,
            'backupCount': 2,
        }},
        'root': {
            'level': 'INFO',
            'handlers': ['file'],
        }
    })

app = Flask(__name__)


@app.route("/")
def hello_world():
    app.logger.info('Request: /')
    return "<p>Hello, World!</p>"OperationThe webserver will start your application when it receives a request to a URL
inhttps://mywebsite.example.com.  This process will generally handle
multiple requests.  After a period of inactivity, the webserver will kill the
process until there is another request.Restarting Your ApplicationPhusion Passenger allows you to force a restart of your application by
updating the timestamp of therestart.txtfile:touch /n/fs/myproject/mywebsitewsgi/www-approot/tmp/restart.txtNote that the restart is not immediate and the file is only checked when a
URL is visited on your site but no more often than every 10 seconds (the value
of PassengerStatThrottleRate in the webserver global configuration)Using Maintenance ModeThe providedpassenger_wsgi.pyfile (viauwsgi-shim install) implements
a mechanism that, when enabled, will not load your website code but, instead,
show a "maintenance" page where all requests to the site return a503 Service Unavailableresponse.  In this mode,  your site-specific code
(e.g., based on Django, Flask, etc.) will not be accessed.To enable maintenance mode, create the sentinel file:touch /n/fs/myproject/mywebsitewsgi/www-approot/tmp/maint.txtTo disable maintenance mode, delete the sentinel file:rm /n/fs/myproject/mywebsitewsgi/www-approot/tmp/maint.txtNote that the existence of this file is only checked when the WSGI server
(e.g., Phusion Passenger) restarts the application.  Therefore, anytime you
enable/disable maintenance mode, you need to force a restart of the
application.  For example, to update the code for a website, one might
follow these steps:Run:touch /n/fs/myproject/mywebsitewsgi/www-approot/tmp/maint.txtRun:touch /n/fs/myproject/mywebsitewsgi/www-approot/tmp/restart.txtReload the site a few times until it shows the maintenance page.  This can
take up to 10 seconds.Update the siteRun:rm /n/fs/myproject/mywebsitewsgi/www-approot/tmp/maint.txtRun:touch /n/fs/myproject/mywebsitewsgi/www-approot/tmp/restart.txtReload the site a few times until it shows the updated content.Note that therestart.txtfile must be world-readable.  If you
delete this file and later recreate it, be sure it is world-readable.TroubleshootingCheck that your website host name is resolvable in DNS:host mywebsite.example.comUse thewsgi-shim checkcommand to verify directory structure and
permissions:cd /n/fs/myproject/mywebsitefiles
  source venv/bin/activate
  wsgi-shim check /n/fs/myproject/mywebsitefiles
  deactivateVerify that your.bashrcand.bash_profileare not exiting
when invoked by the webserver (i.e., whenIN_PASSENGERis set)?If you have set up logging, check your logs.If your application writes to the file system, verify that you
haven't run out of space.If you believe there is a key bit of information in the webserver logs,
contact your sysadmins and specify the URL that is not working along
with a specific timestamp.DevelopmentFor those developing or maintaining thewsgi-shimpackage itself,
be sure to install it with the[dev]option to pull in packages
used when developing.pip install --editable .[dev]When developing, this package usespre-commit.  After the initial
clone of the repository, you will need to set up pre-commit with:# in the top level of the checked-out repository:
pre-commit installChangelog0.1.0 released 2024-01-19Change[wsgi]semantics: removemoduleaddchdirImproved Django support0.0.7 released 2023-10-25move docroot out of approotcapture broader exceptions (e.g., Django configuration)re-vamp CLI parsing0.0.6 released 2023-10-20fixed a README.md typo0.0.5 released 2023-10-20config.toml[passenger]fields updated topassenger_userandpassenger_groupError and maintenance pages now show a timestampconfig.toml now requires[passenger]sectionRunning process user and group checked against[passenger]sectionfixed loading app from package0.0.4 released 2023-10-17Error/Maintenance HTML responses point to developer0.0.3 released 2023-10-14Fixed disconnect between generatedpassenger_wsgi.pyand this module0.0.2 released 2023-10-13Documentation is more site agnosticAdditional tests0.0.1 released 2023-10-05Initial Beta Version