Package: pyramid-notebook
pyramid_notebookembeds IPython Notebook shell on your Pyramid web site. Start a powerful through-the-browser Python shell with a single click.IPython Notebookis the de facto tool for researches, data analysts and software developers to perform visual and batch oriented tasks.pyramid_notebookputs the power of IPython Notebook inside of aPyramid website.BenefitsHow it worksUse casesPrerequisitesDemoInstallationUsageExample codePyramid settingsNotebook context parametersDead man switchWebsocket proxyinguWSGIWebsocket and reverse proxy servicesArchitectureLocal deploymentProduction deploymentScalabilitySecurityHTTPS onlyVPN restrictionsAccess restricted serversLinux containerizationTwo-factor authenticationTroubleshootingTaking down loose notebooksCrashing NotebooksDebugging Notebook daemonSeeing startup script exceptionsDevelopmentTestsManual testingRelated workChangelog for pyramid_notebook0.3.0 (2018-10-09)0.2.1 (2017-01-29)0.2 (2016-12-06)0.1.11 (2016-04-18)0.1.10 (2016-01-31)0.1.9 (2016-01-31)0.1.8 (2016-01-31)0.1.7 (2016-01-16)0.1.6 (2016-01-16)0.1.5 (2016-01-07)0.1.4 (2015-12-19)0.1.3 (2015-12-19)0.1.2 (2015-12-19)0.1.1 (2015-12-19)0.1 (2015-12-19)BenefitsEasy access: Get powerful shell through a web browser, no additional software installations needed.Automatic default variables: Populate Notebook with variables and data depending on the page where you have a shell button.Authentication integration: use the same credentials as you use for the site administration. Each Pyramid user gets his/her own IPython Notebook process.Rich user interface: Place the cursor with a mouse, use desktop shortcuts for line editing.How it worksUse casesVisualize and analyze your website data.Easier and powerful alternative for normal Python prompt - web browser user interface has more candy over old fashioned terminal.Share code sessions and recipes with your team mates.PrerequisitesPyramid web site(can be easily extended to other web frameworks)Python 3.5+OSX, LinuxuWSGI (on production server only)DemoCheckout source code repository:git clone https://github.com/websauna/pyramid_notebook.gitCreate virtualenv for Python 3.5. Install dependencies:cd pyramid_notebook
python3 -m venv env  # Create virtual environment
source env/bin/activate  # Activate virtual environment
pip install requirements.txt
python setup.py developRun demo:pserve pyramid_notebook/demo/development.iniThen point your browser athttp://localhost:9999.Credentials for this demo are:User 1:username:userpassword:passwordUser 2:username:user2password:passwordInstallationIt is recommend to install usingpipandvirtualenv.Python guide for package installation.pip install pyramid_notebookOn production server where you use uWSGI websocket support:pip install pyramid_notebook[uwsgi]UsageYour application needs to configure three custom views.One or multiplelaunch_ipython()notebook launch points. This does user authentication and authorization and then callspyramid_notebook.views.launch_notebook()to open a new Notebook for a user.launch_ipython()takes in Notebook context parameters (see below), starts a new Notebook kernel if needed and then redirects user to Notebook itself.shutdown_ipython()which does authentication and authorization and callspyramid_notebook.views.shutdown_notebook()to force close a notebook for a user.notebook_proxy()which does authentication and authorization and callspyramid_notebook.views.notebook_proxy()to proxy HTTP request to upstream IPython Notebook server bind to a localhost port.notebook_proxyis mapped to/notebook/path in your site URL. Both your site and Notebook upstream server should agree on this location.Example codeThe following is an example how to constructadmin_shellview which launches a Notebook for the currently logged in Pyramid user when the view is visited for the first time. For extra security the permission for the notebook view cannot be assigned through normal groups, but the username must be on the whitelist in the INI settings file. This guarantees the shell is initially accessible only by persons who have shell access to the server itself.For another approach on these views, please see the demo source code.views.py:frompyramid.httpexceptionsimportHTTPFoundfrompyramid.viewimportview_configfrompyramid_notebookimportstartupfrompyramid_notebook.viewsimportlaunch_notebookfrompyramid_notebook.viewsimportshutdown_notebookas_shutdown_notebookfrompyramid_notebook.viewsimportnotebook_proxyas_notebook_proxyfrompyramid_web20.modelsimportBase#: Include our database session in notebook so it is easy to query stuff right away from the promptSCRIPT="""
from pyramid_web20.models import DBSession as session
"""GREETING="""
* **session** - SQLAlchemy database session
"""@view_config(route_name="notebook_proxy",permission="shell")defnotebook_proxy(request):"""Proxy IPython Notebook requests to the upstream server."""return_notebook_proxy(request,request.user.username)@view_config(route_name="admin_shell",permission="shell")defadmin_shell(request):"""Open admin shell with default parameters for the user."""# Make sure we have a logged in usernb={}# Pass around the Pyramid configuration we used to start this applicationglobal_config=request.registry.settings["pyramid_web20.global_config"]# Get the reference to our Pyramid app config file and generate Notebook# bootstrap startup.py script for this applicationconfig_file=global_config["__file__"]startup.make_startup(nb,config_file)startup.add_script(nb,SCRIPT)startup.add_greeting(nb,GREETING)#: Include all our SQLAlchemy models in the notebook variablesstartup.include_sqlalchemy_models(nb,Base)returnlaunch_notebook(request,request.user.username,notebook_context=nb)@view_config(route_name="shutdown_notebook",permission="shell")defshutdown_notebook(request):"""Shutdown the notebook of the current user."""_shutdown_notebook(request,request.user.username)returnHTTPFound(request.route_url("home"))We also need to capture the INI settings file on the server start up, so that we can pass it forward to IPython Notebook process. In__init__.py:defmain(global_config,**settings):settings["pyramid_web20.global_config"]=global_configThen we have a custom principals handler granting theshellpermission for users read from the user whitelist in the configuration file:deffind_groups(userid,request):"""Get applied groups and other for the user"""fromhorus.interfacesimportIUserModeluser_class=request.registry.queryUtility(IUserModel)# Read superuser names from the configsuperusers=aslist(request.registry.settings.get("pyramid_web20.superusers"))user=models.DBSession.query(user_class).get(userid)ifuser:ifuser.can_login():principals=['group:{}'.format(g.name)forginuser.groups]# Allow superuser permissionifuser.usernameinsuperusersoruser.emailinsuperusers:principals.append("superuser:superuser")returnprincipals# User not found, user disabledreturnNoneWe refer tosuperuser:superin Pyramid site root object:class Root:

    __acl__ = [
        ...
        (Allow, "superuser:superuser", 'shell'),
    ]And here is the configuration file bit:pyramid_web20.superusers =
    mikko@example.comPyramid settingspython_notebookreads following parameters from your Pyramid INI configuration file:# Where we store IPython Notebook runtime and persistent files
# (pid, saved notebooks, etc.).
# Each user will get a personal subfolder in this folder
pyramid_notebook.notebook_folder = /tmp/pyramid_notebook

# Automatically shutdown IPython Notebook kernel
# after his many seconds have elapsed since startup
pyramid_notebook.kill_timeout = 3600

# Websocket proxy launch function.
# This is a view function that upgrades the current HTTP request to Websocket (101 upgrade protocol)
# and starts the web server websocket proxy loop. Currently only uWSGI supported
# (see below).
pyramid_notebook.websocket_proxy =

# For uWSGI in production
# pyramid_notebook.websocket_proxy = pyramid_notebook.uwsgi.serve_websocket

# If you need to server websockets from alternative domain (See below).
# Example value: https://ws.example.com
pyramid_notebook.alternative_domain =Notebook context parametersNotebooks can be opened with context sensitive parameters. Some are filled in by the framework, some of those you can set yourself.You pass in your Notebook context parameters when you calllaunch_notebook().To have custom context variables changestartupscript.To have different info screen changegreetingtextExample of what context information you can pass below:{

    # Extra Python script executed on notebook startup - this is saved as startup.py
    "startup": ""

    # Markdown text displayed at the beginning of the notebook
    "greeting": ""

    # List of paths where to load IPython Notebook Jinja templates
    # http://ipython.org/ipython-doc/3/config/options/notebook.html
    "extra_template_paths": []

    # Notebook daemon process id - filled it in by the daemon itself
    "pid": 1234,

    # Notebook daemon kill timeout in seconds - filled in by the the daemon itself after parsing command line arguments
    "kill_timeout": 5,

    # Bound localhost port for this notebook - filled in by the daemon itself after parsing command line arguments
    "http_port": 9999,

    # Set Notebook HTTP Allow Origin header to tell where websockets are allowed to connect
    "allow_origin": 'localhost:9999',

    # Override websocket URL
    "websocket_url": 'ws://localhost:9998',

    # Path in URL where Notebook is proxyed, must match notebook_proxy() view
    "notebook_path": '/notebook/',

    # Hash of this context. This is generated automatically from supplied context dictionary if not given. If the hash changes the notebook is restarted with new context data.
    "context_hash": 'foo',
}Dead man switchLaunched Notebook processes have maximum life time after which they terminate themselves. Currently the termation is unconditional seconds since the start up, but in the future versions this is expected to change to a dead man switchs where the process only terminates itself if there has not been recent activity.Websocket proxyingIPython Notebook needs two different kind of connections to function properlyHTTP connection for loading the pages, assetsWebsocket for real-time communication with Notebook kernelWhen you run Pyramid’spservedevelopment server on your local machine and enter the Notebook shell, the websocket connection is made directly to IPython Notebook port bound localhost. This is becausepservedoes not have any kind of support for websockets. This behavior is controlled bypyramid_notebook.websocket_proxysetting.On the production server, you usually run a web server which spawns processes to execute WSGI requests, the Python standard for hosting web applications. Unfortunately, like WSGI for HTTP, there doesn’t exist a standard for doing websocket requests in a Python application. Thus, one has to add support for websockets for each web server separately. Currentlypyramid_notebooksupports the following web serversuWSGIIt is ok to have another web server at the front of uWSGI, like Nginx, as these web servers can usually do proxy pass for websocket connections. You might need to add following to your Nginx config:# include a variable for the upgrade header
map $http_upgrade $connection_upgrade {
    default   upgrade;
    ''        close;
}

server {
    location / {
        include uwsgi_params;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }
}uWSGITo turn on websocket support on your uWSGI production server add following to your production INI settings:pyramid_notebook.websocket_proxy = pyramid_notebook.uwsgi.serve_websocketAlso you need to enable websockets in your uWSGI settings:http-websockets = trueWebsocket and reverse proxy servicesReverse proxy services, like CloudFlare <https://support.cloudflare.com/hc/en-us/articles/200169466-Can-I-use-CloudFlare-with-WebSockets->`_, might give only limited or no support for websockets. This may manifest itself in the form of400 Bad Requestresponses from the server because the reverse proxy service strips outConnection: UpgradeHTTP Request header. In this case it is recommended that you serve IPython Notebook from a separate domain where the websocket connection gets unhindered access to your server.You need toConfigure your naked web server to respond to an alternative domain name (ws.example.com).Configurepyramid_notebookto rewrite notebook URLs to come from the alternative domain:pyramid_notebook.alternative_domain = https://ws.example.comPyramidAuthTktAuthenticationPolicy, by default, supports wildcard authentication cookies.You can limit the naked domain to expose/notebook/URLs only.ArchitectureEach Pyramid user has a named Notebook process. Each Notebook process gets their own working folder, dynamically created upon the first lanch. Notebooks are managed byNotebookManagerclass which detects changes in notebook context and restarts the Notebook process for the user with a new context if needed.Notebook bind itself to localhost ports. Pyramid view proxyes/notebook/HTTP requests to Notebook and first checks the HTTP request has necessary permissions by performing authentication and authorization checks. The proxy view is also responsible for starting a web server specific websocket proxy loop.Launched Notebook processes are daemonized and separated from the web server process. The communication between the web server and the daemon process happens through command line, PID file and context file (JSON dump of notebook context parameters, as described above).Local deploymentProduction deploymentScalabilityThe tool is intended for team internal use only. The default settings limit the number of users who can create and access notebooks to 10 people.Currently a new daemon process is launched for each user in non-scalable manner. If 100+ users scalability is required there exist several ways to make the tool more lightweight. For example,you can offload Websockets away from main uWSGI server to a dedicated gevent server.SecurityWith great power comes great responsibility.NoteGiving a userpyramid_notebookaccess is equal to giving him/her SSH access to a website UNIX user.pyramid_notebookrelies on user authorization and authentication by Pyramid web framework. It is your site, so the authentication and authorization system is as good as you made it to be. If you do not feel comfortable exposing this much of power over website authentication, you can still have notebook sessions e.g. over SSH tunneling.Below are some security matters you should consider.HTTPS onlypyramid_notebookaccepts HTTPS connections only. HTTP connections are unencrypted and leaking information over HTTP could lead severe compromises.VPN restrictionsYou can configure your web server to allow access to/notebook/URLs from whitelisted IP networks only.Access restricted serversYou do not need to runpyramid_notebooksessions on the main web servers. You can configure a server with limited data and code separately for runningpyramid_notebook.The access restricted server can haveRead-only account on the databaseSource code and configuration files containing sensitive secrets removed (HTTPS keys, API tokens, etc.)Linux containerizationNotebook process can be made to start inside Linux container. Thus, it would still run on the same server, but you can limit the access to file system and network by the kernel.Read more about Linux cgroups.Two-factor authenticationConsider requiring your website admins to usetwo-factor authenticationto protect against admin credential loss due to malware, keylogging and such nasties. Exampletwo-factor library for Python.TroubleshootingTaking down loose notebooksIn the case the notebook daemon processes get stuck, e.g. by user starting a infinite loop and do not terminate properly, you can take them down.Any time you launch a notebook with different context (different parameters) for the user, the prior notebook process gets terminated forcefullyYou can manually terminate all notebook processes. Ex:pkill -f notebook_daemon.pyCrashing NotebooksThe following are indication of crashed Notebook process.
The following page on Notebook when you try try to start Notebook through web:Apparently IPython Notebook daemon process is not running for user… or the IPython Notebook dialogConnecting failedand connecting to kernel does not work.Notebook has most likely died because of Python exception. There exists a filenotebook.stderr.log, one per each user, where you should be able to read traceback what happened.Debugging Notebook daemonThe notebook daemon can be started from a command line and supports normal UNIX daemonstart,stopandfgcommands. You need to give mandatory pid file, working folder, HTTP port and kill timeout arguments.Example how to start Notebook daemon manually:python $SOMEWHERE/pyramid_notebook/server/notebook_daemon.py fg /tmp/pyramid_notebook/$USER/notebook.pid /tmp/pyramid_notebook/$USER 8899 3600Seeing startup script exceptionsIf the startup script does not populate your Notebook with default variables as you hope, you can always doprint(locals())to see what local variables are setprint(gocals())to see what global variables are setManually execute startup script inside IPython Notebook, e.g.exec(open("/tmp/pyramid_notebook/user-1/.jupyter/profile_default/startup/startup.py").read())(check the actual path by exploring/tmp/pyramid_notebookon your local filesystem).DevelopmentSource codeIssue trackerDocumentationTestsNoteDue to UI complexity of IPython Notebook interaction browser tests must be executed with full Firefox or Chrome driver.Install test dependencies:pip install -e ".[test]"Running manual instance:pserve pyramid_notebook/demo/development.ini --reloadUsername isusernameand passwordpassword.Running tests:py.test tests --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.iniRunning a single test:py.test tests/* --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.ini -s -k test_notebook_templateRun full test coverage:py.test tests/* --cov pyramid_notebook --cov-report xml --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.ini -s -k test_notebook_templateRunning uWSGI server with websockets:uwsgi --virtualenv=venv --wsgi-file=pyramid_notebook/demo/wsgi.py --pythonpath=venv/bin/python uwsgi.iniRunning uWSGI under Nginx for manual websocket proxy testing (OSX):pkill nginx ; nginx -c `pwd`/nginx.conf
uwsgi --virtualenv=venv --wsgi-file=pyramid_notebook/demo/wsgi.py --pythonpath=venv/bin/python uwsgi-under-nginx.iniNoteSelenium Firefox has a bug which prevents typing ( on keyboard, preventing running tests on Firefox.Manual testingYou can manually launch the process to see any errors from IPython Notebook start.Runtest_start_stoptest and capture log output in stdout:py.test tests --splinter-webdriver=chrome --splinter-make-screenshot-on-failure=false --ini=pyramid_notebook/demo/development.ini -s -k test_start_stop
...
INFO:pyramid_notebook.notebookmanager:Running notebook command: python ./pyramid_notebook/server/notebook_daemon.py start /tmp/pyramid_notebook_tests/testuser1/notebook.pid /tmp/pyramid_notebook_tests/testuser1 40007 60You can:python ./pyramid_notebook/server/notebook_daemon.py start /tmp/pyramid_notebook_tests/testuser1/notebook.pid /tmp/pyramid_notebook_tests/testuser1 40005 60Related workhttps://github.com/jupyter/jupyterhubhttps://github.com/Carreau/IPython-notebook-proxyhttps://github.com/UnataInc/ipydra/tree/master/ipydrarChangelog for pyramid_notebook0.3.0 (2018-10-09)Upgraded to IPython 7.0.1 / Jupyter.Add support to Python 3.6, 3.7 and 3.8Remove support to Python 3.4.Enforce Python style checks.Update README.0.2.1 (2017-01-29)Fixed unnecessary “Enter token” prompt with the latest Jupyter Notebook versions0.2 (2016-12-06)Upgraded to IPython 5.1 / JupyterBetter error messages in various situationsAdd custom shutdown command by customizing IPython toolbar menu0.1.11 (2016-04-18)Upgrade to Daemonocle 1.00.1.10 (2016-01-31)Allow easily override IPython Notebook current working directory.0.1.9 (2016-01-31)Make it possible to override the default bootstrap scripts and greeting formake_startup()0.1.8 (2016-01-31)Adding ws4py as a dependency as it is required for uWSGI which is currently the only supported implementation0.1.7 (2016-01-16)Fixed README reST syntax0.1.6 (2016-01-16)Switch to xdaemonocle fork of Daemonocle library0.1.5 (2016-01-07)Keep IPython Notebook in 3.x series now by setup.py pinnings, as IPython 4.0 is not supported yet0.1.4 (2015-12-19)Fixed relative image links to absolute in README0.1.3 (2015-12-19)Fixing MANIFEST.in and release issues due to setuptools_git not present0.1.2 (2015-12-19)Fixed README markupFixed broken drone.io integration which prevented tests to pass on CI0.1.1 (2015-12-19)Fixed broken setup.py classifiers0.1 (2015-12-19)Initial release.