Repository: ipython/ipython

Issue:
Custom web socket host/port
The issue is that the IPython notebook uses a fixed host name for web socket connections. This is fine in most cases, but some (PaaS) environments support web sockets only on specific ports. OpenShift, for example, [supports web sockets](https://www.openshift.com/blogs/paas-websockets) only on ports 8000 and 8443. I can't run the IPython server on that port because I'm running other web apps that also use web socket connections.

It seems that in previous versions, the IPython notebook had an option `c.NotebookApp.websocket_host` that allowed users to override the web socket host url. This option would satisfy my needs, but appears to have been dropped in the current version (2.x). Was this a conscious decision? If not, can we bring back this option?

I was able to solve this issue by modifying this [line](https://github.com/ipython/ipython/blob/2.x/IPython/html/static/services/kernels/js/kernel.js#L128) from

```
this.ws_host = location.protocol.replace('http', 'ws') + "//" + location.host;
```

to

```
this.ws_host = location.protocol.replace('http', 'ws') + "//" + location.host + ":8000";
```

This solution works well. I have nginx reverse proxy connections to /ipython/ to the ipython server backend running on an arbitrary port. Because I've manually added port `8000` to the `ws_host`, the web sockets function correctly on OpenShift. But it would nice to specify the port and host as a configuration parameter.



Discussion:
This was a conscious decision. The extra config added unnecessary complexity, and we were never able to come up with a use case for the websockets being run on a different host from the rest of the server. The idea of two proxies pointing to one server definitely hadn't occurred to us (and seems super bizarre).  We can consider restoring the ws_host configurable.

I don't understand. The notebook server is one process that listens on a single port for both HTTP and WebSocket traffic. The reason we got rid of the configurability is that if you set the WebSocket port to something different than the HTTP port in the JavaScript frontend, you have to write a entirely custom notebook server that runs these services on different ports - something that our server has never done. I don't see how any proxy magic can change that unless you are also customizing the server itself in a non-trivial way. Can you clarify?

It's definitely possible that my use case is strange or I'm not handling it in the best way. Basically, I have an OpenShift app, which is is kind of like a VM, with an Apache web server listening on 80. Apache routes 80 to 8080, where I have nginx listening. (I would have just put nginx on 80, but the global Apache is an OpenShift requirement).

Next, nginx is set as a reverse proxy to send /ipython/ requests to the ipython backend and /cloud9/ requests to the cloud9 backend. Both IPython and Cloud9 use websockets, so I don't believe I can run both on port 8000. (Is this correct??) Cloud9 has a config parameter that allows one to set the port it uses for web socket connections (but not the full host url). For IPython, I had to modify the source code as mentioned above. Everything works the way I have it set up, but it's not good to hack the source. Here's a chat log from someone in the past who had a similar [problem](http://logs.nodejs.org/openshift/2013-03-26) (search for user `zemanel`).

It seems like the crux of the issue is that **OpenShift only supports websockets on specific ports**.

Here is my interpretation of your setup:

Internet ----> Apache (80) ----> nginx (8080) ----> /ipython/ ----> ipython
(8000)

 ----> /cloud9/ ----> cloud9 (????)

Questions:
- Are the ipython server and cloud9 servers running on the same machine?
- What ports are the ipython server and cloud9 server listening on?
- When you say that "*OpenShift only supports websockets on specific ports"
  *which of these links does that refer to?
- Can you show us your nginx proxy config?
- What port did you change the WebSocket port to in the JavaScript code? I
  would guess this has to be 80 as that is the only public port for the
  apache server?

I think I am starting to understand the issue though. Basically, the
IPython web server listens for web socket on one port p1=8000, but the
apache proxy in front listens on port port2=80. The JavaScript code needs
to make its WebSocket connections on port 80. One way of fixing this is to
use the same port number everywhere (port1=port2), but OpenShift prevents
this by only supporting WebSockets on port 8080. Is this a good summary?

@minrk if this is right, then this will definitely affect our multiuser
server as the single user notebook servers will be using some random high
port, but the main proxy will be using something like 80/443.

Cheers,

Brian

On Tue, Jun 17, 2014 at 11:18 AM, Steven Anton notifications@github.com
wrote:

> It's definitely possible that my use case is strange or I'm not handling
> it in the best way. Basically, I have an OpenShift app, which is is kind of
> like a VM, with an Apache web server listening on 80. Apache routes 80 to
> 8080, where I have nginx listening. (I would have just put nginx on 80, but
> the global Apache is an OpenShift requirement).
> 
> Next, nginx is set as a reverse proxy to send /ipython/ requests to the
> ipython backend and /cloud9/ requests to the cloud9 backend. Both IPython
> and Cloud9 use websockets, so I don't believe I can run both on port 8000.
> (Is this correct??) Cloud9 has a config parameter that allows one to set
> the port it uses for web socket connections (but not the full host url).
> For IPython, I had to modify the source code as mentioned above. Everything
> works the way I have it set up, but it's not good to hack the source.
> Here's a chat log from someone in the past who had a similar problem
> http://logs.nodejs.org/openshift/2013-03-26 (search for user zemanel).
> 
> It seems like the crux of the issue is that _OpenShift only supports
> websockets on specific ports_.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ipython/ipython/issues/6001#issuecomment-46345077.

## 

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

@ellisonbg: Thanks for looking into this! Let me clarify a bit better:

The diagram is close to what you drew, but the port for the ipython server is not 8000. In my case it's 5001, but that's just a random choice. It's the same with cloud9:
Internet ----> Apache (80) ----> nginx (8080) ----> /ipython/ ----> ipython (5001)
Internet ----> Apache (80) ----> nginx (8080) ----> /cloud9/ ----> cloud9 (3131)

Also, here are some answers to your questions:

> - Are the ipython server and cloud9 servers running on the same machine?

Yes

> - What ports are the ipython server and cloud9 server listening on?

5001 and 3131, respectively, although this choice is arbitrary

> - When you say that "*OpenShift only supports websockets on specific ports" which of these links does that refer to?

[Here's](https://www.openshift.com/blogs/paas-websockets) a good explanation

> - Can you show us your nginx proxy config?

Here's the relevant IPython part:

```
upstream ipython_server {
    server <%= ENV['OPENSHIFT_DIY_IP'] %>:5001;
}
...
server{
    ...
    location /ipython {
        proxy_pass http://ipython_server;

        # Needed for the websockets connections:
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Host and Origin in the request header don't seem to get passed
        # correctly by nginx. Host gets passed as http://ipython_server
        # and Origin is blank. This is just a hack, which really shouldn't
        # be this way:
        proxy_set_header Host <%= ENV['OPENSHIFT_DIY_IP'] %>:8000;
        proxy_set_header Origin http://<%= ENV['OPENSHIFT_DIY_IP'] %>:8000;
    }
    ...
}
```

> - What port did you change the WebSocket port to in the JavaScript code? I
>   would guess this has to be 80 as that is the only public port for the
>   apache server?

I changed the web sockets to use port 8000. This port is open on the machine and I don't think it's handled by the Apache server. (I could be wrong about the latter, though.)

> I think I am starting to understand the issue though. Basically, the
> IPython web server listens for web socket on one port p1=8000, but the
> apache proxy in front listens on port port2=80. The JavaScript code needs
> to make its WebSocket connections on port 80. One way of fixing this is to
> use the same port number everywhere (port1=port2), but OpenShift prevents
> this by only supporting WebSockets on port 8080. Is this a good summary?

I think that this is basically correct if you change the port numbers around. The IPython server runs on an arbitrary port (5001 in my case). Nginx forwards incoming /ipython requests to the IPython server. When a new notebook is created, my hack to kernel.js (shown above) makes IPython upgrade to a web socket connection on port 8000. How exactly the web socket connection gets routed, I'm not entirely sure, but it seems to work just fine and plays well with Cloud9, which is also running web sockets on port 8000. (Note that Cloud9 has a configuration file where you can specify a web socket connection port.) I'm also able to run multiple notebooks simultaneously, so the web socket connections don't appear to "collide" so to speak. 

Thanks again for looking into this!
-Steve

Thanks this is helpful. A few more...
- Which server is listening on port 8000? Same server that runs apache?
- Is that server that is facing the public internet?
- Where does incoming traffic on that public server:8000 get redirected -
  to the nginx proxy?

On Mon, Jun 23, 2014 at 1:57 PM, Steven Anton notifications@github.com
wrote:

> @ellisonbg https://github.com/ellisonbg: Thanks for looking into this!
> Let me clarify a bit better:
> 
> The diagram is close to what you drew, but the port for the ipython server
> is not 8000. In my case it's 5001, but that's just a random choice. It's
> the same with cloud9:
> Internet ----> Apache (80) ----> nginx (8080) ----> /ipython/ ---->
> ipython (5001)
> Internet ----> Apache (80) ----> nginx (8080) ----> /cloud9/ ----> cloud9
> (3131)
> 
> Also, here are some answers to your questions:
> - Are the ipython server and cloud9 servers running on the same
>   machine?
>   
>   Yes
>   - What ports are the ipython server and cloud9 server listening on?
>   
>   5001 and 3131, respectively, although this choice is arbitrary
>   - When you say that "*OpenShift only supports websockets on specific
>     ports" which of these links does that refer to?
>   
>   Here's https://www.openshift.com/blogs/paas-websockets a good
>   explanation
> - Can you show us your nginx proxy config?
>   
>   Here's the relevant IPython part:
> 
> upstream ipython_server {
>     server <%= ENV['OPENSHIFT_DIY_IP'] %>:5001;
> }
> ...
> server{
>     ...
>     location /ipython {
>         proxy_pass http://ipython_server;
> 
> ```
>     # Needed for the websockets connections:
>     proxy_http_version 1.1;
>     proxy_set_header Upgrade $http_upgrade;
>     proxy_set_header Connection "upgrade";
> 
>     # Host and Origin in the request header don't seem to get passed
>     # correctly by nginx. Host gets passed as http://ipython_server
>     # and Origin is blank. This is just a hack, which really shouldn't
>     # be this way:
>     proxy_set_header Host <%= ENV['OPENSHIFT_DIY_IP'] %>:8000;
>     proxy_set_header Origin http://<%= ENV['OPENSHIFT_DIY_IP'] %>:8000;
> }
> ...
> ```
> 
> }
> - What port did you change the WebSocket port to in the JavaScript
>   code? I would guess this has to be 80 as that is the only public port for
>   the apache server?
>   
>   I changed the web sockets to use port 8000. This port is open on the
>   machine and I don't think it's handled by the Apache server. (I could be
>   wrong about the latter, though.)
> 
> I think I am starting to understand the issue though. Basically, the
> IPython web server listens for web socket on one port p1=8000, but the
> apache proxy in front listens on port port2=80. The JavaScript code needs
> to make its WebSocket connections on port 80. One way of fixing this is to
> use the same port number everywhere (port1=port2), but OpenShift prevents
> this by only supporting WebSockets on port 8080. Is this a good summary?
> 
> I think that this is basically correct if you change the port numbers
> around. The IPython server runs on an arbitrary port (5001 in my case).
> Nginx forwards incoming /ipython requests to the IPython server. When a new
> notebook is created, my hack to kernel.js (shown above) makes IPython
> upgrade to a web socket connection on port 8000. How exactly the web socket
> connection gets routed, I'm not entirely sure, but it seems to work just
> fine and plays well with Cloud9, which is also running web sockets on port
> 8000. (Note that Cloud9 has a configuration file where you can specify a
> web socket connection port.) I'm also able to run multiple notebooks
> simultaneously, so the web socket connections don't appear to "collide" so
> to speak.
> 
> Thanks again for looking into this!
> -Steve
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ipython/ipython/issues/6001#issuecomment-46901001.

## 

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

- I actually don't have anything running on port 8000, which puzzles me. I don't really understand what happens with the web socket connection.
- I'm also not sure if the server is facing the public internet. It's behind a corporate firewall and I have to open a VPN connection to access the website.
- Traffic to port 8000 goes to the nginx proxy. When testing access to my homepage (located at root, /), adding port 8000 does nothing, but 8080 gets blocked and I don't know why.

OK, my interpretation is that OpenShift runs a WebSocket routing system on
port 8000 for all of its servers. Basically, it runs next to Apache on port
80 and handles all of the WS traffic for that server. The WebSocket routing
system forwards to the WS traffic to the backend, which for your ase in the
nginx server. Based on your descriptions, I think we do need to make this
configurable again.

@minrk do you agree with my assessment? How do you think we should design
this to work with the multiuser notebook server?

On Mon, Jun 23, 2014 at 2:35 PM, Steven Anton notifications@github.com
wrote:

> - I actually don't have anything running on port 8000, which puzzles
>   me. I don't really understand what happens with the web socket connection.
> - I'm also not sure if the server is facing the public internet. It's
>   behind a corporate firewall and I have to open a VPN connection to access
>   the website.
> - Traffic to port 8000 goes to the nginx proxy. When testing access to
>   my homepage (located at root, /), adding port 8000 does nothing, but 8080
>   gets blocked and I don't know why.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ipython/ipython/issues/6001#issuecomment-46905407.

## 

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

This seems like a really awful system. I don't think there's anything that would be different about the multiuser server from the single-user server, since as far as all of our server-side code is concerned, everything is running on one host (behind node-http-proxy). It's purely client-side config for setting this URL, and none of our code will ever make any use of it. So this is strictly adding config to support completely bizarre multi-hop proxy setups.  But it's still pretty easy, just adding the ws host back to the kernel model, and leaving it empty by default, so the current behavior would be completely unchanged.

Let's say that our multiuser server looks like this:

multiuser server proxy (443) ----> single user A (10001)
                                              ----> single user B (10002)
                                              ----> single user C (10003)

The JavaScript/WebSocket code in kernel.js for all three users needs to
point to port 443. Right now I don't think that is possible as the
WebSocket port will always match the main server port (10001, etc. for each
user). So don't we have to allow for this customization to get the
multiuser server to work? I want to make sure I am not missing something.

On Mon, Jun 23, 2014 at 2:45 PM, Min RK notifications@github.com wrote:

> This seems like a really awful system. I don't think there's anything that
> would be different about the multiuser server from the single-user server,
> since as far as all of our server-side code is concerned, everything is
> running on one host (behind node-http-proxy). It's purely client-side
> config for setting this URL, and none of our code will ever make any use of
> it. So this is strictly adding config to support completely bizarre
> multi-hop proxy setups. But it's still pretty easy, just adding the ws host
> back to the kernel model, and leaving it empty by default, so the current
> behavior would be completely unchanged.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ipython/ipython/issues/6001#issuecomment-46906498.

## 

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgranger@calpoly.edu and ellisonbg@gmail.com

Based on this [link](http://support.run.pivotal.io/entries/49512163-Does-cloudfoundry-support-websocket-) it looks like CloudFoundary might also only support websockets on one port...


Resolving patch:
diff --git a/IPython/html/base/handlers.py b/IPython/html/base/handlers.py
--- a/IPython/html/base/handlers.py
+++ b/IPython/html/base/handlers.py
@@ -127,6 +127,10 @@ def mathjax_url(self):
     @property
     def base_url(self):
         return self.settings.get('base_url', '/')
+
+    @property
+    def ws_url(self):
+        return self.settings.get('websocket_url', '')
     
     #---------------------------------------------------------------
     # Manager objects
@@ -215,6 +219,7 @@ def render_template(self, name, **ns):
     def template_namespace(self):
         return dict(
             base_url=self.base_url,
+            ws_url=self.ws_url,
             logged_in=self.logged_in,
             login_available=self.login_available,
             static_url=self.static_url,
diff --git a/IPython/html/notebookapp.py b/IPython/html/notebookapp.py
--- a/IPython/html/notebookapp.py
+++ b/IPython/html/notebookapp.py
@@ -172,6 +172,7 @@ def init_settings(self, ipython_app, kernel_manager, notebook_manager,
 
             # IPython stuff
             nbextensions_path = ipython_app.nbextensions_path,
+            websocket_url=ipython_app.websocket_url,
             mathjax_url=ipython_app.mathjax_url,
             config=ipython_app.config,
             jinja2_env=env,
@@ -512,6 +513,13 @@ def static_file_path(self):
     def _nbextensions_path_default(self):
         return [os.path.join(get_ipython_dir(), 'nbextensions')]
 
+    websocket_url = Unicode("", config=True,
+        help="""The base URL for websockets,
+        if it differs from the HTTP server (hint: it almost certainly doesn't).
+        
+        Should be in the form of an HTTP origin: ws[s]://hostname[:port]
+        """
+    )
     mathjax_url = Unicode("", config=True,
         help="""The url for MathJax.js."""
     )


Test patch: