Repository: mitmproxy/mitmproxy

Issue:
HTTP2 Error when opening news.ycombinator.com
##### Steps to reproduce the problem:
1. Open news.ycombinator.com in Firefox
##### What is the expected behavior?

No errors.
##### What went wrong?

```
Traceback (most recent call last):
  File "c:\python27\Lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "c:\users\user\git\mitmproxy\mitmproxy\protocol\http2.py", line 512, in run
    self()
  File "c:\users\user\git\mitmproxy\mitmproxy\protocol\http2.py", line 518, in __call__
    layer()
  File "c:\users\user\git\mitmproxy\mitmproxy\protocol\http.py", line 178, in __call__
    self.get_response_from_server(flow)
  File "c:\users\user\git\mitmproxy\mitmproxy\protocol\http.py", line 276, in get_response_from_server
    get_response()
  File "c:\users\user\git\mitmproxy\mitmproxy\protocol\http.py", line 272, in get_response
    self.send_request(flow.request)
  File "c:\users\user\git\mitmproxy\mitmproxy\protocol\http2.py", line 429, in send_request
    raise e
ProtocolError: Stream 3 may not depend on itself
```

---

Mitmproxy Version: master
Operating System: Win64



Discussion:
It seams that mitmproxy send request_header with priority_depend_on stream id that was a "client stream id", not  a server stream id  
at http2.py: 418

``` python
 try:
     self.server_conn.h2.safe_send_headers(
         self.is_zombie,
         self.server_stream_id,
         headers,
         end_stream=self.no_body,
         priority_weight=self.priority_weight,
         priority_depends_on=self.priority_depends_on,
         priority_exclusive=self.priority_exclusive,
     )
 except Exception as e:
     raise e
 finally:
     self.server_conn.h2.lock.release()
```

@chhsiao90: Fantastic find, thanks! :smiley: Do you want to submit a PR that fixes this?

@mhils: No problem, I would like to work on this!

`3` is a client stream id. I assume you are using Firefox?
I think Firefox has this odd behaviour, that it sends a few PriorityFrames at the beginning of a connection, to "reserve" streams (or something like that).

The problem is much complicated as I thought if we want to handle PriorityUpdated event.
I had some debug tool to record the h2 event from firefox to server.
And it will create several PriorityUpdated event and each had its stream_id and depends_on stream_id before first Request sent.

```
client: <RemoteSettingsChanged changed_settings:{4: ChangedSetting(setting=4, original_value=65535, new_value=131072), 5: ChangedSetting(setting=5, original_value=16384, new_val
ue=16384)}>
client: <WindowUpdated stream_id:0, delta:12517377>
client: <PriorityUpdated stream_id:3, weight:201, depends_on:0, exclusive:False>
client: <PriorityUpdated stream_id:5, weight:101, depends_on:0, exclusive:False>
client: <PriorityUpdated stream_id:7, weight:1, depends_on:0, exclusive:False>
client: <PriorityUpdated stream_id:9, weight:1, depends_on:7, exclusive:False>
client: <PriorityUpdated stream_id:11, weight:1, depends_on:3, exclusive:False>
client: <RequestReceived stream_id:13, headers:[(u':method', u'GET'), (u':path', u'/from?site=nytimes.com'), (u':authority', u'news.ycombinator.com'), (u':scheme', u'https'), (u
'user-agent', u'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0'), (u'accept', u'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0
.8'), (u'accept-language', u'en-US,en;q=0.5'), (u'accept-encoding', u'gzip, deflate, br'), (u'cookie', u'__cfduid=d44db1f90228df2c224a681f646e888ce1467951413'), (u'pragma', u'no
-cache'), (u'cache-control', u'no-cache')]>
client: <StreamEnded stream_id:13>
client: <PriorityUpdated stream_id:13, weight:32, depends_on:11, exclusive:False>
client: <WindowUpdated stream_id:13, delta:12451840>
server: <RemoteSettingsChanged changed_settings:{3: ChangedSetting(setting=3, original_value=None, new_value=128), 4: ChangedSetting(setting=4, original_value=65535, new_value=2
147483647), 5: ChangedSetting(setting=5, original_value=16384, new_value=16777215)}>
server: <WindowUpdated stream_id:0, delta:2147418112>
client: <SettingsAcknowledged changed_settings:{}>
server: <SettingsAcknowledged changed_settings:{}>
client: <RequestReceived stream_id:15, headers:[(u':method', u'GET'), (u':path', u'/news.css?XcgHNoMJ8uG2XlyGKzF0'), (u':authority', u'news.ycombinator.com'), (u':scheme', u'htt
ps'), (u'user-agent', u'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Firefox/45.0'), (u'accept', u'text/css,*/*;q=0.1'), (u'accept-language', u'en-US,en
;q=0.5'), (u'accept-encoding', u'gzip, deflate, br'), (u'referer', u'https://news.ycombinator.com'), (u'cookie', u'__cfduid=d44db1f90228df2c224a681f646e888ce1467951413'), (u'pra
gma', u'no-cache'), (u'cache-control', u'no-cache')]>
client: <StreamEnded stream_id:15>
client: <PriorityUpdated stream_id:15, weight:32, depends_on:3, exclusive:False>

```

I had tried created a new stream_id at server connection for each PriorityUpdated arrived.
But I had found that hyper-h2 would always give me stream_id: 1 before first header was sent or push
So I couldn't assign a new stream_id on server connection with each PriorityUpdated event.

From hyper-h2 documentation: get_next_available_stream_id()

> The return value from this function does not change until the stream ID has actually been used by sending or pushing headers on that stream. For that reason, it should be called as close as possible to the actual use of the stream ID.

So I had no idea how to handle these weird PriorityUpdated event

@Kriechi : 
Thanks for your advice!
I used Firefox as the test client.
Is it abnormal to send several PriorityUpdated event before request sent?
I'm not familiar with PriorityUpdated control on http2.
So maybe the problem is caused by browser?

Firefox is allowed to do that.
I got a PR almost ready which fixes the problem. However, I think we cannot guarantee 100% accurate priority information between client and server. But it should be good enough to not cause problems.

Got it! thanks


Resolving patch:
diff --git a/mitmproxy/protocol/http2.py b/mitmproxy/protocol/http2.py
--- a/mitmproxy/protocol/http2.py
+++ b/mitmproxy/protocol/http2.py
@@ -206,37 +206,46 @@ def _handle_event(self, event, source_conn, other_conn, is_server):
             self.streams[event.pushed_stream_id].request_data_finished.set()
             self.streams[event.pushed_stream_id].start()
         elif isinstance(event, events.PriorityUpdated):
+            mapped_stream_id = event.stream_id
+            if mapped_stream_id in self.streams and self.streams[mapped_stream_id].server_stream_id:
+                # if the stream is already up and running and was sent to the server
+                # use the mapped server stream id to update priority information
+                mapped_stream_id = self.streams[mapped_stream_id].server_stream_id
+
             if eid in self.streams:
                 if self.streams[eid].handled_priority_event is event:
-                    # This event was already handled during stream creation
+                    # this event was already handled during stream creation
                     # HeadersFrame + Priority information as RequestReceived
                     return True
-                if eid in self.streams:
-                    self.streams[eid].priority_weight = event.weight
-                    self.streams[eid].priority_depends_on = event.depends_on
-                    self.streams[eid].priority_exclusive = event.exclusive
-
-            stream_id = event.stream_id
-            if stream_id in self.streams.keys() and self.streams[stream_id].server_stream_id:
-                stream_id = self.streams[stream_id].server_stream_id
-
-            depends_on = event.depends_on
-            if depends_on in self.streams.keys() and self.streams[depends_on].server_stream_id:
-                depends_on = self.streams[depends_on].server_stream_id
+                self.streams[eid].priority_weight = event.weight
+                self.streams[eid].priority_depends_on = event.depends_on
+                self.streams[eid].priority_exclusive = event.exclusive
 
             with self.server_conn.h2.lock:
                 self.server_conn.h2.prioritize(
-                    stream_id,
+                    mapped_stream_id,
                     weight=event.weight,
-                    depends_on=depends_on,
+                    depends_on=self._map_depends_on_stream_id(mapped_stream_id, event.depends_on),
                     exclusive=event.exclusive
                 )
                 self.server_conn.send(self.server_conn.h2.data_to_send())
         elif isinstance(event, events.TrailersReceived):
-            raise NotImplementedError()
+            raise NotImplementedError("TrailersReceived not implemented")
 
         return True
 
+    def _map_depends_on_stream_id(self, stream_id, depends_on):
+        mapped_depends_on = depends_on
+        if mapped_depends_on in self.streams and self.streams[mapped_depends_on].server_stream_id:
+            # if the depends-on-stream is already up and running and was sent to the server
+            # use the mapped server stream id to update priority information
+            mapped_depends_on = self.streams[mapped_depends_on].server_stream_id
+        if stream_id == mapped_depends_on:
+            # looks like one of the streams wasn't opened yet
+            # prevent self-dependent streams which result in ProtocolError
+            mapped_depends_on += 2
+        return mapped_depends_on
+
     def _cleanup_streams(self):
         death_time = time.time() - 10
         for stream_id in self.streams.keys():
@@ -412,8 +421,6 @@ def send_request(self, message):
         headers.insert(0, ":path", message.path)
         headers.insert(0, ":method", message.method)
         headers.insert(0, ":scheme", message.scheme)
-        self.server_stream_id = self.server_conn.h2.get_next_available_stream_id()
-        self.server_to_client_stream_ids[self.server_stream_id] = self.client_stream_id
 
         try:
             self.server_conn.h2.safe_send_headers(
@@ -422,7 +429,7 @@ def send_request(self, message):
                 headers,
                 end_stream=self.no_body,
                 priority_weight=self.priority_weight,
-                priority_depends_on=self.priority_depends_on,
+                priority_depends_on=self._map_depends_on_stream_id(self.server_stream_id, self.priority_depends_on),
                 priority_exclusive=self.priority_exclusive,
             )
         except Exception as e:


Test patch: