Repository: urllib3/urllib3

Issue:
Could calling  self._decoder.decompress after flush() has been called on the decompressobj instance cause some problems ?
Disclaimer : I don’t know much about python. I am calling, from Java (through jython), a python script that uses a 3rd party python API that makes use of urllib3 to call a REST api. This whole setup obviously makes it challenging to troubleshoot issues.
The 3rd party python API we’re using uses urllib3 version 1.12
While I know it would be very useful, I do not have a small sample to replicate the issue (I’d have to peel a lot of layers that I do not have a good understanding of at this point). I might need to work on it if nobody can help me without it.

The issue I am having is thrown by the zlib java implementation that is called by jython:

```
java.lang.NullPointerException: java.lang.NullPointerException: Inflater has been closed

File "C:\Users\laurentd\.gradle\caches\modules-2\files-2.1\org.python\jython-standalone\2.7.0\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\jython-standalone-2.7.0.jar\Lib\zlib.py", line 202, in _get_inflate_data
    at java.util.zip.Inflater.ensureOpen(Inflater.java:389)
    at java.util.zip.Inflater.inflate(Inflater.java:257)
    at java.util.zip.Inflater.inflate(Inflater.java:280)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
```

The issue seems to be linked to the fact that the flush() method is called on a zlip.decompressobj object but that somehow, somewhere, we’re still trying to call the decompress method on that zlibdecompressobj object.
This is something that the zlib documentation mentions should not be done:
“After calling flush(), the decompress() method cannot be called again; the only realistic action is to delete the object.” (from https://docs.python.org/2/library/zlib.html)

Further debugging took me to the response.py file, in the read_chunked method, especially in the last line of this part:

```
            while True:
                self._update_chunk_length()
                if self.chunk_left == 0:
                    break
                chunk = self._handle_chunk(amt)
                yield self._decode(chunk, decode_content=decode_content,
                                   flush_decoder=True)
```

It looks like when self._decode is called, we always set flush_decoder to True.

Because of this, if I understand correctly, the self._decoder object (which is, in my case, a GzipDecoder, and a decompressobj?) is always flushed at the end of the call to self._decode (line 200).
However, the self._decoder seems to be initialized only once and seems to be reused (at least in my case) throughout the multiple calls to self._decode.

Though I am probably missing a few things, it could explain why decompressobj.decompress is called on a decompressobj object that has been flushed, and this could (?) have unpredictable results, including that NPE in the jython implementation of zlib (and ultimately in java).

Here is a deeper (and sanitized) portion of the call stack when the exception happens.

```
....\scripts\requests\sessions.py", line 608, in send
    r.content
....\scripts\requests\models.py", line 734, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
....\scripts\requests\models.py", line 734, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
....\scripts\requests\models.py", line 657, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
....\scripts\requests\models.py", line 657, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
....\scripts\requests\packages\urllib3\response.py", line 322, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
....\scripts\requests\packages\urllib3\response.py", line 471, in read_chunked
    yield self._decode(chunk, decode_content=decode_content,
....\scripts\requests\packages\urllib3\response.py", line 190, in _decode
    data = self._decoder.decompress(data)
....\scripts\requests\packages\urllib3\response.py", line 190, in _decode
    data = self._decoder.decompress(data)
....\scripts\requests\packages\urllib3\response.py", line 56, in decompress
    return self._obj.decompress(data)
  File "C:\Users\laurentd\.gradle\caches\modules-2\files-2.1\org.python\jython-standalone\2.7.0\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\jython-standalone-2.7.0.jar\Lib\zlib.py", line 155, in decompress
  File "C:\Users\laurentd\.gradle\caches\modules-2\files-2.1\org.python\jython-standalone\2.7.0\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\jython-standalone-2.7.0.jar\Lib\zlib.py", line 202, in _get_inflate_data
  File "C:\Users\laurentd\.gradle\caches\modules-2\files-2.1\org.python\jython-standalone\2.7.0\cdfb38bc6f8343bcf1d6accc2e1147e8e7b63b75\jython-standalone-2.7.0.jar\Lib\zlib.py", line 202, in _get_inflate_data
```

Is it possible that the _decode method is causing the problem by calling decompress on a decompressobj that has been flushed or am I going down the wrong track?

Note that the problem is gone if I reinitialize the decompressobj at the beginning of _decode by mimicking what _init_decoder does, which I can’t believe is a good fix (although self._decoder seems to be really used only in _decode):

```
        content_encoding = self.headers.get('content-encoding', '').lower()
        self._decoder = _get_decoder(content_encoding)
```

Thanks in advance.

Laurent.



Discussion:
Hmm, this is interesting. You're right that this is what happened. We're doing this to ensure that each gzipped chunk yields some data, and that appears to work in CPython and PyPy.

In this case, we should consider whether there is a solution built around one of the other mode constants in zlib.

So, in CPython zlib actually appears to _totally allow_ flushing the decompressor between each call to decompress. No problem occurs there, everything works fine. Which explains why we didn't see this before: it really only affects Jython and chunked, gzipped content. Quite a small proportion of use cases!

I'll see if I can get hold of a Jython install and play around with it.

Ah, well, we can't use the other mode constants because Jython doesn't accept them as arguments to flush. That's somewhat limiting.

@shazow I think the only way we can get this to work is to not flush the decompressor as the chunks come in. This means that read_chunks will potentially return zero-length chunks, which means we'll need to update `stream()`, but that's probably fine. How does that solution sound to you?

@lukasa I think it's fine since `read_chunked` is a generator anyways (unlike `read()`).

Should there be a bug filed against Jython too? It makes sense to me that `zlib`'s behaviour should be the same between CPython, PyPy, and Jython (Since they're kind of the big three, right?)

@sigmavirus24 I mean, I _suppose_? It's hard to see how useful that bug report would be for the Jython developers, really. The most that the bug could say is:
1. You haven't exported some constants that exist in the CPython zlib library.
2. You use zlib differently to CPython/PyPy, but totally within the bounds of what the docs say is allowed.

Point (1) is vaguely reasonable, point (2) is less clear. The docs certainly allow their current behaviour. That said, it can't hurt to at least tell them. =)

I'd be surprised if we're the only people using zlib like we are and if more people aren't going to stumble on this. That's my only line of thinking here. I'm all for fixing it here too, I just find it hard to believe we're a snowflake in this regard. =P

So, with regard to fixing this problem, we have two options.

First, we can switch to yielding everything from the decoder, and then once more from the flush. This means we'll always yield once for each chunk, and then once at the end. This is potentially a bit unexpected, and we'll also occasionally yield the empty string.

Second, we can do that but only yield if the string isn't empty: that is, if there's some decompressed data. This means we never yield the empty string, but also potentially means the number of yields is unrelated to the actual number of chunks.

Does anyone have a preference here?

>  we'll also occasionally yield the empty string

This makes me lean towards the second option. We've already had some friction over yielding empty strings from methods that intuitively shouldn't.

+1 to @sigmavirus24.

If someone wanted to do some meaty refactoring then I'd say that we should decouple the raw response reading (lower level, maybe some generic io interface implementation that can be swapped out) from the decoding-aware response reading (higher level, default, takes an io thing and does magic). If you really really want 1:1 read amount control, you'd fall back to the raw response reading and handle decoding with our helpers.

Agreed that there's some discussion to be had about the way our I/O looks long term. Right now I'm inclined to want to patch this, rather than bite the bullet on that big work.


Resolving patch:
diff --git a/urllib3/response.py b/urllib3/response.py
--- a/urllib3/response.py
+++ b/urllib3/response.py
@@ -195,12 +195,22 @@ def _decode(self, data, decode_content, flush_decoder):
                 "Received response with content-encoding: %s, but "
                 "failed to decode it." % content_encoding, e)
 
-        if flush_decoder and decode_content and self._decoder:
-            buf = self._decoder.decompress(binary_type())
-            data += buf + self._decoder.flush()
+        if flush_decoder and decode_content:
+            data += self._flush_decoder()
 
         return data
 
+    def _flush_decoder(self):
+        """
+        Flushes the decoder. Should only be called if the decoder is actually
+        being used.
+        """
+        if self._decoder:
+            buf = self._decoder.decompress(b'')
+            return buf + self._decoder.flush()
+
+        return b''
+
     @contextmanager
     def _error_catcher(self):
         """
@@ -475,8 +485,18 @@ def read_chunked(self, amt=None, decode_content=None):
                 if self.chunk_left == 0:
                     break
                 chunk = self._handle_chunk(amt)
-                yield self._decode(chunk, decode_content=decode_content,
-                                   flush_decoder=True)
+                decoded = self._decode(chunk, decode_content=decode_content,
+                                       flush_decoder=False)
+                if decoded:
+                    yield decoded
+
+            if decode_content:
+                # On CPython and PyPy, we should never need to flush the
+                # decoder. However, on Jython we *might* need to, so
+                # lets defensively do it anyway.
+                decoded = self._flush_decoder()
+                if decoded:  # Platform-specific: Jython.
+                    yield decoded
 
             # Chunk content ends with \r\n: discard it.
             while True:


Test patch: