Repository: nltk/nltk

Issue:
BufferedGzipFile in Python3.5
On the [CI tests for Python 3.5](https://nltk.ci.cloudbees.com/job/nltk/TOXENV=py35-jenkins,jdk=jdk8latestOnlineInstall/lastCompletedBuild/testReport/%28root%29/data_doctest/data_doctest/), `BufferedGzipFile` test is breaking at https://github.com/nltk/nltk/blob/develop/nltk/test/data.doctest#L327

This is because of the change in how buffer is read when code was modified from Python3.4 -> 3.5:
- **Python3.4**: https://github.com/python/cpython/blob/3.4/Lib/gzip.py#L189
- **Python3.5**: https://github.com/python/cpython/blob/3.5/Lib/gzip.py#L174

There's a new `GzipFile._buffer` variable in Python3.5 that clashes with `nltk.data.BufferedGzipFile._buffer`, so at https://github.com/nltk/nltk/blob/develop/nltk/data.py#L372 , 
- [When initializing a `nltk.data.BufferedGzipFile`](https://github.com/nltk/nltk/blob/develop/nltk/data.py#L370), it uses the `gzip.GzipFile.__init__` that [sets the `._buffer` variable to the file object](https://github.com/python/cpython/blob/3.5/Lib/gzip.py#L173)
- And [when `nltk.data.BufferedGzipFile` sets `self._buffer = BytesIO()`](https://github.com/nltk/nltk/blob/develop/nltk/data.py#L372), the super-class' buffer was overwritten and unloaded the file object within the `self._buffer` and it reads an empty file causing `test.read()` in the doctest to be None.

Since by default Python3.5 uses a buffer reader, it should have the same functionality as `nltk.data.BufferedGzipFile`. Assuming that the core Python team don't backport this functionality, we should consider deprecating this when we start dropping `python2.7` support.

Meanwhile I think the easiest fix is to refactor `BufferedGzipFile._buffer` to do something like `BufferedGzipFile._nltk_buffer`. And that should resolve Python3.4 -> 3.5 related issues for now =)



Discussion:

Resolving patch:
diff --git a/nltk/data.py b/nltk/data.py
--- a/nltk/data.py
+++ b/nltk/data.py
@@ -369,25 +369,30 @@ def __init__(self, filename=None, mode=None, compresslevel=9,
         """
         GzipFile.__init__(self, filename, mode, compresslevel, fileobj)
         self._size = kwargs.get('size', self.SIZE)
-        self._buffer = BytesIO()
+        # Note: In > Python3.5, GzipFile is already using a 
+        # buffered reader in the backend which has a variable self._buffer
+        # See https://github.com/nltk/nltk/issues/1308
+        if sys.version.startswith('3.5'):
+            sys.stderr.write("Use the native Python gzip.GzipFile instead.")
+        self._nltk_buffer = BytesIO()
         # cStringIO does not support len.
         self._len = 0
 
     def _reset_buffer(self):
         # For some reason calling BytesIO.truncate() here will lead to
         # inconsistent writes so just set _buffer to a new BytesIO object.
-        self._buffer = BytesIO()
+        self._nltk_buffer = BytesIO()
         self._len = 0
 
     def _write_buffer(self, data):
         # Simply write to the buffer and increment the buffer size.
         if data is not None:
-            self._buffer.write(data)
+            self._nltk_buffer.write(data)
             self._len += len(data)
 
     def _write_gzip(self, data):
         # Write the current buffer to the GzipFile.
-        GzipFile.write(self, self._buffer.getvalue())
+        GzipFile.write(self, self._nltk_buffer.getvalue())
         # Then reset the buffer and write the new data to the buffer.
         self._reset_buffer()
         self._write_buffer(data)
@@ -400,7 +405,7 @@ def close(self):
         return GzipFile.close(self)
 
     def flush(self, lib_mode=FLUSH):
-        self._buffer.flush()
+        self._nltk_buffer.flush()
         GzipFile.flush(self, lib_mode)
 
     def read(self, size=None):


Test patch: