Repository: scrapy/scrapy

Issue:
response.body is duplicate
Access the [text page(not mine)](http://files.qidian.com/Author4/3615059/88542882.txt) by browsers or wget and you will find  the response content is not duplicate, but scrapy's `response.body` is duplicate. I had tried set the scrapy's headers same as a real brower's, but it is still duplicate.

Just use the follow sample code, and you will find the issue.

```
scrapy shell "http://files.qidian.com/Author4/3615059/88542882.txt"
```

Sorry for my bad english. 



Discussion:
Hm I can replicate this on my machine. The body is not fully duplicated, but almost:

``` python
jakob@MosEisley ~ % scrapy shell http://files.qidian.com/Author4/3615059/88542882.txt               
2015-11-17 01:40:37 [scrapy] INFO: Scrapy 1.0.3 started (bot: scrapybot)
2015-11-17 01:40:37 [scrapy] INFO: Optional features available: ssl, http11
2015-11-17 01:40:37 [scrapy] INFO: Overridden settings: {'LOGSTATS_INTERVAL': 0}
2015-11-17 01:40:38 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, SpiderState
2015-11-17 01:40:38 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-11-17 01:40:38 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-11-17 01:40:38 [scrapy] INFO: Enabled item pipelines: 
2015-11-17 01:40:38 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6025
2015-11-17 01:40:38 [scrapy] INFO: Spider opened
2015-11-17 01:40:38 [scrapy] DEBUG: Crawled (200) <GET http://files.qidian.com/Author4/3615059/88542882.txt> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f78dff83ad0>
[s]   item       {}
[s]   request    <GET http://files.qidian.com/Author4/3615059/88542882.txt>
[s]   response   <200 http://files.qidian.com/Author4/3615059/88542882.txt>
[s]   settings   <scrapy.settings.Settings object at 0x7f78d749bbd0>
[s]   spider     <DefaultSpider 'default' at 0x7f78dc893190>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser
2015-11-17 01:40:38 [root] DEBUG: Using default logger
2015-11-17 01:40:38 [root] DEBUG: Using default logger

In [1]: len(response.body)
Out[1]: 16661

In [2]: import requests

In [3]: resp = requests.get("http://files.qidian.com/Author4/3615059/88542882.txt")
2015-11-17 01:41:01 [requests.packages.urllib3.connectionpool] INFO: Starting new HTTP connection (1): files.qidian.com
2015-11-17 01:41:01 [requests.packages.urllib3.connectionpool] DEBUG: "GET /Author4/3615059/88542882.txt HTTP/1.1" 200 None

In [4]: len(resp.text)
Out[4]: 8465

In [5]: response.body.count('document')
Out[5]: 2

In [6]: resp.text.count('document')
Out[6]: 1

In [7]: response.body.find('document', 1)
Out[7]: 8196

In [8]: len(response.body[8196:])
Out[8]: 8465

In [9]: response.body_as_unicode()[8196:] == resp.text
Out[9]: True
```

Downloading the file from a different server (http://pastebin.com/raw.php?i=37ewyTRK) doesn't change any of the `requests` stuff, but interestingly changes Scrapy's response:

``` python
In [12]: resp2 = requests.get("http://pastebin.com/raw.php?i=37ewyTRK")
2015-11-17 01:49:17 [requests.packages.urllib3.connectionpool] INFO: Starting new HTTP connection (1): pastebin.com
2015-11-17 01:49:18 [requests.packages.urllib3.connectionpool] DEBUG: "GET /raw.php?i=37ewyTRK HTTP/1.1" 200 None

In [14]: resp2.text == resp.text
Out[14]: True

In [15]: fetch("http://pastebin.com/raw.php?i=37ewyTRK")
2015-11-17 01:49:40 [scrapy] DEBUG: Crawled (200) <GET http://pastebin.com/raw.php?i=37ewyTRK> (referer: None)
[s] Available Scrapy objects:
[s]   crawler    <scrapy.crawler.Crawler object at 0x7f78dff83ad0>
[s]   item       {}
[s]   request    <GET http://pastebin.com/raw.php?i=37ewyTRK>
[s]   response   <200 http://pastebin.com/raw.php?i=37ewyTRK>
[s]   settings   <scrapy.settings.Settings object at 0x7f78d749bbd0>
[s]   spider     <DefaultSpider 'default' at 0x7f78dc893190>
[s] Useful shortcuts:
[s]   shelp()           Shell help (print this help)
[s]   fetch(req_or_url) Fetch request (or URL) and update local objects
[s]   view(response)    View response in a browser

In [16]: len(response.body)
Out[16]: 16595

In [17]: response.body.count('document')
Out[17]: 1

In [18]: response.body.find('document')
Out[18]: 0
```

```
```

Thank you very mutch! @jdemaeyer ! Your detail is very useful!.


Resolving patch:
diff --git a/scrapy/utils/gz.py b/scrapy/utils/gz.py
--- a/scrapy/utils/gz.py
+++ b/scrapy/utils/gz.py
@@ -43,7 +43,7 @@ def gunzip(data):
             # contains the whole page content
             if output or getattr(f, 'extrabuf', None):
                 try:
-                    output += f.extrabuf
+                    output += f.extrabuf[-f.extrasize:]
                 finally:
                     break
             else:


Test patch:
diff --git a/tests/sample_data/compressed/unexpected-eof-output.txt b/tests/sample_data/compressed/unexpected-eof-output.txt
new file mode 100644
--- /dev/null
+++ b/tests/sample_data/compressed/unexpected-eof-output.txt
@@ -0,0 +1 @@
+document.write('¡¡¡¡¡°ÎÒ½ÐºéÐ¡±¦£¬ÊÇÒ»¸ö·Ç³£¸ß¶Ë·Ç³£Å£±ÆµÄÖ°Òµ¶þÊÀ×æ£¬È´ÔÚÒ»¸öÔÂºÚ·ç¸ßÉìÊÖ²»¼ûÎåÖ¸µÄÒ¹ÍíÎÒÄªÃûÆäÃîµÄ¾Í±»ÀÏ°Ö¸øÒ»½Åõßµ½ÁËÕâÃ´¸öµØ·½¡­¡­¡±<p>¡¡¡¡¡°ÔÚ´©Ô½µÄÄÇÒ»¿Ì£¬ÎÒÏëÆðÁËÔÚÑ§Ð£Ñ§µÄÓïÊýÍâÎï»¯Éú¡ª¡ªÎÒ¶¼²»»á¡­¡­¡±<p>¡¡¡¡¡­¡­<p>¡¡¡¡ÌìÔË´óÂ½£¬³àÔÆ¹ú»Ê¹¬Ö®ÖÐ£¬»ÊµÛºéÎÄÇåµÄÊé·¿ÐþÇå¸óÄÚ¡£<p>¡¡¡¡¡°±ÝÏÂ£¬¡±Ò»Ãû»¤ÎÀ´Ò´ÒÀ´±¨£º¡°Íâ³öÔÆÓÎÊ®¶þÄêµÄ¹úÊ¦»ØÀ´ÁË£¬ÏÖÔÚÕýÔÚÐþÑôµîÄÚµÈºò±ÝÏÂ¡£¡±<p>¡¡¡¡¡°Å¶£¿¹úÊ¦»ØÀ´ÁË£¿¡±Ìýµ½»¤ÎÀµÄ»°£¬»ÊµÛºéÎÄÇåÏÈÊÇÒ»ã¶£¬Ëæºó·ÅÏÂÊÖÖÐµÄ×àÕÛ£¬ÆðÉíÐ¦µÀ£º¡°Í¨±¨Ò»Éù£¬ËµëÞ¾Íµ½¡£¡±<p>¡¡¡¡ºéÎÄÇå£¬ËÄÊ®°ËËê£¬³àÔÆ¹úµÚÊ®°ËÈÎ»ÊµÛ£¬ÉúµÄ·½Ãæ´ó¶ú£¬ÏàÃ²ÌÃÌÃ£¬ÓÈÆä°®ÃñÈç×Ó£¬ÉîÊÜ°ÙÐÕ°®´÷£¬ËãµÃÉÏÊÇÖÎ¹úÓÐ·½µÄÒ»´úÃ÷¾ý¡£<p>¡¡¡¡¡°ÊÇ¡£¡±»¤ÎÀ½ÓÁî£¬Õâ¾Í×¼±¸Àë¿ª£¬²»¹ý×ßÖ®ºóºöÈ»¶ÙÁËÏÂ£¬ËæºóµÀ£º¡°¶ÔÁË£¬±ÝÏÂ£¬³ýÁË¹úÊ¦´óÈËÒÔ¼°ËûµÄÅ®¶ùÔÆ²ÊóïÐ¡½ãÖ®Íâ£¬»¹ÓÐÃûÄêÇáÈË¡£¾ÝÊôÏÂ°µÖÐ´òÌ½£¬´ËÈË¾ÝËµÊÇÀ×Òô¸ó¶þ³¤ÀÏÒ¶·ÉºèÖ®×ÓÒ¶½úÄþ¡£¡±<p>¡¡¡¡¡°À×Òô¸ó¶þ³¤ÀÏÖ®×Ó£¿¡±ºéÎÄÇåÖåÁËÖåÃ¼£¬ËæºóÇáÇá»ÓÁË»ÓÊÖ£º¡°ÄãÏÈÏÂÈ¥°É¡£¡±<p>¡¡¡¡¡°ÊÇ¡£¡±<p>¡¡¡¡»¤ÎÀÏÂÈ¥ºó£¬ºéÎÄÇåÏëÁËÏë£¬²»Ñ¡ÁúÅÛ£¬·´¶øÌØÒâ´©ÉÏÒ»Éí±ã×°£¬Ö®ºóÃþÁËÃþÏÂ°ÍÉÏµÄºúÐë£¬à«à«µÀ£º¡°¹úÊ¦ÔÆÓÎÊ®¶þÄê£¬Èç½ñ»ØÀ´£¬Äª²»ÊÇÒòÎªÄÇ¸ö»éÔ¼£¿¶÷£¬°´ÕÕÄê¼ÍÀ´Ëã£¬ÏëÀ´Ó¦¸ÃÊÇÁË¡£Ö»²»¹ý£¬Õâ¸öÀ×Òô¸ó¶þ³¤ÀÏµÄ¶ù×Ó£¬ÓÖÀ´×öÊ²Ã´£¿¡±<p>¡¡¡¡¡­¡­<p>¡¡¡¡ÐþÑôÒéÊÂ´óµîÖ®ÄÚ¡£<p>¡¡¡¡¡°¹þ¹þ£¬Ê®¶þÄê²»¼û£¬¹úÊ¦·ç²ÉÓÈÊ¤ÍùÎô£¬Ïë±ØÎäÑ§ÓÖÓÐ½ø¾³£¬¿ÉÏ²¿ÉºØ°¡¡£¡±»ÊµÛºéÎÄÇåÒ»½øÒéÊÂ´óµî£¬µÚÒ»ÑÛ¿´µ½µîÄÚÒ»ÃûÇàÅÛÄÐ×Ó£¬Á¢Ê±±ã¼±ÐÐÁ½²½£¬Á½ÈËË«ÊÖÏàÎÕ£¬¹þ¹þ´óÐ¦£¬ºéÎÄÇåµÀ£º¡°ÕâÒ»±ðÊ®¶þÄê£¬¿ÉÕæÊÇÏëÉ·ÎÒÒ²£¡¡±<p>¡¡¡¡ÄÇÖÐÄêÄÐ×Ó¿´Äê¼Í´óÔ¼ÎåÊ®À´Ëê£¬ÊÖÖÐÒ»°Ñ¶ìÃ«ÓðÉÈ£¬Í·ÉÏÁ½³ßÇà½í£¬ò¢ÏÂÈýç¸³¤Ðë£¬ÉúµÄÏÉ·çµÀ¹Ç£¬¹ËÅÎÖ®¼ä¼«ÓÐÆøÊÆ¡£<p>¡¡¡¡ÄÜ½ÐºéÎÄÇåÕâÎ»»ÊµÛÈç´Ë¶Ô´ý£¬´ËÈËÕýÊÇ³àÔÆ¹ú»¤¹úÎäÉñ£¬±»×ðÎª¹úÊ¦µÄÆßÐÇÎäÕß£¬ÔÆÊ¥Ðù£¡<p>¡¡¡¡¡°±ÝÏÂ¿ÍÆø¡£¡±ÉíÎª»¤¹úÎäÉñ£¬ÔÆÊ¥ÐùÓë»ÊµÛºéÎÄÇåÖ®¼äµÄ¹ØÏµÒ»Ïò¼«ÎªÁ¼ºÃ£¬ÏòÀ´Æ½ÆðÆ½×ø£¬Ò²²»ÒâÍâ£¬Î¢Ð¦µÀ£º¡°´Ë·¬ÔÆÄ³ÔÆÓÎ¹éÀ´£¬ÊµÊÇÎªÁËÐ¡Å®²ÊóïµÄ»éÊÂ¶øÀ´¡£¡±<p>¡¡¡¡¹ûÈ»£¡<p>¡¡¡¡ÌýÁËÔÆÊ¥ÐùµÄ»°£¬ºéÎÄÇåµ±¼´ÏòÖ®Ç°Ò»Ö±Õ¾ÔÚÔÆÊ¥Ðù²»Ô¶´¦µÄÒ»ÃûÉÙÅ®¿´È¥¡£<p>¡¡¡¡ÄÇÉÙÅ®´©×ÅÒ»Éíµ­ÂÌÉ«µÄ³¤È¹£¬´óÔ¼¶þ°ËÄê»ª£¬Ò»Ë«ËÆË®µÄíø×Ó£¬Áé¹âÉÁ¶¯¡£·ôÈçÄýÖ¬£¬Ñ©°×ÖÐÍ¸×Å·Ûºì£¬ËÆºõÄÜÅ¡³öË®À´¡£ÓñÊÖÊ®Ö¸ÏËÏË£¬ÈáÈôÎÞ¹Ç£¬ÇáÇáµÄÎÕ×ÅÒ»Ö§Í¨Ìå±ÌÂÌµÄÓñóï¡£Ò»Í·¼°ÑüÇàË¿Ëæ·çÎè¶¯£¬ÏÉ×Ó°ãÉñ²ÊÍÑË×¡£<p>¡¡¡¡¡°ÔÆ²Êóï£¬¼û¹ý±ÝÏÂ¡£¡±ÑÛ¼ûºéÎÄÇåÍûÀ´£¬ÉÙÅ®µ±¼´ÐÐÀñ¡£<p>¡¡¡¡¾øÃÀµÄÈË£¬¾øºÃµÄ³öÉí¡£<p>¡¡¡¡ÕýÊÇ»¤¹úÎäÉñÔÆÊ¥ÐùÎ¨Ò»µÄÕÆÉÏÃ÷Öé£¬ÔÆ²Êóï£¡<p>¡¡¡¡¡°°¥Ñ½£¬ÕâÊ®¶þÄêÃ»¼û£¬²Êóï¾ÓÈ»ÒÑ¾­³öÂäµÄÈç´ËÃÀÀö£¡¡±ºéÎÄÇåÖ®Ç°¾Í×¢Òâµ½ÁËÕâ¾øÃÀµÄÉÙÅ®£¬´ËÊ±ÌýËýÈ·ÈÏ£¬Ô½·¢ÐÀÏ²£¬Ð¦µÀ£º¡°´ó¼Ò¶¼²»ÊÇÍâÈË£¬½ÐÎÒÊåÊå°É£¬ÕâÑùÏÔµÃÇ×½üÐ©¡£¡±<p>¡¡¡¡ÔÆ²ÊóïÇáÇáµÄ¡°¶÷¡±ÁËÒ»Éù£¬ËãÊÇ´ðÓ¦¡£<p>¡¡¡¡ÔÆÊ¥ÐùÁ½ÈËº®êÑÍê±Ï£¬ºéÎÄÇå×îºó¿´ÏòÄÇÃû»¤ÎÀ¿ÚÖÐµÄÄêÇáÈË£¬Î¢Ð¦×ÅÎÊµÀ£º¡°ÕâÎ»ÉÙÏÀÊÇ¡­¡­¡±<p>¡¡¡¡ÌýµÃºéÎÄÇåÎÊÆð£¬ÄêÇáÈË¸Ï½ô±§È­ÐÐÀñ£º¡°Áù´óÕýÍ³Ö®Ò»£¬À×Òô¸ó×ùÏÂ¶þ³¤ÀÏÖ®×Ó£¬Ò¶½úÄþ¼û¹ý±ÝÏÂ¡£¡±ÕâÒ¶½úÄþÉúµÄÒÇ±íÌÃÌÃ£¬½£Ã¼ÐÇÄ¿£¬Ò»Éí°×ÒÂÈçÑ©£¬¶ËµÄÒ»¸±ºÃÂôÏà¡£Ö»²»¹ýËûËäÈ»ÊÇÔÚÏòºéÎÄÇåÐÐÀñ£¬¿ÉÊÇÑÛÉñÈ´²»Ê±ÍµÍµ¿´ÏòÔÆ²Êóï£¬ÐÐÎª¹îÒì¡£<p>¡¡¡¡¡°¶÷£¬Ó¢ÐÛ³öÉÙÄê£¬²»´í£¬²»´í¡£¡±Ò¶½úÄþµÄÑÛÉñ±»ºéÎÄÇå¿´ÔÚÑÛÀï£¬Ë²Ï¢Ö®¼äºéÎÄÇå±ãÃ÷°×¹ýÀ´£¬ËûÃ¼Í·Î¢Î¢ÖåÁËÖå£¬²»¹ýºÜ¿ìÊæÕ¹¿ªÀ´£¬ÇáÇáµãÁËµãÍ·£¬Í¬Ê±ÐÄÖÐÈ´°µµÀ£º¡°²ÊóïÓ×Äê±ãÒòÎªÌì¸³¼«¼Ñ£¬ËÍÍùÁù´óÕýÍ³Ö®Ò»µÄÀ×Òô¸óÐÞÁ¶£¬Èç½ñËý»ØÀ´£¬È´¸úÀ´ÁËÕâÃ´Ò»¸öÈË£¬¿´Ñù×Ó½ñÌìµÄÊÂÇé£¬ÅÂÊÇÃ»ÄÇÃ´¼òµ¥¡£¡±<p>¡¡¡¡¡°ºÇºÇ£¬±ÝÏÂ£¬¡±Èç½ñÈËÒÑ¼û¹ý£¬ÔÆÊ¥ÐùÓðÉÈÇáÒ¡£¬Ð¡ÉùÌáÐÑµÀ£º¡°°´ÕÕµ±ÄêµÄ»éÔ¼£¬²»Öª¾Å»Ê×Ó£¬¿É»¹°²ºÃ£¿¡±<p>¡¡¡¡³àÔÆ¹ú£¬Ã¿ÈÎ»ÊµÛ£¬Óëµ±´ú»¤¹úÎäÉñ£¬±ØÐëÒª½á³É»éÔ¼£¬ÓÃÒÔÀ­½üË«·½Ö®¼äµÄÓÑºÃ¹ØÏµ¡£<p>¡¡¡¡Èç¹û»¤¹úÎäÉñÑ¡µÄÊÇÄÐ£¬ÄÇ»ÊµÛÕâ±ß£¬±ãÑ¡Ò»¹«Ö÷ÏÂ¼Þ£»¶øÈôÊÇ»¤¹úÎäÉñÕâ±ßÊÇÅ®£¬ÄÇ»ÊµÛÕâ±ß£¬ÔòÒªÑ¡Ò»Ãû»Ê×ÓÓ­È¢´ËÅ®¹ýÃÅ¡£<p>¡¡¡¡±¾ÈÎ»¤¹úÎäÉñÔÆÊ¥ÐùÏ¥ÏÂÖ»ÓÐÒ»Å®£¬¹Ê¶ø£¬ºéÎÄÇå±ãÐèÒªÑ¡³öÒ»Ãû»Ê×ÓÀ´ÓëÖ®³É»é¡£¶øÕâÃû±»Ñ¡¶¨µÄ»Ê×Ó£¬ÕýÊÇÓëÔÆ²ÊóïÄêÁäÏàµ±£¬×îÊÜºéÎÄÇåÏ²°®ÐÄÌÛµÄ¾Å»Ê×Ó£¬ºéÐ¡±¦£¡<p>¡¡¡¡Ö®ËùÒÔ½ÐºéÐ¡±¦£¬ÊÇÒòÎª¾Å»Ê×Ó³öÉúÊ±Éí×Ó¹ÇÏà¶Ôµ¥±¡£¬ÓÖÊÇ×îÐ¡µÄ±¦±´¸í´ñ£¬ËùÒÔÆðÃûÐ¡±¦£¬ÒâË¼ÊÇÏ£ÍûËûÏñ´«¼Ò±¦Ò»Ñù¼áÍ¦¡ª¡ªËµÆðÀ´£¬Õâ¸öÃû×Ö»¹ÊÇµ±ÄêÔÆÊ¥ÐùÇ××ÔÈ¡µÄ¡£<p>¡¡¡¡¡°°¦£¬ÎÒ¼ÒÐ¡±¦Õâº¢×Ó£¬¡±ÌáÆð×Ô¼ºÕâ¸ö×îÌÛ°®È´Ò²ÊÇ×î½ÐÈËÍ·ÌÛµÄµÚ¾Å×Ó£¬ºéÎÄÇåÒ²ÊÇÓÐÐ©ÓôÃÆ£º¡°ËûÒ»Ììµ¹ÊÇÍ¦¿ªÐÄµÄ£¬¾ÍÊÇÓÐÐ©²»ÎñÕýÒµ£¬½ÐÈËÍ·ÌÛ¡£¡±<p>¡¡¡¡¡°Å¶£¿²»ÎñÕýÒµ£¿¡±ÌýÁËÕâ»°£¬ÔÆÊ¥Ðù¶ÙÊ±ÖåÃ¼µÀ£º¡°Äª·Çµ±ÄêÎÒÁô¸øËûµÄÄÇ²¿¡¶Á÷ÔÆÓ¥×¦¹¦¡·£¬ËûÒ»Ö±¶¼Ã»ÓÐÐÞÁ¶£¿¡±<p>¡¡¡¡ºéÐ¡±¦ÌåÖÊÈõ£¬ÔÆÊ¥ÐùÊÇÖªµÀµÄ£¬Ëû³öÓÎÇ°ÌØÒâÁô¸øºéÐ¡±¦Ò»²¿¹¦·¨ÃØ¼®½ÐËûÐÞÁ¶£¬ÒÔ±ã½ÐËûÇ¿Éí½¡Ìå£¬¿ÉÊÇÌýºéÎÄÇåµÄ»°£¬Õâ¹¦·¨£¬ËûÃ»Á·£¿<p>¡¡¡¡¡°°¦£¬Ò»ÑÔÄÑ¾¡°¡£¬¡±ºéÎÄÇå³¤Ì¾¿ÚÆø£¬ÎÞÄÎµÄÒ¡ÁËÒ¡Í·£¬µÀ£º¡°¹úÊ¦»¹ÊÇÓëÎÒÒ»µÀ£¬ÏÈÈ¥¿´¿´ÔÙËµ°É¡£¡±<p>¡¡¡¡ÕâÊÇÔõÃ´»ØÊÂ£¬¿´Ñù×ÓÊÂÇéºÃÏñ²»´ó¶ÔÍ·¡£<p>¡¡¡¡ÔÆÊ¥ÐùÓëÅ®¶ùÔÆ²Êóï¶ÔÊÓÒ»ÑÛ£¬Ò»ÆðµãÍ·£º¡°ºÃ¡£¡±Õâ¾Í³öÃÅ£¬Ç°Íù¾Å»Ê×ÓËùÔÚµÄÐþÏö¸ó¡£Ò»Â·ÉÏ£¬Ò¶½úÄþ½ô¸úÔÆ²ÊóïÉí±ß£¬ºéÎÄÇå¿´ÔÚÑÛÖÐ£¬ÐÄÏÂÒÑ¾­ÁËÈ»¡£<p>¡¡¡¡½á¹û¸ÕÒ»½øÐþÏö¸óµÄ´óÃÅ£¬¾Í¿´µ½ÎÞÊýµÄÆÍÈËÄÃ×ÅÌúÇÂ²ù×ÓÒ»Â··É±¼£¬Óë´ËÍ¬Ê±Ò»Õó¶ñ³ôÎ¶´«À´£¬ºéÎÄÇåµÈËÄÈËÅ¤Í·¿´È¥£¬¾ÓÈ»¿´µ½¼¸ÃûÆÍÈËÍÆ¹ýÀ´Ò»Á¾·à³µ£¡<p>¡¡¡¡Ò»ÏòÏÉ·çµÀ¹ÇÆÄÓÐµãÆ®Æ®ºõÓð»¯µÇÏÉÎ¶µÀµÄÔÆÊ¥ÐùÖ±½Ó¾Í¿´´ôÁËÑÛ£º¡°Ê²Ã´Çé¿ö£¿¡±<p>¡¡¡¡ÕýËùÎ½ÊÂ³ö·´³£±ØÓÐÑý£¬Ò»½øÃÅ¾Í¿´µ½ÕâÃ´¸öÇé¿ö£¬ËÄÈË·´µ¹²»×Å¼±ÁË£¬ºéÎÄÇåÖÆÖ¹ÁËÓûÒªÇë°²µÄÆÍÈË£¬×öÁË¸öàäÉùµÄÊÖÊÆ£¬Ëæºó¸úÔÚºóÃæ£¬¿´¿´Õâµ½µ×ÊÇÔÚ¸ãÊ²Ã´Ðþ»ú¡£<p>¡¡¡¡Ò»Â·Ç°ÐÐ£¬ºÜ¿ì£¬ËÄÈË¾ÍÌýµ½Ò»°ÑÂÔÎ¢ÓÐÐ©ÄêÇáµÄÉùÒôÏìÆð¡ª¡ª<p>¡¡¡¡¡°ÚÀ£¬ÊÖ½ÅÒ»¶¨Òª×ÐÏ¸µã£¬¶¼Ìýµ½ÁËÂð£¿Õâ¿ÉÊÇ×î¹Ø¼üµÄÒ»²½£¬²»ÄÜÓÐ°ëµã²î´í£¡¡±ÄÇÉùÒôÌýÆðÀ´´óÔ¼¶þÊ®À´Ëê£¬ËäÈ»²»ËãµÍ³ÁÐÛºñ£¬¿ÉÊÇÌýÔÚ¶úÖÐµ¹Ò²Í¦Êæ·þ£¬ºéÎÄÇåÐ¦µÀ£º¡°Ð¡±¦Õâº¢×Ó£¬²»ÖªÔÚ¸ãÊ²Ã´£¬ÎÒÃÇÇÒÏÈ¿´ËûÒ»¿´£¡¡±<p>¡¡¡¡ÔÆÊ¥ÐùºÍÔÆ²ÊóïÒ²ÊÇºÃÆæ£¬Ò¶½úÄþÔò¹´ÁË¹´×ì½Ç£¬³¶³öÒ»Ë¿²»Ð¼µÄÐ¦Òâ¡£ÈýÈË¸úºéÎÄÇåÒ»ÆðÈÌ×¡³ôÎ¶£¬ÕÒÁË¸öÒþ±ÎµÄµØ·½£¬ÍµÍµ¹Û¿´¡£<p>¡¡¡¡ÕâÀïµØ´¦ÐþÏö¸óµÄºóÔº£¬ÖÜÎ§ÂÌÊ÷´Ð´Ð£¬Ð¡ÇÅÁ÷Ë®£¬·ç¾°¼«¼Ñ¡£¶ø¾Å»Ê×ÓºéÐ¡±¦£¬ÔòÕýÖ¸»Ó×ÅÆÍÈËÃÇÔÚµØÏÂÍÚ×ÅÊ²Ã´£¬±ßÉÏ»¹·Å×Å²»ÉÙµÄÌú¹Ü¡¢ÌÕ´ÉÆ¬¡¢Ä¾Í·Ö®ÀàµÄÆ÷Îï¡£×îÆæÃîµÄÊÇ£¬¾ÍÔÚºéÐ¡±¦µÄ²»Ô¶´¦£¬»¹°Ú×ÅÒ»¸öÆæ¹ÖµÄ°×É«ÌÕ´ÉÔì¾ÍµÄ¶«Î÷£¡<p>¡¡¡¡ÄÇ¸ö¶«Î÷ÓÐ×ÅÒ»¸öÍÖÔ²ÐÎµÄ¿Ú×Ó£¬ÏÂÃæÊÇµ××ù£¬ºóÃæ»¹´ø×ÅÒ»¸ö·½·½ÕýÕýµÄÏä×Ó£¬²»ÖªºÎÓÃ¡£<p>¡¡¡¡²»¹ýºÜ¿ìËÄÈË¾ÍÖªµÀÁË¡£Ö»ÌýºéÐ¡±¦ËµµÀ£º¡°¶¼Ð¡ÐÄÐ©¹þ£¬Õâ¿ÉÊÇÎÒÐÂÑÐÖÆ³öÀ´µÄÂíÍ°£¬ºÜ½ð¹óµÄ£¬ÌÕ´É×öµÄ£¬±ðÅö»µÁË£¡¡±<p>¡¡¡¡ÂíÍ°£¿ÄÇÊÇÊ²Ã´¶«Î÷£¿<p>¡¡¡¡Ç¿ÈÌ×¡ÐÄÖÐºÃÆæ£¬ºéÎÄÇåËÄÈË¼ÌÐøÍµ¿´¡£<p>¡¡¡¡¡°¶Ô¶Ô¶Ô£¬¾ÍÕâÑù£¬¼ÌÐøÍÚ£¬¶÷£¬ÔÙÍÚÉîµã£¬´óµã£¬ÒªÖ±ÉÏÖ±ÏÂµÄ£¡¡±ºéÐ¡±¦¼ÌÐøÖ¸»Ó£¬È´²»ÏëÆÍÈËÃÇÕýÍÚ×ÅÄØ£¬ºöÈ»Ò»ÃûÆÍÈË´óÉùµÀ£º¡°¾Å»Ê×Ó£¬ÎÒÍÚ³öÀ´Ò»¸ö¶«Î÷£¡¡±<p>¡¡¡¡ºéÐ¡±¦ºÃÆæµÀ£º¡°Ê²Ã´¶«Î÷£¿¸øÎÒ¿´¿´£¡¡±<p>¡¡¡¡¡°ÔÚÕâÄØ¡£¡±ÄÇÆÍÈË¸Ï½ô°ÑÍÚµ½µÄ¶«Î÷½»µ½ºéÐ¡±¦ÕâÎ»¾Å»Ê×ÓµÄÊÖÉÏ¡£¶«Î÷ºÜÔà£¬ÉÏÃæÕ´ÂúÄàÍÁ£¬ºéÐ¡±¦Ò²²»½éÒâ£¬Éì³öÐä×ÓÖ±½Ó²ÁÁË¸É¾»£¬Ö®ºóÖÕÓÚ¿´µ½Õâµ½µ×ÊÇ¸öÊ²Ã´¶«Î÷¡£<p>¡¡¡¡ÄÇÊÇÒ»¸ö·Ç³£Æ¯ÁÁµÄÊÖïí£¬ÉÏÃæÏâ×Åºì³È»ÆÂÌÇàÀ¶×ÏºÚ°×Ò»¹²¾Å¸öÑÕÉ«µÄÓñÊ¯£¬ÄÇÓñÊ¯ÉÏ»¹É¢·¢×ÅÒ»Ë¿µ­µ­µÄÎíÆø£¬ÃþÔÚÊÖÀï»¹ÓÐÐ©Å¯Å¯µÄ£¬·Ç³£ÉñÆæ¡£<p>¡¡¡¡¡°ºÃ¶«Î÷£¡¡±ºéÐ¡±¦´óÏ²£¬Ö±½Ó°ÑÄÇ´®Öé×Ó´÷ÊÖÍóÉÏ£¬Ð¦µÀ£º¡°Ã¿ÈËÉÍ¶þÁ½£¬»ØÍ·×Ô¼ºÈ¥Áì£¡¡±<p>¡¡¡¡ÖÚÆÍÈË¸Ð¶÷´÷µÂ£º¡°Ð»¾Å»Ê×Ó£¡¡±È»ºó¼ÌÐø¿ªÍÚ£¡<p>¡¡¡¡ÕâÒ»ÍÚ¾Í×ã×ãÍÚÁËÐ¡°ë¸öÊ±³½£¬ÖÕÓÚ´ó¿ÓÍÚºÃ£¬¾Å»Ê×ÓÏÈ½ÐÈË°ÑÀïÃæÓÃ´óÔ¼Áù³ß×óÓÒ³¤¶ÌµÄÌÕ´ÉÆ¬µæºÃ£¬È»ºóÏÂ¹ÜÂñÍÁ£¬ÔÙÖ®ºó°Ñ·à³µÍÆÀ´£¬ºéÐ¡±¦ÑÚ×Å±Ç×Ó½ÐÆÍÈË°Ñ·à³µÀïÃæÄÇ¶ñ³ôµÄ±ãÄçÖ®ÎïÈ«²¿µ¹ÁËÏÂÈ¥¡ª¡ªÕâÒ»ÏÂ³¡ÉÏ¸ü³ôÁË¡­¡­<p>¡¡¡¡ÕâÐ©»î¸ÉÍê£¬Ö®ºó¾ÍÊÇÉÏ°å¹Ì¶¨£¬×îºóÓÖ°ÑÂíÍ°¹Ì¶¨ºÃ£¬Ò»ÕóÎ¢·ç´µ¹ý£¬×ÜËãÒ»ÇÐ¸ã¶¨¡£<p>¡¡¡¡¡°Ð¡±¦£¬ÄãÕâ¹Äµ·Ê²Ã´ÄØ£¿¡±ÑÛ¿´ÕâÊ±ºòÆÍÈËÒÑ¾­¿ªÊ¼ÔÚÖÜÎ§¸ÇÐ¡·¿×Ó£¬ºéÎÄÇåËÄÈËÕâ²Å×ß³öÀ´£¬ºéÎÄÇåÖ¸×ÅÄÇ¸öÂíÍ°£¬ÎÊµÀ£º¡°Õâ¶«Î÷¸ÉÊ²Ã´µÄ£¿¡±<p>¡¡¡¡ÑÛ¼û¸¸»ÊÀ´ÎÊ£¬ºéÐ¡±¦ºÙºÙÐ¦ÁËÐ¦£¬ÓÐµã²»ºÃÒâË¼£¬´ðµÀ£º¡°Õâ¶«Î÷ÎÒ¹ÜËû½Ð³éË®ÂíÍ°£¬¿ÉÓÐÓÃÁËÄØ£¬È¥Ã©²ÞµÄÊ±ºòÖ»Òª×øÔÚÕâÉÏÃæ·½±ã£¬µÈ·½±ãÍêÁË£¬Ò»°´Õâ¿ª¹Ø£¬ÕâºóÃæË®ÏäÀïµÄË®¾ÍÁ÷ÏÂÀ´£¬È»ºó¾Í³å¸É¾»ÁË£¬ºÜÊµÓÃµÄ£¡¡±<p>¡¡¡¡¡°Äã¹Äµ·ÕâÃ´°ëÌì£¬¾ÍÊÇÅªÁË¸öÃ©²Þ³öÀ´£¿£¡¡±ºéÎÄÇåËÄÈËÈ«²¿¶¼¾ª´ôÁË£¡<p>¡¡¡¡ÓÈÆäÊÇÔÆÊ¥ÐùºÍÔÆ²Êóï£¬¸üÊÇÌýµÄÄ¿µÉ¿Ú´ô£¡Ò¶½úÄþÔòÊÇÅ¤¹ýÉíÈ¥£¬¼ç°ò²»×¡¶¶¶¯¡£<p>¡¡¡¡Õâ¾Å»Ê×ÓÖ®Ç°ºéÎÄÇå¾ÍËµËû²»ÎñÕýÒµ£¬ÏÖÔÚ¿´À´£¬»¹ÕæÊÇ£¡<p>¡¡¡¡¡°°¡£¬µ±È»£¬²»¹âÊÇÃ©²Þ£¬¡±ºéÐ¡±¦·Ç³£µÃÒâ£¬µÀ£º¡°ÕâÊÇÎÒµÄÒ»ÏîÐÂ·¢Ã÷£¬ÎÒ¹ÜÕâ¸ö½ÐÕÓÆø³Ø£¡¼òµ¥À´Ëµ£¬¾ÍÊÇµ±ÕâÐ©¡­¡­¶÷¶÷£¬»Æ°×Ö®ÎïÔÚÕâ±ã³ØÀï·¢½ÍÖ®ºó£¬»á²úÉúÒ»ÖÖÆøÌå£¬¿ÉÒÔÓÃÀ´µã»ðÔì·¹ÓÃµÄ£¬·½±ã£¬»·±££¬·Ç³£ºÃÓÃ£¡¡±<p>¡¡¡¡ÔÆÊ¥ÐùÒÑ¾­ÓÐµãÌý²»ÏÂÈ¥ÁË¡ª¡ª²»ºÃºÃÁ·Îä£¬¸ãÕâÐ©ÓÐµÄÃ»µÄµÄÂÒÆß°ËÔãµÄ¶«Î÷£¬ÓÐÓÃ£¿<p>¡¡¡¡¡°¶÷£¬Ð¡±¦°¡£¬¡±ÔÆÊ¥ÐùÉÏÇ°Ò»²½£º¡°ÎÒµ±³õÁô¸øÄãµÄÄÇ²¿¡¶Á÷ÔÆÓ¥×¦¹¦¡·£¬Äã¿ÉÔøÏ°Á·£¿¡±<p>¡¡¡¡¡°ÄúÊÇÔÆ²®²®£¿¡±¿´µ½ÔÆÊ¥Ðù£¬ºéÐ¡±¦¼±Ã¦ÉÏÇ°¼ûÀñ£º¡°ÔÆ²®²®£¬ºÃ¾ÃÃ»¼ûÀ²£¡Äú¸øÎÒÁôµÄ¹¦·¨£¬ÎÒÊÇÒ»¶¨ÒªÁ·µÄÑ½£¡¡±<p>¡¡¡¡¡°ÄÇÁ·µÄÈçºÎÁË£¿¸øÎÒÊ¹À´¿´¿´£¿¡±ÔÆÊ¥Ðù¼±ÎÊ¡£<p>¡¡¡¡¡°°¡£¬ÄÇÅÂÊÇ²»ÐÐ£¬¡±ºéÐ¡±¦ÂúÁ³²»ºÃÒâË¼µÄ»ØµÀ£º¡°Á·ÊÇÁ·ÁË£¬¾ÍÊÇÃ»Á·³É£¡¡±<p>¡¡¡¡ÕâÊ±ºòÒ»ÅÔµÄÔÆ²Êóï¶¼ÓÐµãÌý²»ÏÂÈ¥ÁË£¬ÖåÁËÖåÃ¼£¬Å¤¹ýÉí×Ó¡£Ò¶½úÄþÔò¡°àÍ¡±µÄÒ»ÉùÐ¦ÁË³öÀ´£¬´Õµ½ÔÆ²Êóï¶ú±ß£¬Ð¡ÉùµÀ£º¡°¿´À´ÄãµÄÕâÎ´»é·ò£¬¹ûÈ»¸ú´«ËµÖÐÒ»Ñù£¬²»Ö»²»ÎñÕýÒµ£¬»¹ÊÇ¸ö·Ï²Ä£¡¡±<p>¡¡¡¡¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª<p>¡¡¡¡ÐÂÊéÐÂÆøÏó£¬ÐÂÊéÉÏ´«À²£¬ºé¼ÒÐ¡±¦¸ç»ªÀöµÇ³¡£¡<p>¡¡¡¡ºÀÃËµÄÐÖµÜÃÇ£¬ÄãÃÇ¿É»¹ÔÚ·ñ£¿<p>¡¡¡¡ÐÂµÄÕ÷³Ì¿ªÆô£¬»¹¿É·ñÅãÎÒÒ»Æð³Ë·çÆÆÀË£¬Åû¾£Õ¶¼¬£¿£¡<p>¡¡¡¡ÇóÊÕ²Ø£¬ÇóÍÆ¼ö£¬Çó´òÉÍ£¬ÇóÊ®·ÖÆÀ¼ÛÆ±£¡ÐÂÊéÇóÒ»ÇÐÖ§³Ö£¡£¡<p>¡¡¡¡¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª¡ª<p>¡¡¡¡ÁíÍâÍÆÒ»ÏÂÍê±¾ÀÏÊé¡¶ÖØÉúÖ®Éñ¼¶°Ü¼Ò×Ó¡·£¬¾ù¶©ÆÆÍò×÷Æ·£¬ÖÊÁ¿±£Ö¤£¬Êé»Ä¿É¿´¡£<p>¡¡¡¡<a href=http://www.qidian.com>ÆðµãÖÐÎÄÍøwww.qidian.com»¶Ó­¹ã´óÊéÓÑ¹âÁÙÔÄ¶Á£¬×îÐÂ¡¢×î¿ì¡¢×î»ðµÄÁ¬ÔØ×÷Æ·¾¡ÔÚÆðµãÔ­´´£¡</a><p>¡¡¡¡<a href=http://www.qidian.com>ÆðµãÖÐÎÄÍø www.qidian.com »¶Ó­¹ã´óÊéÓÑ¹âÁÙÔÄ¶Á£¬×îÐÂ¡¢×î¿ì¡¢×î»ðµÄÁ¬ÔØ×÷Æ·¾¡ÔÚÆðµãÔ­´´£¡</a><a>ÊÖ»úÓÃ»§Çëµ½m.qidian.comÔÄ¶Á¡£</a>');
\ No newline at end of file
diff --git a/tests/sample_data/compressed/unexpected-eof.gz b/tests/sample_data/compressed/unexpected-eof.gz
new file mode 100644
Binary files /dev/null and b/tests/sample_data/compressed/unexpected-eof.gz differ
diff --git a/tests/test_utils_gz.py b/tests/test_utils_gz.py
--- a/tests/test_utils_gz.py
+++ b/tests/test_utils_gz.py
@@ -1,6 +1,8 @@
 import unittest
 from os.path import join
 
+from w3lib.encoding import html_to_unicode
+
 from scrapy.utils.gz import gunzip, is_gzipped
 from scrapy.http import Response, Headers
 from tests import tests_datadir
@@ -66,3 +68,11 @@ def test_is_gzipped_with_charset(self):
         hdrs = Headers({"Content-Type": "application/x-gzip;charset=utf-8"})
         r1 = Response("http://www.example.com", headers=hdrs)
         self.assertTrue(is_gzipped(r1))
+
+    def test_gunzip_illegal_eof(self):
+        with open(join(SAMPLEDIR, 'unexpected-eof.gz'), 'rb') as f:
+            text = html_to_unicode('charset=cp1252', gunzip(f.read()))[1]
+            with open(join(SAMPLEDIR, 'unexpected-eof-output.txt'), 'rb') as o:
+                expected_text = o.read().decode("utf-8")
+                self.assertEqual(len(text), len(expected_text))
+                self.assertEqual(text, expected_text)