Repository: scrapy/scrapy

Issue:
Add more extensions to IGNORED_EXTENSIONS
A follow-up to https://github.com/scrapy/scrapy/pull/1835. For the record, in my last project I had to also add the following extensions to ignored:

```
{
    '7z', '7zip', 'xz', 'gz', 'tar', 'bz2',   # archives
    'cdr',  # Corel Draw files
    'apk', # Android packages
}
```

cdr and apk look safe; I wonder if we can add .gz: Scrapy decompresses gz responses automatically, so if we add .gz to `IGNORED_EXTENSIONS` spider may stop following e.g. sitemap.xml.gz links. 

On the other hand, for broad crawls most .gz files are archives, and we're filtering out .zip files already.

Updated the list of ignored extension in the linkextractor component
- Added following extension to the list: .7z, .7zip, .bz2, .gz, .tar, .xz, .cdr, .ico, .apk
- Moved archives into the separate section

See scrapy/scrapy#1837 issue for details and discussion.



Discussion:
I would exclude gz, that seems too fickle.
Serving gzipped content (with or without the file extension) is perfectly valid.

I think to make that case safe, the link would at least need to be pre-fetched with a HEAD request, to check if a `Content-Disposition: attachment` header is set, in which case it can be ignored.
If a crawl is seeing many downloadable `gz` archives, that means as many useless requests, but it won't break in strange ways.

> I think to make that case safe, the link would at least need to be pre-fetched with a HEAD request, to check if a Content-Disposition: attachment header is set, in which case it can be ignored.
> If a crawl is seeing many downloadable gz archives, that means as many useless requests, but it won't break in strange ways.

it looks more like https://github.com/scrapy/scrapy/issues/1772

My two cents...

I think anything that could generate the error `AttributeError: 'Response' object has no attribute 'body_as_unicode'` should not be allowed. So all archive/compression extensions should be added to `IGNORED_EXTENSIONS` by default.

But most important, I think it should exist a easy way to extend/override `IGNORED_EXTENSIONS`. Currently it is possible to completely override the values but not to extend them. So the user must inevitably dive into the code when it is needed to add one extension. If there were get/set methods to `IGNORED_EXTENSIONS`, it would be very easier to add/remove one or more extensions!

At the time a response can throw an `AttributeError` for `body_as_unicode`, the whole response must have been downloaded and decompressed. That's what `IGNORED_EXTENSIONS` is trying to avoid in the first place: downloading the resource. Otherwise we could just use MIME detection on the response content.

`IGNORED_EXTENSIONS` are probably defined in `LinkExtractor` (instead of settings) because the module was formerly in `contrib`.
Depending on whether it will stay now, or extracted into another package, the settings could be merged into the `default_settings` for easier overriding. Still, it's not that hard to do.

``` py
# in the spider or in your settings:
from scrapy.linkextractors import IGNORED_EXTENSIONS
IGNORED_EXTENSIONS += ['my', 'custom', 'extensions']
# if added in settings:
#from myproject.settings import IGNORED_EXTENSIONS
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
class MySpider(CrawlSpider):
    lx = LinkExtractor(deny_extensions=IGNORED_EXTENSIONS, allow=(), restrict_xpaths=())
    rules = (lx,)
    # ...
```

@kmike: Is cancelling a response/download the right way though? It galls me that there is no better way. Sending two requests for every resource ending in `.gz` is ugly (HEAD first, then possibly GET).

But `RST`ing the connection seems like it shouldn't be a "default action", either.
The server will be sending the response body already at that time. A RST on the connection might impact things like HTTP pipelining (Without research that's just a guess, though, not sure if twisted even supports it.)
Depending on the number of such aborted requests, that might be considered a DoS attack, since it asks the server for the _full_ response first, then aborts when the server did work to fetch or compute the ressource (possibly on-the-fly compression of something big).
Since that's a special case only for `.gz` extensions, it might not happen that often, but we should be aware of that.

I vote for adding .cdr extension.

## [Current coverage](https://codecov.io/gh/scrapy/scrapy/pull/2067?src=pr) is **83.33%**

> Merging [#2067](https://codecov.io/gh/scrapy/scrapy/pull/2067?src=pr) into [master](https://codecov.io/gh/scrapy/scrapy/branch/master?src=pr) will not change coverage
> 
> Powered by [Codecov](https://codecov.io?src=pr). Last updated by [80c296e...55f3240](https://codecov.io/gh/scrapy/scrapy/compare/80c296e091d83e91d391bed5c3cf54d55a472fcf...55f3240c5d6a7face71d67c63f327b66a1578e03)

+1, but without the `gz` extension

I've already reasoned this in the mentioned #1837 issue:
zlib and gzip are recognized/accepted for on-the-fly decompression by browsers.
Whether a gzipped file (e.g. `*.html.gz` or `*.xml.z`) is interpreted as a download or served as a page, depends on the headers which are served with the ressource.
You can't assume them to be downloads in beforehand, from a link.
To determine that reliably, at least a `HEAD` request to the ressource is necessary to determine the `Content-Type` header: "application/x-gzip" for a download, "text/html" or similar for an "inline ressource".
Filtering all these links out _in the default_ `LinkExtractor` setup is bad, IMO.

In the name of sane defaults, I would rather see these URLs in the log and fix my LE on a case-by-case basis, if they happen to be archives, than having them silently dropped and perhaps miss a whole lot of actual website content.

I would add 'tar.gz' however.

@nyov so do you propose to remove .gz, .bz2, .xz, but add tar.(gz|bz2|xz) ?

No, only `.gz` -> `.tar.gz`.
And I might veto the inclusion of `.z` entirely, if you had included that in your list.

Deflate/gzip/zlib are well-known for use with on-the-fly encoding (afaik) -> [RFC](https://tools.ietf.org/html/rfc2616#section-3.5).
The others were (back in '97 as per RFC dates) likely too memory-intensive or cpu heavy to be considered for 'on-the-fly' decompression in browsers.

A few more codings have made it into the official [IANA list](https://www.iana.org/assignments/http-parameters/http-parameters.xhtml#content-coding), and others are used unofficially (such as `bzip2`, `lzma`, `xz` - [Wikipedia article](https://en.wikipedia.org/wiki/HTTP_compression#Content-Encoding_tokens)), but these are much much less common.
I think they can be exempted from consideration here.

So, '.gz' alone is fickle but '.tar.gz' is 99% a tarball archive; all other compression file-endings are _reasonably certain_ to not contain browser-interpreted content. (`.bz2` might on Apache, see `mime.conf`)

Considering this is a default setting, that is as far as I'd go with the presets.
It's much harder to debug why your crawler _does not see and fetch these friggin' URLs_, I think, than to add them to the list when they get mis-handled, case-by-case basis.

---

The only sure-fire way, however, is to handle things like a browser, not assume content from URI names. Checking HTTP headers, detecting file MIME magic -- don't judge the book by it's cover:

```
<IfModule mod_mime.c>
<Files "*.html.ohandfuckscrapy.xz">
        RemoveType .xz
        AddType text/html .xz
</Files>

<Location "/noscrapedocs">
    # *.whocares.about.file.endings
    ForceType text/html
</Location>
</IfModule>
```

N.B.: default Apache `mime.conf`:

```
<IfModule mod_mime.c>

        # AddEncoding allows you to have certain browsers uncompress
        # information on the fly. Note: Not all browsers support this.
        # Despite the name similarity, the following Add* directives have
        # nothing to do with the FancyIndexing customization directives above.
        #
        #AddEncoding x-compress .Z
        #AddEncoding x-gzip .gz .tgz
        #AddEncoding x-bzip2 .bz2
        #
        # If the AddEncoding directives above are commented-out, then you
        # probably should define those extensions to indicate media types:
        #
        AddType application/x-compress .Z
        AddType application/x-gzip .gz .tgz
        AddType application/x-bzip2 .bz2
[...]
```


Resolving patch:
diff --git a/scrapy/linkextractors/__init__.py b/scrapy/linkextractors/__init__.py
--- a/scrapy/linkextractors/__init__.py
+++ b/scrapy/linkextractors/__init__.py
@@ -19,23 +19,26 @@
 
 # common file extensions that are not followed if they occur in links
 IGNORED_EXTENSIONS = [
+    # archives
+    '7z', '7zip', 'bz2', 'rar', 'tar', 'tar.gz', 'xz', 'zip',
+
     # images
     'mng', 'pct', 'bmp', 'gif', 'jpg', 'jpeg', 'png', 'pst', 'psp', 'tif',
-    'tiff', 'ai', 'drw', 'dxf', 'eps', 'ps', 'svg',
+    'tiff', 'ai', 'drw', 'dxf', 'eps', 'ps', 'svg', 'cdr', 'ico',
 
     # audio
     'mp3', 'wma', 'ogg', 'wav', 'ra', 'aac', 'mid', 'au', 'aiff',
 
     # video
     '3gp', 'asf', 'asx', 'avi', 'mov', 'mp4', 'mpg', 'qt', 'rm', 'swf', 'wmv',
-    'm4a', 'm4v', 'flv',
+    'm4a', 'm4v', 'flv', 'webm',
 
     # office suites
     'xls', 'xlsx', 'ppt', 'pptx', 'pps', 'doc', 'docx', 'odt', 'ods', 'odg',
     'odp',
 
     # other
-    'css', 'pdf', 'exe', 'bin', 'rss', 'zip', 'rar',
+    'css', 'pdf', 'exe', 'bin', 'rss', 'dmg', 'iso', 'apk'
 ]
 
 


Test patch: