Repository: scrapy/scrapy

Issue:
OSError when downloading a very long url
When you run into some horrible image url, like this：

```
https://o.aolcdn.com/images/dims?resize=2000%2C2000%2Cshrink&image_uri=https%3A%2F%2Fo.aolcdn.com%2Fimages%2Fdimse%2F5845cadfecd996e0372f%2Fccc34660c41122e3170c0d586c151a29397c0fcf%2FY3JvcD0xOTIwJTJDMTA5NyUyQzAlMkMwJnF1YWxpdHk9ODUmZm9ybWF0PWpwZyZyZXNpemU9MTYwMCUyQzkxNCZpbWFnZV91cmk9aHR0cHMlM0ElMkYlMkZzLnlpbWcuY29tJTJGb3MlMkZjcmVhdHItdXBsb2FkZWQtaW1hZ2VzJTJGMjAxOS0wOCUyRjg2YjNlYjkwLWI5YjgtMTFlOS05ZWFlLTQ5YWU2NTcxMjM0MyZjbGllbnQ9YTFhY2FjM2UxYjMyOTA5MTdkOTImc2lnbmF0dXJlPTZmZWJkYjQwN2E0NzU0YzM0YTJjY2ViMDczNDc1YTE1ZjBiODA3OGQ%3D&client=a1acac3e1b3290917d92&signature=bf3461468aef0cb3ecaea00d2ed611e04a88bc70
```
Then...

```
Traceback (most recent call last):
  File "c:\program files\python37\lib\site-packages\scrapy\pipelines\files.py", line 419, in media_downloaded
    checksum = self.file_downloaded(response, request, info)
  File "c:\program files\python37\lib\site-packages\scrapy\pipelines\files.py", line 452, in file_downloaded
    self.store.persist_file(path, buf, info)
  File "c:\program files\python37\lib\site-packages\scrapy\pipelines\files.py", line 53, in persist_file
    with open(absolute_path, 'wb') as f:
OSError: [Errno 22] Invalid argument: 'E:\\2019-08-12\\resources\\885443110bae0e1149e017dbea5ca3935efa38c0.com%2Fimages%2Fdimse%2F5845cadfecd996e0372f%2F108a4af73772ae197fa2c4ec4e9fe7a47390433c%2FY3JvcD0xMTc0JTJDNTgwJTJDMCUyQzAmcXVhbGl0eT04NSZmb3JtYXQ9anBnJnJlc2l6ZT0xNjAwJTJDNzkxJmltYWdlX3VyaT1odHRwcyUzQSUyRiUyRnMueWltZy5jb20lMkZvcyUyRmNyZWF0ci11cGxvYWRlZC1pbWFnZXMlMkYyMDE5LTA4JTJGMWJmZGQxNDAtYjliYy0xMWU5LWJmZjMtMjMyNzcwMTg1MzE5JmNsaWVudD1hMWFjYWMzZTFiMzI5MDkxN2Q5MiZzaWduYXR1cmU9OTFiNzQ3Y2MyZTY5ODY3OGIxNWI0OTkyMjdjM2NmZWRlYTE1NGIxOA%3D%3D&client=a1acac3e1b3290917d92&signature=6517aece82e79d536edeaccc275ad88090df0252'
```

So，I think that when downloading a file, you should use a random name instead of intercepting it from the url.For some particularly weird urls, this will cause an OSErro when writing to the file
Files Pipeline fails on URLs with "?"/get attributes
The Files Pipeline does not strip get parameters from the extension and then fails to write the file, at least on Windows/NTFS where "?" is disallowed in filenames.

e.g. `files_urls = [ "http://foo.bar/baz.txt?fizz" ]` tries to create `<hash>.txt?fizz`, raising an IOError.



Discussion:
Funny, the code actually uses a hash as file name. However, it picks the whole `.com%2Fimages%2Fdimse%2F5845cadfecd996e0372f%2F108a4af73772ae197fa2c4ec4e9fe7a47390433c%2FY3JvcD0xMTc0JTJDNTgwJTJDMCUyQzAmcXVhbGl0eT04NSZmb3JtYXQ9anBnJnJlc2l6ZT0xNjAwJTJDNzkxJmltYWdlX3VyaT1odHRwcyUzQSUyRiUyRnMueWltZy5jb20lMkZvcyUyRmNyZWF0ci11cGxvYWRlZC1pbWFnZXMlMkYyMDE5LTA4JTJGMWJmZGQxNDAtYjliYy0xMWU5LWJmZjMtMjMyNzcwMTg1MzE5JmNsaWVudD1hMWFjYWMzZTFiMzI5MDkxN2Q5MiZzaWduYXR1cmU9OTFiNzQ3Y2MyZTY5ODY3OGIxNWI0OTkyMjdjM2NmZWRlYTE1NGIxOA%3D%3D&client=a1acac3e1b3290917d92&signature=6517aece82e79d536edeaccc275ad88090df0252` as file extension :slightly_smiling_face: 
If you don’t want to wait for a fix, you can override the `file_path` method in a custom subclass. See https://docs.scrapy.org/en/latest/topics/media-pipeline.html#scrapy.pipelines.files.FilesPipeline
@Gallaecio Working on that
@mrwonko && @jschnurr what is the state of this bug? I see that fix was not merged into master. I ran into this issue Today,... Cheers!

Hey @otobrglez,

There is a PR https://github.com/scrapy/scrapy/pull/1548 which has merge+1 vote; we have a policy that two Scrapy commiters must approve a pull request before merging it to master, so currently it waits for another Scrapy commiter to approve and merge it.

Hey @kmike! 👋 Is there a way I can help to speed this up?

@otobrglez it will help if you check that PR works for you and add a comment with the results to #1548.


Resolving patch:
diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py
--- a/scrapy/pipelines/files.py
+++ b/scrapy/pipelines/files.py
@@ -5,6 +5,7 @@
 """
 import functools
 import hashlib
+import mimetypes
 import os
 import os.path
 import time
@@ -14,6 +15,7 @@
 from collections import defaultdict
 import six
 
+
 try:
     from cStringIO import StringIO as BytesIO
 except ImportError:
@@ -460,4 +462,11 @@ def item_completed(self, results, item, info):
     def file_path(self, request, response=None, info=None):
         media_guid = hashlib.sha1(to_bytes(request.url)).hexdigest()
         media_ext = os.path.splitext(request.url)[1]
+        # Handles empty and wild extensions by trying to guess the
+        # mime type then extension or default to empty string otherwise
+        if media_ext not in mimetypes.types_map:
+            media_ext = ''
+            media_type = mimetypes.guess_type(request.url)[0]
+            if media_type:
+                media_ext = mimetypes.guess_extension(media_type)
         return 'full/%s%s' % (media_guid, media_ext)


Test patch:
diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py
--- a/tests/test_pipeline_files.py
+++ b/tests/test_pipeline_files.py
@@ -57,6 +57,12 @@ def test_file_path(self):
                                    response=Response("http://www.dorma.co.uk/images/product_details/2532"),
                                    info=object()),
                          'full/244e0dd7d96a3b7b01f54eded250c9e272577aa1')
+        self.assertEqual(file_path(Request("http://www.dfsonline.co.uk/get_prod_image.php?img=status_0907_mdm.jpg.bohaha")),
+                         'full/76c00cef2ef669ae65052661f68d451162829507')
+        self.assertEqual(file_path(Request("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAR0AAACxCAMAAADOHZloAAACClBMVEX/\
+                                    //+F0tzCwMK76ZKQ21AMqr7oAAC96JvD5aWM2kvZ78J0N7fmAAC46Y4Ap7y")),
+                         'full/178059cbeba2e34120a67f2dc1afc3ecc09b61cb.png')
+                         
 
     def test_fs_store(self):
         assert isinstance(self.pipeline.store, FSFilesStore)