Repository: scrapy/scrapy

Issue:
Feature suggest: add downloaded/uptodate status to information about downloaded media
Currently File/Image pipelines populate `files`/`images` fields with dicts containing information about the downloaded files (the downloaded path, the original scraped url, and the file checksum). It would be useful to have downloaded/uptodate status in this dict ([motivation](https://stackoverflow.com/questions/45823196/scrapy-how-to-get-status-of-file-download)).

It goes along with other features requests such as having width/height of images also in the dict output.


Discussion:
Makes sense.
Also, the format of the dicts is not clearly documented ([it is here](https://docs.scrapy.org/en/latest/topics/media-pipeline.html#scrapy.pipelines.files.FilesPipeline.get_media_requests), but not obvious).
If this proposal is implemented, I would also suggest improving the docs on the structure of results.
+1
this is a great feature suggestion, that is general enough to have many applications on projects that utilize Scrapy!

Hope I find the time to have a try on this soon!

*update*: did a bit of research and realized that this enhancement is achievable with minimal source code alterations. Namely:

- `FilesPipeline.media_to_download#_onsuccess` callback should now return: `{'url': request.url, 'path': path, 'checksum': checksum, 'status': 'uptodate'}`
- `FilesPipeline.media_downloaded` should now return: `{'url': request.url, 'path': path, 'checksum': checksum, 'status': status}`
- the necessary testing and documentation considerations

Any feedback on this will be deeply appreciated and if this is indeed the case, I will certainly open a pull request (if that ok with you) :)
I wonder if we want to make a distinction between:
- It had been already downloaded in a previous crawl.
- It had been already downloaded in the current crawl.
I think the first approach (in a previous crawl) is sufficient and well-defined, at the right level of abstractness. 

The second approach is probably going to face significant definition challenges in Scrapy projects that utilize more than one Media Pipelines or heavily depend on a specific file expiration policy. Since the whole point of the file expiration policy is to "avoid downloading files that were downloaded recently", I think to take into account the "current crawl" semantics would really confuse the end-user.

Overall, maintaining consistency with the `file_status_count` statistics is definitely the way.

I hope I did not miss your point on this distinction :)

Resolving patch:
diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py
--- a/scrapy/pipelines/files.py
+++ b/scrapy/pipelines/files.py
@@ -417,7 +417,7 @@ def _onsuccess(result):
             self.inc_stats(info.spider, 'uptodate')
 
             checksum = result.get('checksum', None)
-            return {'url': request.url, 'path': path, 'checksum': checksum}
+            return {'url': request.url, 'path': path, 'checksum': checksum, 'status': 'uptodate'}
 
         path = self.file_path(request, info=info)
         dfd = defer.maybeDeferred(self.store.stat_file, path, info)
@@ -494,7 +494,7 @@ def media_downloaded(self, response, request, info):
             )
             raise FileException(str(exc))
 
-        return {'url': request.url, 'path': path, 'checksum': checksum}
+        return {'url': request.url, 'path': path, 'checksum': checksum, 'status': status}
 
     def inc_stats(self, spider, status):
         spider.crawler.stats.inc_value('file_count', spider=spider)


Test patch:
diff --git a/tests/test_pipeline_crawl.py b/tests/test_pipeline_crawl.py
--- a/tests/test_pipeline_crawl.py
+++ b/tests/test_pipeline_crawl.py
@@ -94,6 +94,11 @@ def _assert_files_downloaded(self, items, logs):
         file_dl_success = 'File (downloaded): Downloaded file from'
         self.assertEqual(logs.count(file_dl_success), 3)
 
+        # check that the images/files status is `downloaded`
+        for item in items:
+            for i in item[self.media_key]:
+                self.assertEqual(i['status'], 'downloaded')
+
         # check that the images/files checksums are what we know they should be
         if self.expected_checksums is not None:
             checksums = set(
diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py
--- a/tests/test_pipeline_files.py
+++ b/tests/test_pipeline_files.py
@@ -84,6 +84,7 @@ def test_file_not_expired(self):
 
         result = yield self.pipeline.process_item(item, None)
         self.assertEqual(result['files'][0]['checksum'], 'abc')
+        self.assertEqual(result['files'][0]['status'], 'uptodate')
 
         for p in patchers:
             p.stop()
@@ -105,6 +106,29 @@ def test_file_expired(self):
 
         result = yield self.pipeline.process_item(item, None)
         self.assertNotEqual(result['files'][0]['checksum'], 'abc')
+        self.assertEqual(result['files'][0]['status'], 'downloaded')
+
+        for p in patchers:
+            p.stop()
+
+    @defer.inlineCallbacks
+    def test_file_cached(self):
+        item_url = "http://example.com/file3.pdf"
+        item = _create_item_with_files(item_url)
+        patchers = [
+            mock.patch.object(FilesPipeline, 'inc_stats', return_value=True),
+            mock.patch.object(FSFilesStore, 'stat_file', return_value={
+                'checksum': 'abc',
+                'last_modified': time.time() - (self.pipeline.expires * 60 * 60 * 24 * 2)}),
+            mock.patch.object(FilesPipeline, 'get_media_requests',
+                              return_value=[_prepare_request_object(item_url, flags=['cached'])])
+        ]
+        for p in patchers:
+            p.start()
+
+        result = yield self.pipeline.process_item(item, None)
+        self.assertNotEqual(result['files'][0]['checksum'], 'abc')
+        self.assertEqual(result['files'][0]['status'], 'cached')
 
         for p in patchers:
             p.stop()
@@ -403,10 +427,10 @@ def _create_item_with_files(*files):
     return item
 
 
-def _prepare_request_object(item_url):
+def _prepare_request_object(item_url, flags=None):
     return Request(
         item_url,
-        meta={'response': Response(item_url, status=200, body=b'data')})
+        meta={'response': Response(item_url, status=200, body=b'data', flags=flags)})
 
 
 if __name__ == "__main__":