Repository: scrapy/scrapy

Issue:
FileFeedStorage creates empty file when no items are scraped
When no items are scraped, the corresponding file is created none the less, because it is created in by the `storage.open` call in `FeedExporter.open_spider`. This behavior ignores the the setting of `FEED_STORE_EMPTY` when using file export.

My proposal for this would be to add a `cleanup` method to the `IFeedStorage` interface. Then  `FeedExporter.close_spider` can call that method before returning in case `slot.itemcount` is zero and `self.store_empty` is `False`. `cleanup` could also be called internally from the `store` methods of the `IFeedStorage` implementations.

Address BatchDeliveriesTest.test_batch_path_differ failures on Windows
After https://github.com/scrapy/scrapy/pull/5632, the only remaining test failure seems to be that of `BatchDeliveriesTest.test_batch_path_differ`, which seems to be failing in Windows for Python 3.7 (and pinned dependencies), Python 3.9+, but not Python 3.8, for some reason.

It is also weird that the error seems to suggest an extra item is yielded (4 != 3) in those scenarios. I cannot imagine what could cause an extra item yield in a specific platform.


Discussion:
@gbirke I have met the same problem, there are so many empty files when you save data to files, and your proposal sounds like a good idea to me.  `IFeedStorage` need to be added a new method called in `close_spider` of `FeedExporter`

I'm having a similar issue, where the `JSONItemExporter` will write out the opening `[` when calling it's `start_exporting` method but if there are no items scraped, it's `finish_exporting` method never gets called, and the closing `]` is never added, so any resources consuming that JSON file will throw errors because of the invalid JSON format. 

The bug has not been resolved yet. I don't believe the resolution proposed in the PR #2258 was sufficient. Although it may fix the issue of broken JSON, the problem of FileFeedStorage creating empty files remains.
This is because FileFeedStorage.open open the actual file within it, which generates the file even if there is no writing happening.
In my opinion, one of the following is necessary:
1. Only open the file when the first item is received, similar to how start_exporting works.
2. Make FileFeedStorage use a TemporaryFile like BlockingFeedStorage does.
So I hope this issue will be reopened.
https://github.com/scrapy/scrapy/pull/916#issuecomment-247939197
> The problem with removing file is that it doesn't play nicely with the default behavior of appending to the result.

While removing the file may cause problems with appending, I think delaying the direct opening of the file would  be able to work well with the appending behavior and prevent the creation of an empty file.
Since there doesn't seem to be much demand for the patch, I am thinking of making my own patch. However, I am not sure which of the two solutions suggested earlier is better. 
Creating a temporary file and writing to it seems like unnecessary overhead. On the other hand, opening the file for the first time at the moment of writing and trying to write to it would require a major change in logic and look bad.
My gut says postponing file creation until there is an item to write is the way to go, however ugly it might be. But my opinion may change depending on how ugly things can get :sweat_smile: 
Let's see how the return value of IFeedStorage.open() is used.
https://github.com/scrapy/scrapy/blob/8fbebfa943c3352f5ba49f46531a6ccdd0b52b60/scrapy/extensions/feedexport.py#L426-L450
Here is the first problem.
The _FeedSlot is a private class within feedexporter, so it is easy to fix. ItemExporter, on the other hand, is a class outside of the module. I don't yet know the inner workings of scrapy, so I don't know how much modification to the source code would be required to change the interface of the ItemExporter.
Then it seems like a simple answer to let _FeedSlot manage whether the file and ItemExporter are loaded.
The _FeedSlot is given the necessary settings to open the file and initialize the ItemExporter in the constructor, and then processes them when the file or ItemExporter is first requested.
I have finished creating the fix and am checking for conflicts with the test, but in doing so I am not sure if this should be done as a "fix".
Is this something that should be a destructive change to FEED_STORE_EMPTY as a bug...? Or should I create a new option to not export files when empty feeds?

At least the docs.

> FEED_STORE_EMPTY
> Default: False
> Whether to export empty feeds (i.e. feeds with no items).

I feel this part is one of the reasons for this confusion.
What does "Whether" mean?
Write out an empty feed or nothing?
Does "Nothing" mean no file is created? Does it mean an empty file is created?
To me the documentation reads clearly enough: an empty feed is a feed with no items, and `FEED_STORE_EMPTY=False` should cause such feeds to not be exported.

That doesn’t mean we cannot improve it, e.g. by explicitly saying that a file without items is not even created. But it means that it feels like considering the creation of a file a bug when `FEED_STORE_EMPTY=False` seems OK to me.
If the current behavior is a bug, then I feel it is more necessary to fix the behavior of other features as well.
This fix results in the command line option -O or --overwrite-output no longer creating an empty file, which causes the test to fail.
It is easy to modify the test to a test case such that the item exists, but I feel that is not testing an edge case.
When fixing this Issue as a bug, it feels more natural to implicitly set FEED_STORE_EMPTY=True when --output or --overwrite-output, or to set the default for FEED_STORE_EMPTY to True. What would you suggest?

> It feels more natural to implicitly set FEED_STORE_EMPTY=True when --output or --overwrite-output, or to set the default for FEED_STORE_EMPTY to True. What would you suggest?

That’s a very good question, and I am not entirely sure of what would be best. I think it may make sense to do that, but only for `-O`, not for `-o`, and in any case it should be possible to override with `-s FEED_STORE_EMPTY=False`.

Changing the global default may not be good, since it would be a backward incompatible case… unless there is no single scenario where this bug does not happen, in which case `FEED_STORE_EMPTY=True` has been the de-facto default at least since the bug was introduced, and making it the actual default as we fix the bug would be *less* backward-incompatible.

@kmike @wRAR Any thoughts?
>  unless there is no single scenario where this bug does not happen, in which case FEED_STORE_EMPTY=True has been the de-facto default at least since the bug was introduced,

A modest change in behavior will occur.
Whereas before a completely empty file was generated, that modification will now generate a formatted empty representation, e.g., [] for json, xml declaration and empty elements for xml, etc.
However, my personal opinion is that empty files were never really expected in the first place.
Apparently, I can indirectly solve #5633 and directly solve [the issue mentioned in this PR](https://github.com/scrapy/scrapy/pull/5639#issuecomment-1259655714) while solving this issue.
The creation of empty files during batch testing is essentially the same problem as this issue, and could be solved at the same time by adding an exception to match the batch process.
Since the empty file creation will no longer occur, there is no need for Windows exception handling related to this behavior.


Resolving patch:
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -277,8 +277,6 @@ def _store_in_thread(self, file):
 class FeedSlot:
     def __init__(
         self,
-        file,
-        exporter,
         storage,
         uri,
         format,
@@ -286,9 +284,14 @@ def __init__(
         batch_id,
         uri_template,
         filter,
+        feed_options,
+        spider,
+        exporters,
+        settings,
+        crawler,
     ):
-        self.file = file
-        self.exporter = exporter
+        self.file = None
+        self.exporter = None
         self.storage = storage
         # feed params
         self.batch_id = batch_id
@@ -297,15 +300,44 @@ def __init__(
         self.uri_template = uri_template
         self.uri = uri
         self.filter = filter
+        # exporter params
+        self.feed_options = feed_options
+        self.spider = spider
+        self.exporters = exporters
+        self.settings = settings
+        self.crawler = crawler
         # flags
         self.itemcount = 0
         self._exporting = False
+        self._fileloaded = False
 
     def start_exporting(self):
+        if not self._fileloaded:
+            self.file = self.storage.open(self.spider)
+            if "postprocessing" in self.feed_options:
+                self.file = PostProcessingManager(
+                    self.feed_options["postprocessing"], self.file, self.feed_options
+                )
+            self.exporter = self._get_exporter(
+                file=self.file,
+                format=self.feed_options["format"],
+                fields_to_export=self.feed_options["fields"],
+                encoding=self.feed_options["encoding"],
+                indent=self.feed_options["indent"],
+                **self.feed_options["item_export_kwargs"],
+            )
+            self._fileloaded = True
+
         if not self._exporting:
             self.exporter.start_exporting()
             self._exporting = True
 
+    def _get_instance(self, objcls, *args, **kwargs):
+        return create_instance(objcls, self.settings, self.crawler, *args, **kwargs)
+
+    def _get_exporter(self, file, format, *args, **kwargs):
+        return self._get_instance(self.exporters[format], file, *args, **kwargs)
+
     def finish_exporting(self):
         if self._exporting:
             self.exporter.finish_exporting()
@@ -406,11 +438,16 @@ def get_file(slot_):
                 return slot_.file.file
             return slot_.file
 
-        slot.finish_exporting()
-        if not slot.itemcount and not slot.store_empty:
-            # We need to call slot.storage.store nonetheless to get the file
-            # properly closed.
-            return defer.maybeDeferred(slot.storage.store, get_file(slot))
+        if slot.itemcount:
+            # Normal case
+            slot.finish_exporting()
+        elif slot.store_empty and slot.batch_id == 1:
+            # Need to store the empty file
+            slot.start_exporting()
+            slot.finish_exporting()
+        else:
+            # In this case, the file is not stored, so no processing is required.
+            return None
 
         logmsg = f"{slot.format} feed ({slot.itemcount} items) in: {slot.uri}"
         d = defer.maybeDeferred(slot.storage.store, get_file(slot))
@@ -455,23 +492,7 @@ def _start_new_batch(self, batch_id, uri, feed_options, spider, uri_template):
         :param uri_template: template of uri which contains %(batch_time)s or %(batch_id)d to create new uri
         """
         storage = self._get_storage(uri, feed_options)
-        file = storage.open(spider)
-        if "postprocessing" in feed_options:
-            file = PostProcessingManager(
-                feed_options["postprocessing"], file, feed_options
-            )
-
-        exporter = self._get_exporter(
-            file=file,
-            format=feed_options["format"],
-            fields_to_export=feed_options["fields"],
-            encoding=feed_options["encoding"],
-            indent=feed_options["indent"],
-            **feed_options["item_export_kwargs"],
-        )
         slot = FeedSlot(
-            file=file,
-            exporter=exporter,
             storage=storage,
             uri=uri,
             format=feed_options["format"],
@@ -479,9 +500,12 @@ def _start_new_batch(self, batch_id, uri, feed_options, spider, uri_template):
             batch_id=batch_id,
             uri_template=uri_template,
             filter=self.filters[uri_template],
+            feed_options=feed_options,
+            spider=spider,
+            exporters=self.exporters,
+            settings=self.settings,
+            crawler=getattr(self, "crawler", None),
         )
-        if slot.store_empty:
-            slot.start_exporting()
         return slot
 
     def item_scraped(self, item, spider):
@@ -565,14 +589,6 @@ def _storage_supported(self, uri, feed_options):
         else:
             logger.error("Unknown feed storage scheme: %(scheme)s", {"scheme": scheme})
 
-    def _get_instance(self, objcls, *args, **kwargs):
-        return create_instance(
-            objcls, self.settings, getattr(self, "crawler", None), *args, **kwargs
-        )
-
-    def _get_exporter(self, file, format, *args, **kwargs):
-        return self._get_instance(self.exporters[format], file, *args, **kwargs)
-
     def _get_storage(self, uri, feed_options):
         """Fork of create_instance specific to feed storage classes
 
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -141,7 +141,7 @@
 FEED_TEMPDIR = None
 FEEDS = {}
 FEED_URI_PARAMS = None  # a function to extend uri arguments
-FEED_STORE_EMPTY = False
+FEED_STORE_EMPTY = True
 FEED_EXPORT_ENCODING = None
 FEED_EXPORT_FIELDS = None
 FEED_STORAGES = {}


Test patch:
diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@@ -726,10 +726,9 @@ def run_and_export(self, spider_cls, settings):
                 yield crawler.crawl()
 
             for file_path, feed_options in FEEDS.items():
-                if not Path(file_path).exists():
-                    continue
-
-                content[feed_options["format"]] = Path(file_path).read_bytes()
+                content[feed_options["format"]] = (
+                    Path(file_path).read_bytes() if Path(file_path).exists() else None
+                )
 
         finally:
             for file_path in FEEDS.keys():
@@ -946,9 +945,10 @@ def test_export_no_items_not_store_empty(self):
                 "FEEDS": {
                     self._random_temp_filename(): {"format": fmt},
                 },
+                "FEED_STORE_EMPTY": False,
             }
             data = yield self.exported_no_data(settings)
-            self.assertEqual(b"", data[fmt])
+            self.assertEqual(None, data[fmt])
 
     @defer.inlineCallbacks
     def test_start_finish_exporting_items(self):
@@ -1064,8 +1064,7 @@ def test_export_no_items_multiple_feeds(self):
         with LogCapture() as log:
             yield self.exported_no_data(settings)
 
-        print(log)
-        self.assertEqual(str(log).count("Storage.store is called"), 3)
+        self.assertEqual(str(log).count("Storage.store is called"), 0)
 
     @defer.inlineCallbacks
     def test_export_multiple_item_classes(self):
@@ -1732,10 +1731,9 @@ def run_and_export(self, spider_cls, settings):
                 yield crawler.crawl()
 
             for file_path, feed_options in FEEDS.items():
-                if not Path(file_path).exists():
-                    continue
-
-                content[str(file_path)] = Path(file_path).read_bytes()
+                content[str(file_path)] = (
+                    Path(file_path).read_bytes() if Path(file_path).exists() else None
+                )
 
         finally:
             for file_path in FEEDS.keys():
@@ -2236,6 +2234,9 @@ def run_and_export(self, spider_cls, settings):
 
             for path, feed in FEEDS.items():
                 dir_name = Path(path).parent
+                if not dir_name.exists():
+                    content[feed["format"]] = []
+                    continue
                 for file in sorted(dir_name.iterdir()):
                     content[feed["format"]].append(file.read_bytes())
         finally:
@@ -2419,10 +2420,11 @@ def test_export_no_items_not_store_empty(self):
                     / self._file_mark: {"format": fmt},
                 },
                 "FEED_EXPORT_BATCH_ITEM_COUNT": 1,
+                "FEED_STORE_EMPTY": False,
             }
             data = yield self.exported_no_data(settings)
             data = dict(data)
-            self.assertEqual(b"", data[fmt][0])
+            self.assertEqual(0, len(data[fmt]))
 
     @defer.inlineCallbacks
     def test_export_no_items_store_empty(self):
@@ -2536,9 +2538,6 @@ def test_batch_item_count_feeds_setting(self):
             for expected_batch, got_batch in zip(expected, data[fmt]):
                 self.assertEqual(expected_batch, got_batch)
 
-    @pytest.mark.skipif(
-        sys.platform == "win32", reason="Odd behaviour on file creation/output"
-    )
     @defer.inlineCallbacks
     def test_batch_path_differ(self):
         """
@@ -2560,7 +2559,7 @@ def test_batch_path_differ(self):
             "FEED_EXPORT_BATCH_ITEM_COUNT": 1,
         }
         data = yield self.exported_data(items, settings)
-        self.assertEqual(len(items), len([_ for _ in data["json"] if _]))
+        self.assertEqual(len(items), len(data["json"]))
 
     @defer.inlineCallbacks
     def test_stats_batch_file_success(self):
@@ -2647,7 +2646,7 @@ def parse(self, response):
             crawler = get_crawler(TestSpider, settings)
             yield crawler.crawl()
 
-        self.assertEqual(len(CustomS3FeedStorage.stubs), len(items) + 1)
+        self.assertEqual(len(CustomS3FeedStorage.stubs), len(items))
         for stub in CustomS3FeedStorage.stubs[:-1]:
             stub.assert_no_pending_responses()