Repository: scrapy/scrapy

Issue:
_store_in_thread must not receive a closed file when using postprocessing plugins
When using post-processing plugins, the FeedExporter closes the temporary file. This behavior has been introduced in commit 500aaa258fc72e943d7bbd258900aeb5af6c8b0d.

Some storage backends, such as GCSFeedStorage, expect an open file in the _store_in_thread method.

To solve the problem we should only close the e.g. GzipFile instance, not the temporary file. Removing the line self.file.close() in GzipPlugin, Bz2Plugin and LZMAPlugin should resolve the error. I think this change would be a clean solution because usually the component that created a resource should be responsible to close that resource - so closing the BinaryIO object in the plugins is a bit unexpected.

Furthermore, GCSFeedExporter should be modified to close the file in its _store_in_thread method after the upload. FTPFeedStorage and S3FeedStorage do this as well.
resolve issue 5932
Fixes #5932 

Removed the lines closing the temporary file in GzipPlugin, Bz2Plugin, and LMZAPlugin in postprocessing.py.

Placed file closes in GCSFeedExporter, FTPFeedStorage, and S3FeedStorage in feedexport.py.


Discussion:
I'm interested in working on this issue. It seems like a valuable improvement and aligns with my interests. I hope this issue still open and available for contributions?
Please, feel free to give it a try.
## [Codecov](https://app.codecov.io/gh/scrapy/scrapy/pull/6178?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#6178](https://app.codecov.io/gh/scrapy/scrapy/pull/6178?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (329175c) into [master](https://app.codecov.io/gh/scrapy/scrapy/commit/04024f1e796f99e77dda544396722fb9203f1d49?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (04024f1) will **decrease** coverage by `45.57%`.
> Report is 21 commits behind head on master.
> The diff coverage is `0.00%`.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           master    #6178       +/-   ##
===========================================
- Coverage   88.75%   43.18%   -45.57%     
===========================================
  Files         160      159        -1     
  Lines       11414    11581      +167     
  Branches     1860     1885       +25     
===========================================
- Hits        10130     5001     -5129     
- Misses        976     6188     +5212     
- Partials      308      392       +84     
```

| [Files](https://app.codecov.io/gh/scrapy/scrapy/pull/6178?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/extensions/postprocessing.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6178?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2V4dGVuc2lvbnMvcG9zdHByb2Nlc3NpbmcucHk=) | `41.26% <ø> (-57.22%)` | :arrow_down: |
| [scrapy/extensions/feedexport.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6178?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2V4dGVuc2lvbnMvZmVlZGV4cG9ydC5weQ==) | `67.93% <0.00%> (-24.74%)` | :arrow_down: |

... and [141 files with indirect coverage changes](https://app.codecov.io/gh/scrapy/scrapy/pull/6178/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy)

</details>
It seems this approach breaks some feedexport scenarios.

Resolving patch:
diff --git a/scrapy/extensions/postprocessing.py b/scrapy/extensions/postprocessing.py
--- a/scrapy/extensions/postprocessing.py
+++ b/scrapy/extensions/postprocessing.py
@@ -42,7 +42,6 @@ def write(self, data: bytes) -> int:
 
     def close(self) -> None:
         self.gzipfile.close()
-        self.file.close()
 
 
 class Bz2Plugin:
@@ -69,7 +68,6 @@ def write(self, data: bytes) -> int:
 
     def close(self) -> None:
         self.bz2file.close()
-        self.file.close()
 
 
 class LZMAPlugin:
@@ -111,7 +109,6 @@ def write(self, data: bytes) -> int:
 
     def close(self) -> None:
         self.lzmafile.close()
-        self.file.close()
 
 
 # io.IOBase is subclassed here, so that exporters can use the PostProcessingManager


Test patch:
diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@@ -1731,6 +1731,7 @@ def open(self, spider):
 
             def store(self, file):
                 Storage.store_file = file
+                Storage.file_was_closed = file.closed
                 file.close()
 
         settings = {
@@ -1746,6 +1747,7 @@ def store(self, file):
         }
         yield self.exported_no_data(settings)
         self.assertIs(Storage.open_file, Storage.store_file)
+        self.assertFalse(Storage.file_was_closed)
 
 
 class FeedPostProcessedExportsTest(FeedExportTestBase):