Repository: scrapy/scrapy

Issue:
Feeds Enhancement: Post-Processing
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your pull request, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#writing-patches and https://doc.scrapy.org/en/latest/contributing.html#submitting-patches

-->

## Summary

A Feed post-processing enhancement will enable plugins such as compression, minifying, beautifier, etc. which can then be added to the Feed exporting workflow.

## Motivation/Proposal

A post-processing feature can help add more extensions dedicated to before-export-processing to scrapy. To help achieve extensibility, a ```PostProcessingManager``` can be used which will use "plugin" like scrapy components to process the data before writing it to target files.

The ```PostProcessingManager``` can act like a wrapper to the slot's storage so whenever a write event takes place, the data is run through the plugins in a pipeline-ish way to be processed and then written to the target file.

A number of plugins can be created but there will be a need to specify the order in which these plugins are used as some won't be able to process the data after it has been processed by another(eg: minifying won't work on a compression processed file). These plugins will be required to have a certain Interface so that the ```PostProcessingManager``` can use it without breaking down from unidentified components.

Few built-in plugins can be made such as for compressions: gzip, lzma, bz2.

PostProcessingManager class prototype:
```python
class PostProcessingManager:
    """
    This will manage and use declared plugins to process data in a
    pipeline.
    :param plugins: all the declared plugins for the uri
    :type plugins: list
    :param file: target file whose data will be processed before write
    :type file: file like object
    """

    def __init__(self, plugins, file):
        # 1) load the plugins here
        # 2) save file as an attribute

    def write(self, data):
        """
        Uses all the declared plugins to process data first, then writes
        the processed data to target file. 
        :param data: data passed to be written to target file
        :type data: bytes
        :return: returns number of bytes written
        :rtype: int
        """

    def close(self):
        """
        Close the target file along with all the plugins.
        """
```
PostProcessorPlugin class inteface:
```python
class PostProcessorPlugin(Interface):
    """
    Interface for plugins that will be used by PostProcessingManager. This will
    provide necessary processing method.
    """

    def __init__(self, file, feed_options):
        """
        Initialize plugin with target file to which post-processed
        data will be written and the feed-specific options.
        """

    def write(self, data):
        """
        Exposed method which will take data passed, process it and then
        write it to target file.
        :param data: data passed to be written to target file
        :type data: bytes
        :return: returns number of bytes written
        :rtype: int
        """

    def close(self):
        """
        Closes this plugin wrapper.
        """

    @staticmethod
    def process(data):
        """
        This will process the data and return it.
        :param data: input data
        :type data: bytes
        :return: processed data
        :rtype: bytes
        """
```
GzipPlugin example:
```python
@implementer(PostProcessorPlugin)
class GzipPlugin:
    COMPRESS_LEVEL = 9

    def __init__(self. file):
        # initialise various parameters for gzipping
        self.file = gzip.GzipFile(fileobj=file, mode=file.mode,
                                  compresslevel=self.COMPRESS_LEVEL)

    def write(self, data):
        return self.file.write(data)

    def close(self):
        self.file.close()

    @staticmethod
    def process(data):
        return gzip.compress(data, compresslevel=self.COMPRESS_LEVEL)
```
settings.py example:
```python
from myproject.pluginfile import MyPlugin

FEEDS = {
    'item1.json' : {
        'format': 'json',
        'post-processing': ['gzip'],
    },
    'item2.xml' : {
        'post-processing': [MyPlugin,'xz'],    # order is important
    },
}


POST_PROC_PLUGINS_BASE = {
    'gzip': 'scrapy.utils.postprocessors.GzipPlugin',
    'xz' : 'scrapy.utils.postprocessors.LZMAPlugin',
    'bz2': 'scrapy.utils.postprocessors.Bz2Plugin',
}
```

## Describe alternatives you've considered

This feature idea is actually an expansion on compression support(see #2174). Item Pipelines can be used as well for compression. But implementing this feature instead can give more options to user for post-processing while making it easier for the user to activate those post-processing components for specific feeds.

## Additional context

This feature proposal is part of a GSoC project (see #4963). This issue has been created to get inputs from the Scrapy community to refine the proposed feature.

gzip-compressed item exports
I think compressing exported data can be useful in a lot of cases. It'd be good to have a built-in way for compressed exports in Scrapy. I have this implementation, but probably it makes more sense to handle .jl.gz / csv.gz / ... extensions just like .jl / .csv / ... instead of creating a storage, I'm not sure:

``` py
# -*- coding: utf-8 -*-
import os
import gzip

from zope.interface import Interface, implementer
from w3lib.url import file_uri_to_path
from scrapy.extensions.feedexport import IFeedStorage


@implementer(IFeedStorage)
class GzipFileFeedStorage(object):
    """
    Storage which exports data to a gzipped file.
    To use it, add

    ::

        FEED_STORAGES = {
            'gzip': 'myproject.exports.GzipFileFeedStorage',
        }

    to settings.py and then run scrapy crawl like this::

        scrapy crawl foo -o gzip:/path/to/items.jl

    The command above will create ``/path/to/items.jl.gz`` file
    (.gz extension is added automatically).

    Other export formats are also supported, but it is recommended to use .jl.
    If a spider is killed then gz archive may be partially broken.
    In this case it user should read the broken archive line-by-line and stop
    on gzip decoding errors, discarding the tail. It works OK with .jl exports.
    """
    COMPRESS_LEVEL = 4

    def __init__(self, uri):
        self.path = file_uri_to_path(uri) + ".gz"

    def open(self, spider):
        dirname = os.path.dirname(self.path)
        if dirname and not os.path.exists(dirname):
            os.makedirs(dirname, exist_ok=True)
        return gzip.open(self.path, 'ab', compresslevel=self.COMPRESS_LEVEL)

    def store(self, file):
        file.close()
```

What do you think?



Discussion:
As ```PostProcessingManager``` will be the wrapper around the feed's storage, I think ```write``` and ```close``` won't be needed in ```PostProcessorPlugin``` interface.
This comment is exclusively about the plugin interface, which I think is the most important one (the manager will be an internal component, but users will write their own plugins).

I don’t think the processing method should be static. Plugins should be Scrapy components (i.e. it should be possible to add a `from_crawler` method to them to initialize them with access to settings), and hence they could read custom settings that condition their behavior. I think we need a regular method that can access object variables.

And I think we should make the plugin interface similar to, for example, `zipfile.ZipFile`:

```
with Plugin(output_file_object) as plugin:
    plugin.write(data)
```

So that plugin chaining could look like this:

```
with Plugin2(output_file_object) as plugin2:
    with Plugin1(plugin2) as plugin1:
        plugin1.write(data)
```

Of course, with an arbitrary number of plugins, the manager code would have you actually call the open and close methods of those plugins, instead of using `with`, but I hope this example helps visualize the main idea: plugins should get `write` calls, and in those calls they should be able to write into the next plugin (or the final file, in the case of the last plugin), but not required to (e.g. some plugins may store the input data internally, and write it all at once into the next plugin when the input data stops [close method gets called]).
I think chained ```write``` and ```close``` methods looks simplest to me right now.
So something like:
```python
# plugin1,2,3 subclasses a parent plugin class
p1 = plugin1(original_file)
p2 = plugin2(p1)
p3 = plugin3(p2)

p3.write(data) # will be called by manager class
```
So a single write call processes the data through all the plugin and finally writes to the target storage.

> some plugins may store the input data internally, and write it all at once into the next plugin when the input data stops [close method gets called]

This is a valid method as well to achieve the pipeline. Should we keep both methods of pipelining and give user the option to choose one?
> This is a valid method as well to achieve the pipeline. Should we keep both methods of pipelining and give user the option to choose one?

Could you elaborate? Which 2 methods are we talking about?
I mean the way pipelining flow will work. 1) write data into next plugin whenever ```write``` method is called or 2) writes all at once into next plugin when ```close``` method is called as you have said.
I am thinking that we want to allow and encourage 1.

But an API designed for 1 still allows for a plugin to use 2 if needed. Scrapy could write multiple times into plugin1, but instead of calling write on plugin2 each time, plugin1 could store the input data into a temporary file, and once its close method is called, from that method it could perform a single plugin2.write() call followed by plugin.close().
Ok yeah the implementation of plugins will depend ultimately on the user(obviously :sweat_smile:). 
what about something along the lines of:

```
$ git diff
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
index c3fc66d..27e0ad3 100644
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -163,6 +163,7 @@ class FeedExporter(object):
             raise NotConfigured
         self.format = settings['FEED_FORMAT'].lower()
         self.export_encoding = settings['FEED_EXPORT_ENCODING']
+        self.export_compression = settings.get('FEED_EXPORT_COMPRESSION', False)
         self.storages = self._load_components('FEED_STORAGES')
         self.exporters = self._load_components('FEED_EXPORTERS')
         if not self._storage_supported(self.urifmt):
@@ -186,6 +187,9 @@ class FeedExporter(object):
         uri = self.urifmt % self._get_uri_params(spider)
         storage = self._get_storage(uri)
         file = storage.open(spider)
+        if self.export_compression:
+            import gzip
+            file = gzip.GzipFile(fileobj=file)
         exporter = self._get_exporter(file, fields_to_export=self.export_fields,
             encoding=self.export_encoding)
         exporter.start_exporting()
```

+1 for supporting gzip. And +1 for making it part of the feed exporter (instead of feed storage). Below are the sample lines that I'm using for a project:

```python
# coding: utf8

import gzip

from scrapy.exporters import JsonLinesItemExporter


class JsonLinesGzipItemExporter(JsonLinesItemExporter):
    """
    Sample exporter for .jl + .gz format.
    To use it, add
    ::

        FEED_EXPORTERS = {
            'jl.gz': 'myproject.exporters.JsonLinesGzipItemExporter',
        }
        FEED_FORMAT = 'jl.gz'

    to settings.py and then run scrapy crawl like this::

        scrapy crawl foo -o s3://path/to/items.jl.gz

    (if `FEED_FORMAT` is not explicitly specified, you'll need to add
    `-t jl.gz` to the command above)
    """

    def __init__(self, file, **kwargs):
        gzfile = gzip.GzipFile(fileobj=file)
        super(JsonLinesGzipItemExporter, self).__init__(gzfile, **kwargs)

    def finish_exporting(self):
        self.file.close()
```


Hm, supporting only jl.gz without supporting all other formats is not a bad idea; it is likely the most useful and least problematic compressed format.
Thanks @starrify - adapted your class for CSV:

```
from scrapy.exporters import CsvItemExporter

class GzipCsvItemExporter(CsvItemExporter):
    def __init__(self, stream, **kwargs):
        self.gzfile = gzip.GzipFile(fileobj=stream)
        super(GzipCsvItemExporter, self).__init__(self.gzfile, **kwargs)

    def finish_exporting(self):
        self.gzfile.close()
```
Would love to see this merged.  Would anyone object if we merged @redapple's comment on Aug 31, 2016 into the codebase?  I've just added it in https://github.com/scrapy/scrapy/pull/4131

@kmike, @redapple: could we merge?

Resolving patch:
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -20,6 +20,7 @@
 
 from scrapy import signals
 from scrapy.exceptions import NotConfigured, ScrapyDeprecationWarning
+from scrapy.extensions.postprocessing import PostProcessingManager
 from scrapy.utils.boto import is_botocore_available
 from scrapy.utils.conf import feed_complete_default_values_from_settings
 from scrapy.utils.ftp import ftp_store_file
@@ -393,6 +394,9 @@ def _start_new_batch(self, batch_id, uri, feed_options, spider, uri_template):
         """
         storage = self._get_storage(uri, feed_options)
         file = storage.open(spider)
+        if "postprocessing" in feed_options:
+            file = PostProcessingManager(feed_options["postprocessing"], file, feed_options)
+
         exporter = self._get_exporter(
             file=file,
             format=feed_options['format'],
diff --git a/scrapy/extensions/postprocessing.py b/scrapy/extensions/postprocessing.py
new file mode 100644
--- /dev/null
+++ b/scrapy/extensions/postprocessing.py
@@ -0,0 +1,154 @@
+"""
+Extension for processing data before they are exported to feeds.
+"""
+from bz2 import BZ2File
+from gzip import GzipFile
+from io import IOBase
+from lzma import LZMAFile
+from typing import Any, BinaryIO, Dict, List
+
+from scrapy.utils.misc import load_object
+
+
+class GzipPlugin:
+    """
+    Compresses received data using `gzip <https://en.wikipedia.org/wiki/Gzip>`_.
+
+    Accepted ``feed_options`` parameters:
+
+    - `gzip_compresslevel`
+    - `gzip_mtime`
+    - `gzip_filename`
+
+    See :py:class:`gzip.GzipFile` for more info about parameters.
+    """
+
+    def __init__(self, file: BinaryIO, feed_options: Dict[str, Any]) -> None:
+        self.file = file
+        self.feed_options = feed_options
+        compress_level = self.feed_options.get("gzip_compresslevel", 9)
+        mtime = self.feed_options.get("gzip_mtime")
+        filename = self.feed_options.get("gzip_filename")
+        self.gzipfile = GzipFile(fileobj=self.file, mode="wb", compresslevel=compress_level,
+                                 mtime=mtime, filename=filename)
+
+    def write(self, data: bytes) -> int:
+        return self.gzipfile.write(data)
+
+    def close(self) -> None:
+        self.gzipfile.close()
+        self.file.close()
+
+
+class Bz2Plugin:
+    """
+    Compresses received data using `bz2 <https://en.wikipedia.org/wiki/Bzip2>`_.
+
+    Accepted ``feed_options`` parameters:
+
+    - `bz2_compresslevel`
+
+    See :py:class:`bz2.BZ2File` for more info about parameters.
+    """
+
+    def __init__(self, file: BinaryIO, feed_options: Dict[str, Any]) -> None:
+        self.file = file
+        self.feed_options = feed_options
+        compress_level = self.feed_options.get("bz2_compresslevel", 9)
+        self.bz2file = BZ2File(filename=self.file, mode="wb", compresslevel=compress_level)
+
+    def write(self, data: bytes) -> int:
+        return self.bz2file.write(data)
+
+    def close(self) -> None:
+        self.bz2file.close()
+        self.file.close()
+
+
+class LZMAPlugin:
+    """
+    Compresses received data using `lzma <https://en.wikipedia.org/wiki/Lempel–Ziv–Markov_chain_algorithm>`_.
+
+    Accepted ``feed_options`` parameters:
+
+    - `lzma_format`
+    - `lzma_check`
+    - `lzma_preset`
+    - `lzma_filters`
+
+    .. note::
+        ``lzma_filters`` cannot be used in pypy version 7.3.1 and older.
+
+    See :py:class:`lzma.LZMAFile` for more info about parameters.
+    """
+
+    def __init__(self, file: BinaryIO, feed_options: Dict[str, Any]) -> None:
+        self.file = file
+        self.feed_options = feed_options
+
+        format = self.feed_options.get("lzma_format")
+        check = self.feed_options.get("lzma_check", -1)
+        preset = self.feed_options.get("lzma_preset")
+        filters = self.feed_options.get("lzma_filters")
+        self.lzmafile = LZMAFile(filename=self.file, mode="wb", format=format,
+                                 check=check, preset=preset, filters=filters)
+
+    def write(self, data: bytes) -> int:
+        return self.lzmafile.write(data)
+
+    def close(self) -> None:
+        self.lzmafile.close()
+        self.file.close()
+
+
+# io.IOBase is subclassed here, so that exporters can use the PostProcessingManager
+# instance as a file like writable object. This could be needed by some exporters
+# such as CsvItemExporter which wraps the feed storage with io.TextIOWrapper.
+class PostProcessingManager(IOBase):
+    """
+    This will manage and use declared plugins to process data in a
+    pipeline-ish way.
+    :param plugins: all the declared plugins for the feed
+    :type plugins: list
+    :param file: final target file where the processed data will be written
+    :type file: file like object
+    """
+
+    def __init__(self, plugins: List[Any], file: BinaryIO, feed_options: Dict[str, Any]) -> None:
+        self.plugins = self._load_plugins(plugins)
+        self.file = file
+        self.feed_options = feed_options
+        self.head_plugin = self._get_head_plugin()
+
+    def write(self, data: bytes) -> int:
+        """
+        Uses all the declared plugins to process data first, then writes
+        the processed data to target file.
+        :param data: data passed to be written to target file
+        :type data: bytes
+        :return: returns number of bytes written
+        :rtype: int
+        """
+        return self.head_plugin.write(data)
+
+    def tell(self) -> int:
+        return self.file.tell()
+
+    def close(self) -> None:
+        """
+        Close the target file along with all the plugins.
+        """
+        self.head_plugin.close()
+
+    def writable(self) -> bool:
+        return True
+
+    def _load_plugins(self, plugins: List[Any]) -> List[Any]:
+        plugins = [load_object(plugin) for plugin in plugins]
+        return plugins
+
+    def _get_head_plugin(self) -> Any:
+        prev = self.file
+        for plugin in self.plugins[::-1]:
+            prev = plugin(prev, self.feed_options)
+        return prev


Test patch:
diff --git a/tests/test_feedexport.py b/tests/test_feedexport.py
--- a/tests/test_feedexport.py
+++ b/tests/test_feedexport.py
@@ -1,5 +1,8 @@
+import bz2
 import csv
+import gzip
 import json
+import lzma
 import os
 import random
 import shutil
@@ -1469,6 +1472,499 @@ def test_extend_kwargs(self):
             self.assertEqual(row['expected'], data[feed_options['format']])
 
 
+class FeedPostProcessedExportsTest(FeedExportTestBase):
+    __test__ = True
+
+    items = [{'foo': 'bar'}]
+    expected = b'foo\r\nbar\r\n'
+
+    class MyPlugin1:
+        def __init__(self, file, feed_options):
+            self.file = file
+            self.feed_options = feed_options
+            self.char = self.feed_options.get('plugin1_char', b'')
+
+        def write(self, data):
+            written_count = self.file.write(data)
+            written_count += self.file.write(self.char)
+            return written_count
+
+        def close(self):
+            self.file.close()
+
+    def _named_tempfile(self, name):
+        return os.path.join(self.temp_dir, name)
+
+    @defer.inlineCallbacks
+    def run_and_export(self, spider_cls, settings):
+        """ Run spider with specified settings; return exported data with filename. """
+
+        FEEDS = settings.get('FEEDS') or {}
+        settings['FEEDS'] = {
+            printf_escape(path_to_url(file_path)): feed_options
+            for file_path, feed_options in FEEDS.items()
+        }
+
+        content = {}
+        try:
+            with MockServer() as s:
+                runner = CrawlerRunner(Settings(settings))
+                spider_cls.start_urls = [s.url('/')]
+                yield runner.crawl(spider_cls)
+
+            for file_path, feed_options in FEEDS.items():
+                if not os.path.exists(str(file_path)):
+                    continue
+
+                with open(str(file_path), 'rb') as f:
+                    content[str(file_path)] = f.read()
+
+        finally:
+            for file_path in FEEDS.keys():
+                if not os.path.exists(str(file_path)):
+                    continue
+
+                os.remove(str(file_path))
+
+        return content
+
+    def get_gzip_compressed(self, data, compresslevel=9, mtime=0, filename=''):
+        data_stream = BytesIO()
+        gzipf = gzip.GzipFile(fileobj=data_stream, filename=filename, mtime=mtime,
+                              compresslevel=compresslevel, mode="wb")
+        gzipf.write(data)
+        gzipf.close()
+        data_stream.seek(0)
+        return data_stream.read()
+
+    @defer.inlineCallbacks
+    def test_gzip_plugin(self):
+
+        filename = self._named_tempfile('gzip_file')
+
+        settings = {
+            'FEEDS': {
+                filename: {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.GzipPlugin'],
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+        try:
+            gzip.decompress(data[filename])
+        except OSError:
+            self.fail("Received invalid gzip data.")
+
+    @defer.inlineCallbacks
+    def test_gzip_plugin_compresslevel(self):
+
+        filename_to_compressed = {
+            self._named_tempfile('compresslevel_0'): self.get_gzip_compressed(self.expected, compresslevel=0),
+            self._named_tempfile('compresslevel_9'): self.get_gzip_compressed(self.expected, compresslevel=9),
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('compresslevel_0'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.GzipPlugin'],
+                    'gzip_compresslevel': 0,
+                    'gzip_mtime': 0,
+                    'gzip_filename': "",
+                },
+                self._named_tempfile('compresslevel_9'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.GzipPlugin'],
+                    'gzip_compresslevel': 9,
+                    'gzip_mtime': 0,
+                    'gzip_filename': "",
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, compressed in filename_to_compressed.items():
+            result = gzip.decompress(data[filename])
+            self.assertEqual(compressed, data[filename])
+            self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_gzip_plugin_mtime(self):
+        filename_to_compressed = {
+            self._named_tempfile('mtime_123'): self.get_gzip_compressed(self.expected, mtime=123),
+            self._named_tempfile('mtime_123456789'): self.get_gzip_compressed(self.expected, mtime=123456789),
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('mtime_123'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.GzipPlugin'],
+                    'gzip_mtime': 123,
+                    'gzip_filename': "",
+                },
+                self._named_tempfile('mtime_123456789'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.GzipPlugin'],
+                    'gzip_mtime': 123456789,
+                    'gzip_filename': "",
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, compressed in filename_to_compressed.items():
+            result = gzip.decompress(data[filename])
+            self.assertEqual(compressed, data[filename])
+            self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_gzip_plugin_filename(self):
+        filename_to_compressed = {
+            self._named_tempfile('filename_FILE1'): self.get_gzip_compressed(self.expected, filename="FILE1"),
+            self._named_tempfile('filename_FILE2'): self.get_gzip_compressed(self.expected, filename="FILE2"),
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('filename_FILE1'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.GzipPlugin'],
+                    'gzip_mtime': 0,
+                    'gzip_filename': "FILE1",
+                },
+                self._named_tempfile('filename_FILE2'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.GzipPlugin'],
+                    'gzip_mtime': 0,
+                    'gzip_filename': "FILE2",
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, compressed in filename_to_compressed.items():
+            result = gzip.decompress(data[filename])
+            self.assertEqual(compressed, data[filename])
+            self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_lzma_plugin(self):
+
+        filename = self._named_tempfile('lzma_file')
+
+        settings = {
+            'FEEDS': {
+                filename: {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+        try:
+            lzma.decompress(data[filename])
+        except lzma.LZMAError:
+            self.fail("Received invalid lzma data.")
+
+    @defer.inlineCallbacks
+    def test_lzma_plugin_format(self):
+
+        filename_to_compressed = {
+            self._named_tempfile('format_FORMAT_XZ'): lzma.compress(self.expected, format=lzma.FORMAT_XZ),
+            self._named_tempfile('format_FORMAT_ALONE'): lzma.compress(self.expected, format=lzma.FORMAT_ALONE),
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('format_FORMAT_XZ'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'lzma_format': lzma.FORMAT_XZ,
+                },
+                self._named_tempfile('format_FORMAT_ALONE'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'lzma_format': lzma.FORMAT_ALONE,
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, compressed in filename_to_compressed.items():
+            result = lzma.decompress(data[filename])
+            self.assertEqual(compressed, data[filename])
+            self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_lzma_plugin_check(self):
+
+        filename_to_compressed = {
+            self._named_tempfile('check_CHECK_NONE'): lzma.compress(self.expected, check=lzma.CHECK_NONE),
+            self._named_tempfile('check_CHECK_CRC256'): lzma.compress(self.expected, check=lzma.CHECK_SHA256),
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('check_CHECK_NONE'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'lzma_check': lzma.CHECK_NONE,
+                },
+                self._named_tempfile('check_CHECK_CRC256'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'lzma_check': lzma.CHECK_SHA256,
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, compressed in filename_to_compressed.items():
+            result = lzma.decompress(data[filename])
+            self.assertEqual(compressed, data[filename])
+            self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_lzma_plugin_preset(self):
+
+        filename_to_compressed = {
+            self._named_tempfile('preset_PRESET_0'): lzma.compress(self.expected, preset=0),
+            self._named_tempfile('preset_PRESET_9'): lzma.compress(self.expected, preset=9),
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('preset_PRESET_0'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'lzma_preset': 0,
+                },
+                self._named_tempfile('preset_PRESET_9'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'lzma_preset': 9,
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, compressed in filename_to_compressed.items():
+            result = lzma.decompress(data[filename])
+            self.assertEqual(compressed, data[filename])
+            self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_lzma_plugin_filters(self):
+        import sys
+        if "PyPy" in sys.version:
+            # https://foss.heptapod.net/pypy/pypy/-/issues/3527
+            raise unittest.SkipTest("lzma filters doesn't work in PyPy")
+
+        filters = [{'id': lzma.FILTER_LZMA2}]
+        compressed = lzma.compress(self.expected, filters=filters)
+        filename = self._named_tempfile('filters')
+
+        settings = {
+            'FEEDS': {
+                filename: {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'lzma_filters': filters,
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+        self.assertEqual(compressed, data[filename])
+        result = lzma.decompress(data[filename])
+        self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_bz2_plugin(self):
+
+        filename = self._named_tempfile('bz2_file')
+
+        settings = {
+            'FEEDS': {
+                filename: {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.Bz2Plugin'],
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+        try:
+            bz2.decompress(data[filename])
+        except OSError:
+            self.fail("Received invalid bz2 data.")
+
+    @defer.inlineCallbacks
+    def test_bz2_plugin_compresslevel(self):
+
+        filename_to_compressed = {
+            self._named_tempfile('compresslevel_1'): bz2.compress(self.expected, compresslevel=1),
+            self._named_tempfile('compresslevel_9'): bz2.compress(self.expected, compresslevel=9),
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('compresslevel_1'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.Bz2Plugin'],
+                    'bz2_compresslevel': 1,
+                },
+                self._named_tempfile('compresslevel_9'): {
+                    'format': 'csv',
+                    'postprocessing': ['scrapy.extensions.postprocessing.Bz2Plugin'],
+                    'bz2_compresslevel': 9,
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, compressed in filename_to_compressed.items():
+            result = bz2.decompress(data[filename])
+            self.assertEqual(compressed, data[filename])
+            self.assertEqual(self.expected, result)
+
+    @defer.inlineCallbacks
+    def test_custom_plugin(self):
+        filename = self._named_tempfile('csv_file')
+
+        settings = {
+            'FEEDS': {
+                filename: {
+                    'format': 'csv',
+                    'postprocessing': [self.MyPlugin1],
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+        self.assertEqual(self.expected, data[filename])
+
+    @defer.inlineCallbacks
+    def test_custom_plugin_with_parameter(self):
+
+        expected = b'foo\r\n\nbar\r\n\n'
+        filename = self._named_tempfile('newline')
+
+        settings = {
+            'FEEDS': {
+                filename: {
+                    'format': 'csv',
+                    'postprocessing': [self.MyPlugin1],
+                    'plugin1_char': b'\n'
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+        self.assertEqual(expected, data[filename])
+
+    @defer.inlineCallbacks
+    def test_custom_plugin_with_compression(self):
+
+        expected = b'foo\r\n\nbar\r\n\n'
+
+        filename_to_decompressor = {
+            self._named_tempfile('bz2'): bz2.decompress,
+            self._named_tempfile('lzma'): lzma.decompress,
+            self._named_tempfile('gzip'): gzip.decompress,
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('bz2'): {
+                    'format': 'csv',
+                    'postprocessing': [self.MyPlugin1, 'scrapy.extensions.postprocessing.Bz2Plugin'],
+                    'plugin1_char': b'\n',
+                },
+                self._named_tempfile('lzma'): {
+                    'format': 'csv',
+                    'postprocessing': [self.MyPlugin1, 'scrapy.extensions.postprocessing.LZMAPlugin'],
+                    'plugin1_char': b'\n',
+                },
+                self._named_tempfile('gzip'): {
+                    'format': 'csv',
+                    'postprocessing': [self.MyPlugin1, 'scrapy.extensions.postprocessing.GzipPlugin'],
+                    'plugin1_char': b'\n',
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, decompressor in filename_to_decompressor.items():
+            result = decompressor(data[filename])
+            self.assertEqual(expected, result)
+
+    @defer.inlineCallbacks
+    def test_exports_compatibility_with_postproc(self):
+        import marshal
+        import pickle
+        filename_to_expected = {
+            self._named_tempfile('csv'): b'foo\r\nbar\r\n',
+            self._named_tempfile('json'): b'[\n{"foo": "bar"}\n]',
+            self._named_tempfile('jsonlines'): b'{"foo": "bar"}\n',
+            self._named_tempfile('xml'): b'<?xml version="1.0" encoding="utf-8"?>\n'
+                                         b'<items>\n<item><foo>bar</foo></item>\n</items>',
+        }
+
+        settings = {
+            'FEEDS': {
+                self._named_tempfile('csv'): {
+                    'format': 'csv',
+                    'postprocessing': [self.MyPlugin1],
+                    # empty plugin to activate postprocessing.PostProcessingManager
+                },
+                self._named_tempfile('json'): {
+                    'format': 'json',
+                    'postprocessing': [self.MyPlugin1],
+                },
+                self._named_tempfile('jsonlines'): {
+                    'format': 'jsonlines',
+                    'postprocessing': [self.MyPlugin1],
+                },
+                self._named_tempfile('xml'): {
+                    'format': 'xml',
+                    'postprocessing': [self.MyPlugin1],
+                },
+                self._named_tempfile('marshal'): {
+                    'format': 'marshal',
+                    'postprocessing': [self.MyPlugin1],
+                },
+                self._named_tempfile('pickle'): {
+                    'format': 'pickle',
+                    'postprocessing': [self.MyPlugin1],
+                },
+            },
+        }
+
+        data = yield self.exported_data(self.items, settings)
+
+        for filename, result in data.items():
+            if 'pickle' in filename:
+                expected, result = self.items[0], pickle.loads(result)
+            elif 'marshal' in filename:
+                expected, result = self.items[0], marshal.loads(result)
+            else:
+                expected = filename_to_expected[filename]
+            self.assertEqual(expected, result)
+
+
 class BatchDeliveriesTest(FeedExportTestBase):
     __test__ = True
     _file_mark = '_%(batch_time)s_#%(batch_id)02d_'