Repository: scrapy/scrapy
Pull request: option to output data in scrapy parse
- adds an option `-o` or `--output` in `scrapy parse` command to save
scrapped data to a file

- [x] fixes #4317
- [x] tests added/passed
- [x] add documentation

Problem statement:
Having an option to define an Items output file in the `scrapy parse` commnad line
I use `scrapy parse` command to check the Items output for given URL, and I use it a lot.

Sometimes, I need to latter explore the Items extracted by `scrapy parse`, e.g . to check that something I want to fix was fixed for that specific URL without having to touch the source code.

### Proposed solution:

Add a new parameter `-o` or `--output` in the parse command line. Allowing the client to set a desired output file for the items. 

I could also `|` the result of `scrapy parse`, filter out want I want and create a `json` file out of it, but would be much easier if I could simple write `scrapy parse {url} --callback=fixed_function -o items.json`

Hints from issue discussion:
Shall I work on this issue?
@adityaa30 Please, go ahead, no need to ask for permission :slightly_smiling_face: 
Okay sure!
Hi, I m working on this issue and made some changes in scrapy/commands/parse.py but running "scrapy parse" in command line not reflecting changes. Any suggestions?
@Gallaecio how to run scrapy from cloned repository in my local computer? I think "scrapy parse" command running scrapy package installed through "pip install scrapy".
@AKSHAYSHARMAJS you should be able to install Scrapy in "development mode" by executing `pip install -e .`
@elacuesta Thanks!

@raphapassini @Gallaecio I have sent a pr #4377. I have made it a work in progress as I have a few questions:
1. If the user uses `--output` command then it may be not required to show scrapped data in cmd. Shall I add a check such that when `--output` is specified then data won't print in cmd?
2. Shall i add another option to save the list of requests made to another file?
@adityaa30 I think it is better to keep the implementation simple, so I personally would not implement 1 or 2 unless people show up asking for it and explaining their use cases.
@Gallaecio Okay!

Commit message: docs(commands): mention output flag for parse

Patch to review:
diff --git a/docs/topics/commands.rst b/docs/topics/commands.rst
index a0dcba90d36..4fce51abc21 100644
--- a/docs/topics/commands.rst
+++ b/docs/topics/commands.rst
@@ -468,7 +468,7 @@ Supported options:
 * ``--callback`` or ``-c``: spider method to use as callback for parsing the
   response
 
-* ``--meta`` or ``-m``: additional request meta that will be passed to the callback 
+* ``--meta`` or ``-m``: additional request meta that will be passed to the callback
   request. This must be a valid json string. Example: --meta='{"foo" : "bar"}'
 
 * ``--cbkwargs``: additional keyword arguments that will be passed to the callback.
@@ -491,6 +491,8 @@ Supported options:
 
 * ``--verbose`` or ``-v``: display information for each depth level
 
+* ``--output`` or ``-o``: dump scraped items to a file
+
 .. skip: start
 
 Usage example::
diff --git a/scrapy/commands/parse.py b/scrapy/commands/parse.py
index 3ef8ddcb37a..05c0f514052 100644
--- a/scrapy/commands/parse.py
+++ b/scrapy/commands/parse.py
@@ -7,7 +7,7 @@
 from scrapy.http import Request
 from scrapy.item import BaseItem
 from scrapy.utils import display
-from scrapy.utils.conf import arglist_to_dict
+from scrapy.utils.conf import arglist_to_dict, feed_process_params_from_cli
 from scrapy.utils.spider import iterate_spider_output, spidercls_for_request
 from scrapy.exceptions import UsageError
 
@@ -15,7 +15,6 @@
 
 
 class Command(ScrapyCommand):
-
     requires_project = True
 
     spider = None
@@ -56,6 +55,10 @@ def add_options(self, parser):
             help="maximum depth for parsing requests [default: %default]")
         parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
             help="print each depth level one by one")
+        parser.add_option("-o", "--output", metavar="FILE", action="append",
+            help="dump scraped items into FILE (use - for stdout)")
+        parser.add_option("-t", "--output-format", metavar="FORMAT",
+            help="format to use for dumping items with -o")
 
     @property
     def max_level(self):
@@ -206,7 +209,9 @@ def callback(response, **cb_kwargs):
                     req.meta['_depth'] = depth + 1
                     req.meta['_callback'] = req.callback
                     req.callback = callback
-                return requests
+                return items + requests
+            else:
+                return items  # dumps the remaining output to a FILE
 
         # update request meta if any extra meta was passed through the --meta/-m opts.
         if opts.meta:
@@ -233,6 +238,9 @@ def process_spider_arguments(self, opts):
             opts.spargs = arglist_to_dict(opts.spargs)
         except ValueError:
             raise UsageError("Invalid -a value, use -a NAME=VALUE", print_help=False)
+        if opts.output:
+            feeds = feed_process_params_from_cli(self.settings, opts.output, opts.output_format)
+            self.settings.set('FEEDS', feeds, priority='cmdline')
 
     def process_request_meta(self, opts):
         if opts.meta:
diff --git a/tests/test_command_parse.py b/tests/test_command_parse.py
index 85a24d0bc23..9cd40b19491 100644
--- a/tests/test_command_parse.py
+++ b/tests/test_command_parse.py
@@ -1,5 +1,5 @@
 import os
-from os.path import join, abspath
+from os.path import join, abspath, isfile, exists
 from twisted.internet import defer
 from scrapy.utils.testsite import SiteTest
 from scrapy.utils.testproc import ProcessTest
@@ -218,3 +218,24 @@ def test_crawlspider_no_matching_rule(self):
         )
         self.assertRegex(_textmode(out), r"""# Scraped Items  -+\n\[\]""")
         self.assertIn("""Cannot find a rule that matches""", _textmode(stderr))
+
+    @defer.inlineCallbacks
+    def test_output_flag(self):
+        """Checks if a file was created successfully having
+        correct format containing correct data in it.
+        """
+        file_name = 'data.json'
+        file_path = join(self.proj_path, file_name)
+        yield self.execute([
+            '--spider', self.spider_name,
+            '-c', 'parse',
+            '-o', file_name,
+            self.url('/html')
+        ])
+
+        self.assertTrue(exists(file_path))
+        self.assertTrue(isfile(file_path))
+
+        content = '[\n{},\n{"foo": "bar"}\n]'
+        with open(file_path, 'r') as f:
+            self.assertEqual(f.read(), content)


Code review comments:
diff_hunk: @@ -206,7 +209,9 @@ def callback(response, **cb_kwargs):
                     req.meta['_depth'] = depth + 1
                     req.meta['_callback'] = req.callback
                     req.callback = callback
-                return requests
+                return items + requests
+            else:
+                return items  # dumps the remaining output to a FILE
path: scrapy/commands/parse.py
text: @user1:
Maybe we should only do this when the new option is used, to minimize behavior changes?

@author:
Yes, that will be better. I will do that now