Repository: scrapy/scrapy

Issue:
Document LOG_UNSERIALIZABLE_REQUESTS setting
It's implemented in [scrapy/core/scheduler.py#L31](https://github.com/scrapy/scrapy/blob/master/scrapy/core/scheduler.py#L31) and not mentioned anywhere in docs. I think it's really useful, could be mentioned at least in http://scrapy.readthedocs.org/en/latest/topics/jobs.html#request-serialization.

Since it isn't documented we could take the chance and rename it as well, this kind of settings are usually named `<something>_DEBUG`.



Discussion:
Hi !

How about adding this comment [here](http://scrapy.readthedocs.org/en/latest/topics/jobs.html#request-serialization) :
`If you wish to log the requests that couldn't be serialized, you can enable the`**`LOG_UNSERIALIZABLE_REQUESTS`** `by setting it to True in the project's settings page. It is` **`False`**`by default.`

How about renaming this setting to `UNSERIALIZABLE_REQUESTS_DEBUG` in the spirit of settings like `AUTOTHROTTLE_DEBUG`,`COOKIES_DEBUG` ?

@darshanime both suggestions sound good to me :) I'd add that we should mention this setting in the builtin settings list here: http://scrapy.readthedocs.org/en/latest/topics/settings.html#built-in-settings-reference as well.

Could you help us with a pull request with those changes? I'll gladly review it when you submit it.

Sure. 

Isn't this option a `SCHEDULER_DEBUG`?

I would also change it to log once and save stats. Similar to dedupe filter.


Resolving patch:
diff --git a/scrapy/core/scheduler.py b/scrapy/core/scheduler.py
--- a/scrapy/core/scheduler.py
+++ b/scrapy/core/scheduler.py
@@ -30,7 +30,7 @@ def from_crawler(cls, crawler):
         pqclass = load_object(settings['SCHEDULER_PRIORITY_QUEUE'])
         dqclass = load_object(settings['SCHEDULER_DISK_QUEUE'])
         mqclass = load_object(settings['SCHEDULER_MEMORY_QUEUE'])
-        logunser = settings.getbool('LOG_UNSERIALIZABLE_REQUESTS')
+        logunser = settings.getbool('LOG_UNSERIALIZABLE_REQUESTS', settings.getbool('SCHEDULER_DEBUG'))
         return cls(dupefilter, jobdir=job_dir(settings), logunser=logunser,
                    stats=crawler.stats, pqclass=pqclass, dqclass=dqclass, mqclass=mqclass)
 
@@ -84,11 +84,16 @@ def _dqpush(self, request):
         try:
             reqd = request_to_dict(request, self.spider)
             self.dqs.push(reqd, -request.priority)
-        except ValueError as e: # non serializable request
+        except ValueError as e:  # non serializable request
             if self.logunser:
-                logger.error("Unable to serialize request: %(request)s - reason: %(reason)s",
-                             {'request': request, 'reason': e},
+                msg = ("Unable to serialize request: %(request)s - reason:"
+                       " %(reason)s - no more unserializable requests will be"
+                       " logged (stats being collected)")
+                logger.error(msg, {'request': request, 'reason': e},
                              exc_info=True, extra={'spider': self.spider})
+                self.logunser = False
+            self.stats.inc_value('scheduler/unserializable',
+                                 spider=self.spider)
             return
         else:
             return True
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -191,7 +191,7 @@
 LOG_LEVEL = 'DEBUG'
 LOG_FILE = None
 
-LOG_UNSERIALIZABLE_REQUESTS = False
+SCHEDULER_DEBUG = False
 
 LOGSTATS_INTERVAL = 60.0
 
diff --git a/scrapy/settings/deprecated.py b/scrapy/settings/deprecated.py
--- a/scrapy/settings/deprecated.py
+++ b/scrapy/settings/deprecated.py
@@ -14,6 +14,7 @@
     ('AUTOTHROTTLE_MAX_CONCURRENCY', 'use CONCURRENT_REQUESTS_PER_DOMAIN instead'),
     ('AUTOTHROTTLE_MAX_CONCURRENCY', 'use CONCURRENT_REQUESTS_PER_DOMAIN instead'),
     ('REDIRECT_MAX_METAREFRESH_DELAY', 'use METAREFRESH_MAXDELAY instead'),
+    ('LOG_UNSERIALIZABLE_REQUESTS', 'use SCHEDULER_DEBUG instead'),
 ]
 
 


Test patch: