Issue 1:
Hi,

I am using Scrapy version 1.1.2. It is somenthing weird, because depending on the Rules configuration, the callback function is called or not...

My rules:

rules = (
Rule(LinkExtractor(allow=('.*balsamo'))), # get category page
Rule(LinkExtractor(allow=(".*\/\d+.*"),restrict_xpaths=('//*[@id="center_column"]/div[3]')), callback='category'), # get product pages links in the category page
)

If I remove the restrict_xpaths filter, the callback function is called, but when I use restrict_xpaths, the callback function is never called, even when the ouput shows the right crawled pages as "crawled". This is the output resulting from the above rules.

2016-09-28 19:00:57 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/robots.txt (referer: None)
2016-09-28 19:00:57 [scrapy] DEBUG: Crawled (200) GET http://www.example.es (referer: None)
2016-09-28 19:00:58 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/75-balsamo (referer: http://www.example.es)
2016-09-28 19:00:58 [scrapy] DEBUG: Filtered duplicate request: GET http://www.example.es/75-balsamo - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/233-relajante-balsamo-aromatico-dream-6-g-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/balsamo/459-balsamo-de-tigre-rojo-21-ml-varios.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/235-refrescante-bio-balsamo-hierbas-classic-6-g-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/varios/234-tranquilizador-balsamo-aromatico-balance-ha-tha.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] DEBUG: Crawled (200) GET http://www.example.es/balsamo/460-balsamo-de-tigre-blanco-21-ml-varios.html (referer: http://www.example.es/75-balsamo)
2016-09-28 19:01:01 [scrapy] INFO: Closing spider (finished)

Issue 2:
I totally dig this idea, but I also think this needs to be well thought-out or it could spell more trouble than it solves and be hard to deprecate and fix later.

My thoughts on the current proposal:

[addons] in scrapy.cfg would not provide any way to pass configuration settings. So we still need to modify settings.py for, say, the database connection parameters.

That doesn't look very portable.

Also, if two different addons expose or use the same settings, it gets ugly.

Say we have different addons, one with a MongoPipeline and one with a MongoQueue, which by chance both use the MONGO_COLLECTION setting name.

This is obviously already a problem, but I think with add-ons, people would more expect a "plug-and-play" system which "just works", and be less careful with checking individual addon-components for name clashes or dependency issues.

Unless addons should be expected to be configured prior to use, or only as a bundle for hardcoded configuration settings, I think we need another layer deep to have addons and their settings in the same place.

Maybe we can use a [addon_(name)] (or simply [(addon_name)]) ini setting which can then take multiple parameters.

These settings should be namespaced to the addon.

Example:

[httpcache] # (looking in python path, no settings)
#enabled = True # (does ini-style require a parameter to recognize the section?)
[mongodb_pipeline]
path = /path/to/mongodb_pipeline.py # (explicit path)
host = 'localhost'
port = 27017

These settings would then expand to (addon_name)_(setting), e.g. mongodb_pipeline_port or MONGODB_PIPELINE_PORT to prevent namespace clashes between addons.

(Would this copy settings on scrapyd deployment?)

Alternatively, maybe all the addon defining and configuration could be in settings.py, but I think it should be in a single place.