Repository: scrapy/scrapy
Pull request #1586 (2015-11-06 15:17:04)

[MRG+1] Backwards compatible per key priorities
This is a partial reversal of #1149 that provides per-key priorities in a backwards compatible fashion.

The backwards incompatible API change in #1149 was that when writing to dictionary settings, their contents were updated, not completely overwritten. This PR brings back the old behaviour of completely discarding the previous contents of the dictionary. If the dictionary setting written to is a `BaseSettings` instance, it is not overwritten but cleared and then updated (resulting in the same contents, but conserving per-key priorities).

The price for this is:
  - we cannot get rid of the `_BASE` settings
  - it will again be much harder to disable single components from the command line or from a spider's `custom_settings` (because disabling a single component requires writing down the complete dictionary without that component). @kmike had the idea that we could add some helper magic for updating (not overwriting) dictionaries, e.g. allow people to do `scrapy ... -s update:EXTENSIONS={"blah": null}`, or `custom_settings: { "update:EXTENSIONS": {"blah": None}}`. I'll implement that in a follow-up PR to discuss it and see if it makes unforeseen trouble.

As a bonus, I added backwards compatibility to the `build_component_list()` utility function, albeit it not being public. @rolando maybe you could check whether it is indeed backwards compatible with how you use it? Also take a look at the now-public [`BaseSettings.getcomposite()` method](https://github.com/jdemaeyer/scrapy/blob/fix/backwards-compatible-per-key-priorities/scrapy/settings/__init__.py#L197). It helps in separating a utility function that's meant to convert a dictionary into a sorted list from how that dictionary is created.

kmike: @jdemaeyer there is a failure in docs check, could you please fix it?
codecov-io: ## [Current coverage][1] is `82.88%`
> Merging **#1586** into **master** will not affect coverage as of [`d82f286`][3]



[1]: https://codecov.io/github/scrapy/scrapy?ref=d82f286e91d7c23ee640e2b647373e0add76dd9b
[2]: https://codecov.io/github/scrapy/scrapy/features/suggestions?ref=d82f286e91d7c23ee640e2b647373e0add76dd9b
[3]: https://codecov.io/github/scrapy/scrapy/commit/d82f286e91d7c23ee640e2b647373e0add76dd9b
[4]: https://codecov.io/github/scrapy/scrapy/compare/57f87b95d4d705f8afdd8fb9f7551033a7d88ee2...d82f286e91d7c23ee640e2b647373e0add76dd9b

> Powered by [Codecov](https://codecov.io). Updated on successful CI builds.
jdemaeyer: ugh damnit, thanks, fixed.
rolando: @jdemaeyer yes, our code works as expected on this PR. Thanks.
kmike: I'm checking https://github.com/scrapy/scrapy/compare/4a9f76ebd3242b2f91e238c26ff38874e9ee1960...jdemaeyer:fix/backwards-compatible-per-key-priorities?name=fix%2Fbackwards-compatible-per-key-priorities&short_path=7d64b15#diff-7d64b15bbaac9444c9da295924bf42bf:

* How to disable download handlers? Is this feature supported? It looks like relevant code and docs are deleted.
* Are lists in ITEM_PIPELINES supported again after this PR? 
* getcomposite public function: I think it is better to extract "If only one argument is given, a second argument with `_BASE` appended will automatically be added" part into another function.
* I think getpriority can be written a bit simpler, one option is:
```py
if name not in self:
    return None
return self.attributes[name].priority
```

jdemaeyer: > How to disable download handlers? Is this feature supported? It looks like relevant code and docs are deleted.

Yep, it's supported and got lost while I reverted the docs to "with _BASE", will put it back in. They're disabled by setting their class to `None` in the `DOWNLOAD_HANDLERS` dict (much like all other components can be disabled).

> Are lists in ITEM_PIPELINES supported again after this PR?

Nope. Though with this PR we could support it again (although it has been deprecated for over two years).

> getcomposite public function: I think it is better to extract "If only one argument is given, a second argument with _BASE appended will automatically be added" part into another function.

Hm, calling `getcomposite` with only one argument is useless then. But I'm fine with adding a new method if you think it's more consistent.

> I think getpriority can be written a bit simpler, one option is:

Looks good, we should update `__getitem__()` then as well.
kmike: > Hm, calling getcomposite with only one argument is useless then. But I'm fine with adding a new method if you think it's more consistent.

Do we have use cases for a public `.getcomposite` method with 1+ arguments, i.e. is it used in scrapy? If not then we can drop it for now and implement only `_BASE` support. Do you need it for addons PR?
kmike: > Nope. Though with this PR we could support it again (although it has been deprecated for over two years).

Ok, I'm fine with dropping support for ITEM_PIPELINE as a list, but we should remember to add it to the release notes. It is easy to miss because this change was not in a separate PR.
jdemaeyer: All done. Instead of `getwithbase()` (`getcomposite()` before) we could also go abck
kmike: +1 to merge. //cc @curita @dangra
curita: There's a remaining `_getcomposite()` in `scrapy/commands/runspider.py`.
curita: Not sure what's the long term plan for *_BASE settings now, but if we plan to keep them for good we could implement `DEFAULT_REQUEST_HEADERS_BASE` as Paul suggested.
curita: Pointed out some minor issues, but after reviewing these changes I still kind of prefer the approach in #1149. Here are some of my (truly subjective) thoughts about it:

* It has better design. *_BASE settings are redundant having per-key priorities.
* It has better usability. We usually want to update a set of values in those dict settings (not just *_BASE, I often want to update project-defined components with spider ones). I'd rather prefer updating the dictionaries by default and having a helper to occasionally override them.
* We could improve its backward compatibility a little, by defining the old *_BASE settings with copies of the new values, but without using them.

Well, having explained myself, I don't think it's worth of a discussion really, this PR seems the way to go. Probably we should try to follow any backward compatible option if there is one, and the magic helper for updating dictionaries you were talking about is just what I'm looking for. So +1 on my side once the nitpicks I commented before get fixed :+1:
jdemaeyer: Damnit! :/ thanks alot for the catches/feedback everyone! I've implemented Julia's feedback. There is now no deep-copying at all in `getwithbase()`: If we decide to allow Python objects (instead of paths) in the component settings (6f91110, part of #1272), users will expect that these exact objects are used in Scrapy, not deep copies of them.

I'll squash commits once you've looked at the changes.

Patch:
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index 08d8f3edf..cc0254d29 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -23,20 +23,22 @@ Here's an example::
         'myproject.middlewares.CustomDownloaderMiddleware': 543,
     }
 
-The specified :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
-default one (i.e. it does not overwrite it) and then sorted by order to get the
-final sorted list of enabled middlewares: the first middleware is the one
-closer to the engine and the last is the one closer to the downloader.
-
-To decide which order to assign to your middleware see the default
-:setting:`DOWNLOADER_MIDDLEWARES` setting and pick a value according to
+The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
+:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant
+to be overridden) and then sorted by order to get the final sorted list of
+enabled middlewares: the first middleware is the one closer to the engine and
+the last is the one closer to the downloader.
+
+To decide which order to assign to your middleware see the
+:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting and pick a value according to
 where you want to insert the middleware. The order does matter because each
 middleware performs a different action and your middleware could depend on some
 previous (or subsequent) middleware being applied.
 
-If you want to disable a built-in middleware you must define it in your
-project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign ``None`` as its
-value. For example, if you want to disable the user-agent middleware::
+If you want to disable a built-in middleware (the ones defined in
+:setting:`DOWNLOADER_MIDDLEWARES_BASE` and enabled by default) you must define it
+in your project's :setting:`DOWNLOADER_MIDDLEWARES` setting and assign `None`
+as its value.  For example, if you want to disable the user-agent middleware::
 
     DOWNLOADER_MIDDLEWARES = {
         'myproject.middlewares.CustomDownloaderMiddleware': 543,
@@ -162,7 +164,7 @@ middleware, see the :ref:`downloader middleware usage guide
 <topics-downloader-middleware>`.
 
 For a list of the components enabled by default (and their orders) see the
-:setting:`DOWNLOADER_MIDDLEWARES` setting.
+:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting.
 
 .. _cookies-mw:
 
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index 11c0aadb6..847353868 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -42,13 +42,14 @@ by a string: the full Python path to the extension's class name. For example::
 
 As you can see, the :setting:`EXTENSIONS` setting is a dict where the keys are
 the extension paths, and their values are the orders, which define the
-extension *loading* order. The specified :setting:`EXTENSIONS` setting is merged
-with the default one (i.e. it does not overwrite it) and then sorted by order
-to get the final sorted list of enabled extensions.
+extension *loading* order. The :setting:`EXTENSIONS` setting is merged with the
+:setting:`EXTENSIONS_BASE` setting defined in Scrapy (and not meant to be
+overridden) and then sorted by order to get the final sorted list of enabled
+extensions.
 
 As extensions typically do not depend on each other, their loading order is
-irrelevant in most cases. This is why the default :setting:`EXTENSIONS` setting
-defines all extensions with the same order (``500``). However, this feature can
+irrelevant in most cases. This is why the :setting:`EXTENSIONS_BASE` setting
+defines all extensions with the same order (``0``). However, this feature can
 be exploited if you need to add an extension which depends on other extensions
 already loaded.
 
@@ -63,7 +64,7 @@ Disabling an extension
 ======================
 
 In order to disable an extension that comes enabled by default (ie. those
-included in the default :setting:`EXTENSIONS` setting) you must set its order to
+included in the :setting:`EXTENSIONS_BASE` setting) you must set its order to
 ``None``. For example::
 
     EXTENSIONS = {
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index d8b8da166..03c6fb3fb 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -265,6 +265,16 @@ Whether to export empty feeds (ie. feeds with no items).
 FEED_STORAGES
 -------------
 
+Default:: ``{}``
+
+A dict containing additional feed storage backends supported by your project.
+The keys are URI schemes and the values are paths to storage classes.
+
+.. setting:: FEED_STORAGES_BASE
+
+FEED_STORAGES_BASE
+------------------
+
 Default::
 
     {
@@ -275,19 +285,30 @@ Default::
         'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
     }
 
-A dict containing all feed storage backends supported by your project. The keys
-are URI schemes and the values are paths to storage classes.
+A dict containing the built-in feed storage backends supported by Scrapy. You
+can disable any of these backends by assigning ``None`` to their URI scheme in
+:setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend
+(without replacement), place this in your ``settings.py``::
 
-When you set :setting:`FEED_STORAGES` manually, e.g. in your project's settings
-module, it will be merged with the default, not overwrite it. If you want to
-disable any of the default feed storage backends, you must assign ``None`` as
-their value.
+    FEED_STORAGES = {
+        'ftp': None,
+    }
 
 .. setting:: FEED_EXPORTERS
 
 FEED_EXPORTERS
 --------------
 
+Default:: ``{}``
+
+A dict containing additional exporters supported by your project. The keys are
+serialization formats and the values are paths to :ref:`Item exporter
+<topics-exporters>` classes.
+
+.. setting:: FEED_EXPORTERS_BASE
+
+FEED_EXPORTERS_BASE
+-------------------
 Default::
 
     {
@@ -300,14 +321,14 @@ Default::
         'pickle': 'scrapy.exporters.PickleItemExporter',
     }
 
-A dict containing all feed exporters supported by your project. The keys are
-URI schemes and the values are paths to :ref:`Item exporter <topics-exporters>`
-classes.
+A dict containing the built-in feed exporters supported by Scrapy. You can
+disable any of these exporters by assigning ``None`` to their serialization
+format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
+(without replacement), place this in your ``settings.py``::
 
-When you set :setting:`FEED_EXPORTERS` manually, e.g. in your project's settings
-module, it will be merged with the default, not overwrite it. If you want to
-disable any of the default feed exporters, you must assign ``None`` as their
-value.
+    FEED_EXPORTERS = {
+        'csv': None,
+    }
 
 .. _URI: http://en.wikipedia.org/wiki/Uniform_Resource_Identifier
 .. _Amazon S3: http://aws.amazon.com/s3/
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index 8908fae7e..aa0417e1a 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -269,11 +269,6 @@ Default::
 The default headers used for Scrapy HTTP Requests. They're populated in the
 :class:`~scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware`.
 
-When you set :setting:`DEFAULT_REQUEST_HEADERS` manually, e.g. in your
-project's settings module, it will be merged with the default, not overwrite it.
-If you want to disable any of the default request headers (and not replace them)
-you must assign ``None`` as their value.
-
 .. setting:: DEPTH_LIMIT
 
 DEPTH_LIMIT
@@ -355,6 +350,16 @@ The downloader to use for crawling.
 DOWNLOADER_MIDDLEWARES
 ----------------------
 
+Default:: ``{}``
+
+A dict containing the downloader middlewares enabled in your project, and their
+orders. For more info see :ref:`topics-downloader-middleware-setting`.
+
+.. setting:: DOWNLOADER_MIDDLEWARES_BASE
+
+DOWNLOADER_MIDDLEWARES_BASE
+---------------------------
+
 Default::
 
     {
@@ -375,16 +380,11 @@ Default::
         'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
     }
 
-A dict containing the downloader middlewares enabled in your project, and their
-orders. Low orders are closer to the engine, high orders are closer to the
-downloader.
-
-When you set :setting:`DOWNLOADER_MIDDLEWARES` manually, e.g. in your project's
-settings module, it will be merged with the default, not overwrite it. If you
-want to disable any of the default downloader middlewares you must assign
-``None`` as their value.
-
-For more info see :ref:`topics-downloader-middleware-setting`.
+A dict containing the downloader middlewares enabled by default in Scrapy. Low
+orders are closer to the engine, high orders are closer to the downloader. You
+should never modify this setting in your project, modify
+:setting:`DOWNLOADER_MIDDLEWARES` instead.  For more info see
+:ref:`topics-downloader-middleware-setting`.
 
 .. setting:: DOWNLOADER_STATS
 
@@ -425,6 +425,16 @@ spider attribute.
 DOWNLOAD_HANDLERS
 -----------------
 
+Default: ``{}``
+
+A dict containing the request downloader handlers enabled in your project.
+See :setting:`DOWNLOAD_HANDLERS_BASE` for example format.
+
+.. setting:: DOWNLOAD_HANDLERS_BASE
+
+DOWNLOAD_HANDLERS_BASE
+----------------------
+
 Default::
 
     {
@@ -436,15 +446,16 @@ Default::
     }
 
 
-A dict containing the request downloader handlers enabled in your project.
+A dict containing the request download handlers enabled by default in Scrapy.
+You should never modify this setting in your project, modify
+:setting:`DOWNLOAD_HANDLERS` instead.
 
-When you set :setting:`DOWNLOAD_HANDLERS` manually, e.g. in your project's
-settings module, it will be merged with the default, not overwrite it. If you
-want to disable any of the default download handlers you must assign ``None``
-as their value. For example, if you want to disable the file download handler::
+You can disable any of these download handlers by assigning ``None`` to their
+URI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP
+handler (without replacement), place this in your ``settings.py``::
 
     DOWNLOAD_HANDLERS = {
-        'file': None,
+        'ftp': None,
     }
 
 .. setting:: DOWNLOAD_TIMEOUT
@@ -544,6 +555,15 @@ to ``vi`` (on Unix systems) or the IDLE editor (on Windows).
 EXTENSIONS
 ----------
 
+Default:: ``{}``
+
+A dict containing the extensions enabled in your project, and their orders.
+
+.. setting:: EXTENSIONS_BASE
+
+EXTENSIONS_BASE
+---------------
+
 Default::
 
     {
@@ -558,15 +578,10 @@ Default::
         'scrapy.extensions.throttle.AutoThrottle': 0,
     }
 
-A dict containing the extensions enabled in your project, and their orders. By
-default, this setting contains all stable built-in extensions. Keep in mind that
+A dict containing the extensions available by default in Scrapy, and their
+orders. This setting contains all stable built-in extensions. Keep in mind that
 some of them need to be enabled through a setting.
 
-When you set :setting:`EXTENSIONS` manually, e.g. in your project's settings
-module, it will be merged with the default, not overwrite it. If you want to
-disable any of the default enabled extensions you must assign ``None`` as their
-value.
-
 For more information See the :ref:`extensions user guide  <topics-extensions>`
 and the :ref:`list of available extensions <topics-extensions-ref>`.
 
@@ -589,6 +604,16 @@ Example::
        'mybot.pipelines.validate.StoreMyItem': 800,
    }
 
+.. setting:: ITEM_PIPELINES_BASE
+
+ITEM_PIPELINES_BASE
+-------------------
+
+Default: ``{}``
+
+A dict containing the pipelines enabled by default in Scrapy. You should never
+modify this setting in your project, modify :setting:`ITEM_PIPELINES` instead.
+
 .. setting:: LOG_ENABLED
 
 LOG_ENABLED
@@ -878,6 +903,16 @@ The scheduler to use for crawling.
 SPIDER_CONTRACTS
 ----------------
 
+Default:: ``{}``
+
+A dict containing the spider contracts enabled in your project, used for
+testing spiders. For more info see :ref:`topics-contracts`.
+
+.. setting:: SPIDER_CONTRACTS_BASE
+
+SPIDER_CONTRACTS_BASE
+---------------------
+
 Default::
 
     {
@@ -886,13 +921,17 @@ Default::
         'scrapy.contracts.default.ScrapesContract': 3,
     }
 
-A dict containing the scrapy contracts enabled in your project, used for
-testing spiders. For more info see :ref:`topics-contracts`.
+A dict containing the scrapy contracts enabled by default in Scrapy. You should
+never modify this setting in your project, modify :setting:`SPIDER_CONTRACTS`
+instead. For more info see :ref:`topics-contracts`.
+
+You can disable any of these contracts by assigning ``None`` to their class
+path in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in
+``ScrapesContract``, place this in your ``settings.py``::
 
-When you set :setting:`SPIDER_CONTRACTS` manually, e.g. in your project's
-settings module, it will be merged with the default, not overwrite it. If you
-want to disable any of the default contracts you must assign ``None`` as their
-value.
+    SPIDER_CONTRACTS = {
+        'scrapy.contracts.default.ScrapesContract': None,
+    }
 
 .. setting:: SPIDER_LOADER_CLASS
 
@@ -909,6 +948,16 @@ The class that will be used for loading spiders, which must implement the
 SPIDER_MIDDLEWARES
 ------------------
 
+Default:: ``{}``
+
+A dict containing the spider middlewares enabled in your project, and their
+orders. For more info see :ref:`topics-spider-middleware-setting`.
+
+.. setting:: SPIDER_MIDDLEWARES_BASE
+
+SPIDER_MIDDLEWARES_BASE
+-----------------------
+
 Default::
 
     {
@@ -919,14 +968,9 @@ Default::
         'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,
     }
 
-A dict containing the spider middlewares enabled in your project, and their
-orders. Low orders are closer to the engine, high orders are closer to the
-spider. For more info see :ref:`topics-spider-middleware-setting`.
-
-When you set :setting:`SPIDER_MIDDLEWARES` manually, e.g. in your project's
-settings module, it will be merged with the default, not overwrite it. If you
-want to disable any of the default spider middlewares you must assign ``None``
-as their value.
+A dict containing the spider middlewares enabled by default in Scrapy, and
+their orders. Low orders are closer to the engine, high orders are closer to
+the spider. For more info see :ref:`topics-spider-middleware-setting`.
 
 .. setting:: SPIDER_MODULES
 
diff --git a/docs/topics/spider-middleware.rst b/docs/topics/spider-middleware.rst
index d448801d3..84daaaa55 100644
--- a/docs/topics/spider-middleware.rst
+++ b/docs/topics/spider-middleware.rst
@@ -24,20 +24,22 @@ Here's an example::
         'myproject.middlewares.CustomSpiderMiddleware': 543,
     }
 
-The specified :setting:`SPIDER_MIDDLEWARES` setting is merged with the default
-one (i.e. it does not overwrite it) and then sorted by order to get the final
-sorted list of enabled middlewares: the first middleware is the one closer to
-the engine and the last is the one closer to the spider.
-
-To decide which order to assign to your middleware see the default
-:setting:`SPIDER_MIDDLEWARES` setting and pick a value according to where
+The :setting:`SPIDER_MIDDLEWARES` setting is merged with the
+:setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to
+be overridden) and then sorted by order to get the final sorted list of enabled
+middlewares: the first middleware is the one closer to the engine and the last
+is the one closer to the spider.
+
+To decide which order to assign to your middleware see the
+:setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where
 you want to insert the middleware. The order does matter because each
 middleware performs a different action and your middleware could depend on some
 previous (or subsequent) middleware being applied.
 
-If you want to disable a builtin middleware you must define it in your project's
-:setting:`SPIDER_MIDDLEWARES` setting and assign ``None`` as its value. For
-example, if you want to disable the off-site middleware::
+If you want to disable a builtin middleware (the ones defined in
+:setting:`SPIDER_MIDDLEWARES_BASE`, and enabled by default) you must define it
+in your project :setting:`SPIDER_MIDDLEWARES` setting and assign `None` as its
+value.  For example, if you want to disable the off-site middleware::
 
     SPIDER_MIDDLEWARES = {
         'myproject.middlewares.CustomSpiderMiddleware': 543,
@@ -171,7 +173,7 @@ information on how to use them and how to write your own spider middleware, see
 the :ref:`spider middleware usage guide <topics-spider-middleware>`.
 
 For a list of the components enabled by default (and their orders) see the
-:setting:`SPIDER_MIDDLEWARES` setting.
+:setting:`SPIDER_MIDDLEWARES_BASE` setting.
 
 DepthMiddleware
 ---------------
diff --git a/scrapy/commands/check.py b/scrapy/commands/check.py
index a423ba2c9..b8a9ef989 100644
--- a/scrapy/commands/check.py
+++ b/scrapy/commands/check.py
@@ -58,7 +58,7 @@ class Command(ScrapyCommand):
 
     def run(self, args, opts):
         # load contracts
-        contracts = build_component_list(self.settings._getcomposite('SPIDER_CONTRACTS'))
+        contracts = build_component_list(self.settings.getwithbase('SPIDER_CONTRACTS'))
         conman = ContractsManager(load_object(c) for c in contracts)
         runner = TextTestRunner(verbosity=2 if opts.verbose else 1)
         result = TextTestResult(runner.stream, runner.descriptions, runner.verbosity)
diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py
index 7f5c64c20..4b986bf9d 100644
--- a/scrapy/commands/crawl.py
+++ b/scrapy/commands/crawl.py
@@ -35,7 +35,8 @@ class Command(ScrapyCommand):
                 self.settings.set('FEED_URI', 'stdout:', priority='cmdline')
             else:
                 self.settings.set('FEED_URI', opts.output, priority='cmdline')
-            feed_exporters = without_none_values(self.settings._getcomposite('FEED_EXPORTERS'))
+            feed_exporters = without_none_values(
+                self.settings.getwithbase('FEED_EXPORTERS'))
             valid_output_formats = feed_exporters.keys()
             if not opts.output_format:
                 opts.output_format = os.path.splitext(opts.output)[1].replace(".", "")
diff --git a/scrapy/commands/runspider.py b/scrapy/commands/runspider.py
index 72229bcf5..1da09e4da 100644
--- a/scrapy/commands/runspider.py
+++ b/scrapy/commands/runspider.py
@@ -58,7 +58,7 @@ class Command(ScrapyCommand):
                 self.settings.set('FEED_URI', 'stdout:', priority='cmdline')
             else:
                 self.settings.set('FEED_URI', opts.output, priority='cmdline')
-            feed_exporters = without_none_values(self.settings._getcomposite('FEED_EXPORTERS'))
+            feed_exporters = without_none_values(self.settings.getwithbase('FEED_EXPORTERS'))
             valid_output_formats = feed_exporters.keys()
             if not opts.output_format:
                 opts.output_format = os.path.splitext(opts.output)[1].replace(".", "")
diff --git a/scrapy/core/downloader/handlers/__init__.py b/scrapy/core/downloader/handlers/__init__.py
index 0e78e04f4..bc5cd742e 100644
--- a/scrapy/core/downloader/handlers/__init__.py
+++ b/scrapy/core/downloader/handlers/__init__.py
@@ -20,7 +20,8 @@ class DownloadHandlers(object):
         self._schemes = {}  # stores acceptable schemes on instancing
         self._handlers = {}  # stores instanced handlers for schemes
         self._notconfigured = {}  # remembers failed handlers
-        handlers = without_none_values(crawler.settings._getcomposite('DOWNLOAD_HANDLERS'))
+        handlers = without_none_values(
+            crawler.settings.getwithbase('DOWNLOAD_HANDLERS'))
         for scheme, clspath in six.iteritems(handlers):
             self._schemes[scheme] = clspath
 
diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.py
index 958113fc3..c3b23e284 100644
--- a/scrapy/core/downloader/middleware.py
+++ b/scrapy/core/downloader/middleware.py
@@ -19,7 +19,8 @@ class DownloaderMiddlewareManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(settings._getcomposite('DOWNLOADER_MIDDLEWARES'))
+        return build_component_list(
+            settings.getwithbase('DOWNLOADER_MIDDLEWARES'))
 
     def _add_middleware(self, mw):
         if hasattr(mw, 'process_request'):
diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py
index b5c80c350..a206e4b0c 100644
--- a/scrapy/core/spidermw.py
+++ b/scrapy/core/spidermw.py
@@ -18,7 +18,7 @@ class SpiderMiddlewareManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(settings._getcomposite('SPIDER_MIDDLEWARES'))
+        return build_component_list(settings.getwithbase('SPIDER_MIDDLEWARES'))
 
     def _add_middleware(self, mw):
         super(SpiderMiddlewareManager, self)._add_middleware(mw)
diff --git a/scrapy/extension.py b/scrapy/extension.py
index 4ceb32c68..e39e456fa 100644
--- a/scrapy/extension.py
+++ b/scrapy/extension.py
@@ -12,4 +12,4 @@ class ExtensionManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(settings._getcomposite('EXTENSIONS'))
+        return build_component_list(settings.getwithbase('EXTENSIONS'))
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
index 1e27a1e7e..daea551cb 100644
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -196,7 +196,7 @@ class FeedExporter(object):
         return item
 
     def _load_components(self, setting_prefix):
-        conf = without_none_values(self.settings._getcomposite(setting_prefix))
+        conf = without_none_values(self.settings.getwithbase(setting_prefix))
         d = {}
         for k, v in conf.items():
             try:
diff --git a/scrapy/http/request/form.py b/scrapy/http/request/form.py
index 4a9bd732e..a12a2fd07 100644
--- a/scrapy/http/request/form.py
+++ b/scrapy/http/request/form.py
@@ -11,7 +11,6 @@ from parsel.selector import create_root_node
 import six
 from scrapy.http.request import Request
 from scrapy.utils.python import to_bytes, is_listlike
-from scrapy.utils.response import get_base_url
 
 
 class FormRequest(Request):
@@ -45,7 +44,7 @@ class FormRequest(Request):
 
 def _get_form_url(form, url):
     if url is None:
-        return urljoin(form.base_url, form.action)
+        return form.action or form.base_url
     return urljoin(form.base_url, url)
 
 
@@ -59,7 +58,7 @@ def _urlencode(seq, enc):
 def _get_form(response, formname, formid, formnumber, formxpath):
     """Find the form element """
     text = response.body_as_unicode()
-    root = create_root_node(text, lxml.html.HTMLParser, base_url=get_base_url(response))
+    root = create_root_node(text, lxml.html.HTMLParser, base_url=response.url)
     forms = root.xpath('//form')
     if not forms:
         raise ValueError("No <form> element found in %s" % response)
diff --git a/scrapy/pipelines/__init__.py b/scrapy/pipelines/__init__.py
index 8df0d3154..2ef8786d0 100644
--- a/scrapy/pipelines/__init__.py
+++ b/scrapy/pipelines/__init__.py
@@ -13,7 +13,7 @@ class ItemPipelineManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(settings._getcomposite('ITEM_PIPELINES'))
+        return build_component_list(settings.getwithbase('ITEM_PIPELINES'))
 
     def _add_middleware(self, pipe):
         super(ItemPipelineManager, self)._add_middleware(pipe)
diff --git a/scrapy/settings/__init__.py b/scrapy/settings/__init__.py
index 13656298b..342d2585e 100644
--- a/scrapy/settings/__init__.py
+++ b/scrapy/settings/__init__.py
@@ -49,14 +49,11 @@ class SettingsAttribute(object):
 
     def set(self, value, priority):
         """Sets value if priority is higher or equal than current priority."""
-        if isinstance(self.value, BaseSettings):
-            # Ignore self.priority if self.value has per-key priorities
-            self.value.update(value, priority)
-            self.priority = max(self.value.maxpriority(), priority)
-        else:
-            if priority >= self.priority:
-                self.value = value
-                self.priority = priority
+        if priority >= self.priority:
+            if isinstance(self.value, BaseSettings):
+                value = BaseSettings(value, priority=priority)
+            self.value = value
+            self.priority = priority
 
     def __str__(self):
         return "<SettingsAttribute value={self.value!r} " \
@@ -93,10 +90,9 @@ class BaseSettings(MutableMapping):
         self.update(values, priority)
 
     def __getitem__(self, opt_name):
-        value = None
-        if opt_name in self:
-            value = self.attributes[opt_name].value
-        return value
+        if opt_name not in self:
+            return None
+        return self.attributes[opt_name].value
 
     def __contains__(self, name):
         return name in self.attributes
@@ -195,25 +191,17 @@ class BaseSettings(MutableMapping):
             value = json.loads(value)
         return dict(value)
 
-    def _getcomposite(self, name):
-        # DO NOT USE THIS FUNCTION IN YOUR CUSTOM PROJECTS
-        # It's for internal use in the transition away from the _BASE settings
-        # and will be removed along with _BASE support in a future release
-        basename = name + "_BASE"
-        if basename in self:
-            warnings.warn('_BASE settings are deprecated.',
-                          category=ScrapyDeprecationWarning)
-            # When users defined a _BASE setting, they explicitly don't want to
-            # use any of Scrapy's defaults. Therefore, we only use these entries
-            # from self[name] (where the defaults now live) that have a priority
-            # higher than 'default'
-            compsett = BaseSettings(self[basename], priority='default')
-            for k in self[name]:
-                prio = self[name].getpriority(k)
-                if prio > get_settings_priority('default'):
-                    compsett.set(k, self[name][k], prio)
-            return compsett
-        return self[name]
+    def getwithbase(self, name):
+        """Get a composition of a dictionary-like setting and its `_BASE`
+        counterpart.
+
+        :param name: name of the dictionary-like setting
+        :type name: string
+        """
+        compbs = BaseSettings()
+        compbs.update(self[name + '_BASE'])
+        compbs.update(self[name])
+        return compbs
 
     def getpriority(self, name):
         """
@@ -223,10 +211,9 @@ class BaseSettings(MutableMapping):
         :param name: the setting name
         :type name: string
         """
-        prio = None
-        if name in self:
-            prio = self.attributes[name].priority
-        return prio
+        if name not in self:
+            return None
+        return self.attributes[name].priority
 
     def maxpriority(self):
         """
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 375efcdbb..8435b0354 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -63,7 +63,8 @@ DNS_TIMEOUT = 60
 
 DOWNLOAD_DELAY = 0
 
-DOWNLOAD_HANDLERS = {
+DOWNLOAD_HANDLERS = {}
+DOWNLOAD_HANDLERS_BASE = {
     'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler',
     'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
     'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
@@ -81,7 +82,9 @@ DOWNLOADER = 'scrapy.core.downloader.Downloader'
 DOWNLOADER_HTTPCLIENTFACTORY = 'scrapy.core.downloader.webclient.ScrapyHTTPClientFactory'
 DOWNLOADER_CLIENTCONTEXTFACTORY = 'scrapy.core.downloader.contextfactory.ScrapyClientContextFactory'
 
-DOWNLOADER_MIDDLEWARES = {
+DOWNLOADER_MIDDLEWARES = {}
+
+DOWNLOADER_MIDDLEWARES_BASE = {
     # Engine side
     'scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware': 100,
     'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware': 300,
@@ -113,7 +116,9 @@ except KeyError:
     else:
         EDITOR = 'vi'
 
-EXTENSIONS = {
+EXTENSIONS = {}
+
+EXTENSIONS_BASE = {
     'scrapy.extensions.corestats.CoreStats': 0,
     'scrapy.extensions.telnet.TelnetConsole': 0,
     'scrapy.extensions.memusage.MemoryUsage': 0,
@@ -130,14 +135,16 @@ FEED_URI_PARAMS = None  # a function to extend uri arguments
 FEED_FORMAT = 'jsonlines'
 FEED_STORE_EMPTY = False
 FEED_EXPORT_FIELDS = None
-FEED_STORAGES = {
+FEED_STORAGES = {}
+FEED_STORAGES_BASE = {
     '': 'scrapy.extensions.feedexport.FileFeedStorage',
     'file': 'scrapy.extensions.feedexport.FileFeedStorage',
     'stdout': 'scrapy.extensions.feedexport.StdoutFeedStorage',
     's3': 'scrapy.extensions.feedexport.S3FeedStorage',
     'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
 }
-FEED_EXPORTERS = {
+FEED_EXPORTERS = {}
+FEED_EXPORTERS_BASE = {
     'json': 'scrapy.exporters.JsonItemExporter',
     'jsonlines': 'scrapy.exporters.JsonLinesItemExporter',
     'jl': 'scrapy.exporters.JsonLinesItemExporter',
@@ -163,6 +170,7 @@ HTTPCACHE_GZIP = False
 ITEM_PROCESSOR = 'scrapy.pipelines.ItemPipelineManager'
 
 ITEM_PIPELINES = {}
+ITEM_PIPELINES_BASE = {}
 
 LOG_ENABLED = True
 LOG_ENCODING = 'utf-8'
@@ -221,7 +229,9 @@ SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.LifoMemoryQueue'
 
 SPIDER_LOADER_CLASS = 'scrapy.spiderloader.SpiderLoader'
 
-SPIDER_MIDDLEWARES = {
+SPIDER_MIDDLEWARES = {}
+
+SPIDER_MIDDLEWARES_BASE = {
     # Engine side
     'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50,
     'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,
@@ -248,7 +258,8 @@ TELNETCONSOLE_ENABLED = 1
 TELNETCONSOLE_PORT = [6023, 6073]
 TELNETCONSOLE_HOST = '127.0.0.1'
 
-SPIDER_CONTRACTS = {
+SPIDER_CONTRACTS = {}
+SPIDER_CONTRACTS_BASE = {
     'scrapy.contracts.default.UrlContract': 1,
     'scrapy.contracts.default.ReturnsContract': 2,
     'scrapy.contracts.default.ScrapesContract': 3,
diff --git a/scrapy/utils/conf.py b/scrapy/utils/conf.py
index 57f2b6322..e8af90f11 100644
--- a/scrapy/utils/conf.py
+++ b/scrapy/utils/conf.py
@@ -10,7 +10,7 @@ from scrapy.utils.deprecate import update_classpath
 from scrapy.utils.python import without_none_values
 
 
-def build_component_list(compdict, convert=update_classpath):
+def build_component_list(compdict, custom=None, convert=update_classpath):
     """Compose a component list from a { class: order } dictionary."""
 
     def _check_components(complist):
@@ -34,9 +34,15 @@ def build_component_list(compdict, convert=update_classpath):
             _check_components(compdict)
             return {convert(k): v for k, v in six.iteritems(compdict)}
 
-    if isinstance(compdict, (list, tuple)):
-        _check_components(compdict)
-        return type(compdict)(convert(c) for c in compdict)
+    # BEGIN Backwards compatibility for old (base, custom) call signature
+    if isinstance(custom, (list, tuple)):
+        _check_components(custom)
+        return type(custom)(convert(c) for c in custom)
+
+    if custom is not None:
+        compdict.update(custom)
+    # END Backwards compatibility
+
     compdict = without_none_values(_map_keys(compdict))
     return [k for k, v in sorted(six.iteritems(compdict), key=itemgetter(1))]
 


Test patch:
diff --git a/tests/test_cmdline/__init__.py b/tests/test_cmdline/__init__.py
index 5192fb0fa..c2de4fbc8 100644
--- a/tests/test_cmdline/__init__.py
+++ b/tests/test_cmdline/__init__.py
@@ -1,10 +1,11 @@
-import os
 import json
-import sys
-import shutil
+import os
 import pstats
-import tempfile
+import shutil
+import six
 from subprocess import Popen, PIPE
+import sys
+import tempfile
 import unittest
 try:
     from cStringIO import StringIO
@@ -57,14 +58,14 @@ class CmdlineTest(unittest.TestCase):
             shutil.rmtree(path)
 
     def test_override_dict_settings(self):
+        EXT_PATH = "tests.test_cmdline.extensions.DummyExtension"
+        EXTENSIONS = {EXT_PATH: 200}
         settingsstr = self._execute('settings', '--get', 'EXTENSIONS', '-s',
-                                    ('EXTENSIONS={"tests.test_cmdline.extensions.TestExtension": '
-                                     '100, "tests.test_cmdline.extensions.DummyExtension": 200}'))
+                                    'EXTENSIONS=' + json.dumps(EXTENSIONS))
         # XXX: There's gotta be a smarter way to do this...
         self.assertNotIn("...", settingsstr)
         for char in ("'", "<", ">", 'u"'):
             settingsstr = settingsstr.replace(char, '"')
         settingsdict = json.loads(settingsstr)
-        self.assertIn('tests.test_cmdline.extensions.DummyExtension', settingsdict)
-        self.assertIn('value=200', settingsdict['tests.test_cmdline.extensions.DummyExtension'])
-        self.assertIn('value=100', settingsdict['tests.test_cmdline.extensions.TestExtension'])
+        six.assertCountEqual(self, settingsdict.keys(), EXTENSIONS.keys())
+        self.assertIn('value=200', settingsdict[EXT_PATH])
diff --git a/tests/test_http_request.py b/tests/test_http_request.py
index 60fd855dd..ff0941961 100644
--- a/tests/test_http_request.py
+++ b/tests/test_http_request.py
@@ -801,25 +801,6 @@ class FormRequestTest(RequestTest):
         self.assertEqual(fs[b'test2'], [b'val2'])
         self.assertEqual(fs[b'button1'], [b''])
 
-    def test_html_base_form_action(self):
-        response = _buildresponse(
-            """
-            <html>
-                <head>
-                    <base href="http://b.com/">
-                </head>
-                <body>
-                    <form action="test_form">
-                    </form>
-                </body>
-            </html>
-            """,
-            url='http://a.com/'
-        )
-        req = self.request_class.from_response(response)
-        self.assertEqual(req.url, 'http://b.com/test_form')
-
-
 def _buildresponse(body, **kwargs):
     kwargs.setdefault('body', body)
     kwargs.setdefault('url', 'http://example.com')
diff --git a/tests/test_settings/__init__.py b/tests/test_settings/__init__.py
index 4ef08bb0b..44b9b6df3 100644
--- a/tests/test_settings/__init__.py
+++ b/tests/test_settings/__init__.py
@@ -37,21 +37,22 @@ class SettingsAttributeTest(unittest.TestCase):
         self.assertEqual(self.attribute.value, 'value')
         self.assertEqual(self.attribute.priority, 10)
 
-    def test_set_per_key_priorities(self):
-        attribute = SettingsAttribute(
-            BaseSettings({'one': 10, 'two': 20}, 0), 0)
+    def test_overwrite_basesettings(self):
+        original_dict = {'one': 10, 'two': 20}
+        original_settings = BaseSettings(original_dict, 0)
+        attribute = SettingsAttribute(original_settings, 0)
 
-        new_dict = {'one': 11, 'two': 21}
+        new_dict = {'three': 11, 'four': 21}
         attribute.set(new_dict, 10)
-        self.assertEqual(attribute.value['one'], 11)
-        self.assertEqual(attribute.value['two'], 21)
+        self.assertIsInstance(attribute.value, BaseSettings)
+        six.assertCountEqual(self, attribute.value, new_dict)
+        six.assertCountEqual(self, original_settings, original_dict)
 
-        new_settings = BaseSettings()
-        new_settings.set('one', 12, 20)
-        new_settings.set('two', 12, 0)
-        attribute.set(new_settings, 0)
-        self.assertEqual(attribute.value['one'], 12)
-        self.assertEqual(attribute.value['two'], 21)
+        new_settings = BaseSettings({'five': 12}, 0)
+        attribute.set(new_settings, 0)  # Insufficient priority
+        six.assertCountEqual(self, attribute.value, new_dict)
+        attribute.set(new_settings, 10)
+        six.assertCountEqual(self, attribute.value, new_settings)
 
     def test_repr(self):
         self.assertEqual(repr(self.attribute),
@@ -263,24 +264,15 @@ class BaseSettingsTest(unittest.TestCase):
         self.assertEqual(settings.getpriority('key'), 99)
         self.assertEqual(settings.getpriority('nonexistentkey'), None)
 
-    def test_getcomposite(self):
-        s = BaseSettings({'TEST_BASE': {1: 1, 2: 2},
+    def test_getwithbase(self):
+        s = BaseSettings({'TEST_BASE': BaseSettings({1: 1, 2: 2}, 'project'),
                           'TEST': BaseSettings({1: 10, 3: 30}, 'default'),
-                          'HASNOBASE': BaseSettings({1: 1}, 'default')})
-        s['TEST'].set(4, 4, priority='project')
-        # When users specify a _BASE setting they explicitly don't want to use
-        # Scrapy's defaults, so we don't want to see anything that has a
-        # 'default' priority from TEST
-        cs = s._getcomposite('TEST')
-        self.assertEqual(len(cs), 3)
-        self.assertEqual(cs[1], 1)
-        self.assertEqual(cs[2], 2)
-        self.assertEqual(cs[4], 4)
-        cs = s._getcomposite('HASNOBASE')
-        self.assertEqual(len(cs), 1)
-        self.assertEqual(cs[1], 1)
-        cs = s._getcomposite('NONEXISTENT')
-        self.assertIsNone(cs)
+                          'HASNOBASE': BaseSettings({3: 3000}, 'default')})
+        s['TEST'].set(2, 200, 'cmdline')
+        six.assertCountEqual(self, s.getwithbase('TEST'),
+                             {1: 1, 2: 200, 3: 30})
+        six.assertCountEqual(self, s.getwithbase('HASNOBASE'), s['HASNOBASE'])
+        self.assertEqual(s.getwithbase('NONEXISTENT'), {})
 
     def test_maxpriority(self):
         # Empty settings should return 'default'
diff --git a/tests/test_utils_conf.py b/tests/test_utils_conf.py
index af15d3184..dab41ac8d 100644
--- a/tests/test_utils_conf.py
+++ b/tests/test_utils_conf.py
@@ -8,46 +8,59 @@ class BuildComponentListTest(unittest.TestCase):
 
     def test_build_dict(self):
         d = {'one': 1, 'two': None, 'three': 8, 'four': 4}
-        self.assertEqual(build_component_list(d, lambda x: x),
+        self.assertEqual(build_component_list(d, convert=lambda x: x),
                          ['one', 'four', 'three'])
 
+    def test_backwards_compatible_build_dict(self):
+        base = {'one': 1, 'two': 2, 'three': 3, 'five': 5, 'six': None}
+        custom = {'two': None, 'three': 8, 'four': 4}
+        self.assertEqual(build_component_list(base, custom,
+                                              convert=lambda x: x),
+                         ['one', 'four', 'five', 'three'])
+
     def test_return_list(self):
         custom = ['a', 'b', 'c']
-        self.assertEqual(build_component_list(custom, lambda x: x), custom)
+        self.assertEqual(build_component_list(None, custom,
+                                              convert=lambda x: x),
+                         custom)
 
     def test_map_dict(self):
         custom = {'one': 1, 'two': 2, 'three': 3}
-        self.assertEqual(build_component_list(custom, lambda x: x.upper()),
+        self.assertEqual(build_component_list({}, custom,
+                                              convert=lambda x: x.upper()),
                          ['ONE', 'TWO', 'THREE'])
 
     def test_map_list(self):
         custom = ['a', 'b', 'c']
-        self.assertEqual(build_component_list(custom, lambda x: x.upper()),
+        self.assertEqual(build_component_list(None, custom,
+                                              lambda x: x.upper()),
                          ['A', 'B', 'C'])
 
     def test_duplicate_components_in_dict(self):
         duplicate_dict = {'one': 1, 'two': 2, 'ONE': 4}
-        self.assertRaises(ValueError,
-                          build_component_list, duplicate_dict, lambda x: x.lower())
+        self.assertRaises(ValueError, build_component_list, {}, duplicate_dict,
+                          convert=lambda x: x.lower())
 
     def test_duplicate_components_in_list(self):
         duplicate_list = ['a', 'b', 'a']
-        self.assertRaises(ValueError,
-                          build_component_list, duplicate_list, lambda x: x)
+        self.assertRaises(ValueError, build_component_list, None,
+                          duplicate_list, convert=lambda x: x)
 
     def test_duplicate_components_in_basesettings(self):
         # Higher priority takes precedence
         duplicate_bs = BaseSettings({'one': 1, 'two': 2}, priority=0)
         duplicate_bs.set('ONE', 4, priority=10)
-        self.assertEqual(build_component_list(duplicate_bs, convert=lambda x: x.lower()),
+        self.assertEqual(build_component_list(duplicate_bs,
+                                              convert=lambda x: x.lower()),
                          ['two', 'one'])
         duplicate_bs.set('one', duplicate_bs['one'], priority=20)
-        self.assertEqual(build_component_list(duplicate_bs, convert=lambda x: x.lower()),
+        self.assertEqual(build_component_list(duplicate_bs,
+                                              convert=lambda x: x.lower()),
                          ['one', 'two'])
         # Same priority raises ValueError
         duplicate_bs.set('ONE', duplicate_bs['ONE'], priority=20)
-        self.assertRaises(ValueError,
-                          build_component_list, duplicate_bs, convert=lambda x: x.lower())
+        self.assertRaises(ValueError, build_component_list, duplicate_bs,
+                          convert=lambda x: x.lower())
 
 
 class UtilsConfTestCase(unittest.TestCase):