Repository: scrapy/scrapy
Pull request #1544 (2015-10-16 00:40:20)

Added installation notes about using Conda for Windows.
This PR adds installation notes about installing Scrapy via conda and the Scrapinghub's anaconda.org channel.

![](http://i.imgur.com/xs6grbI.png)

![](http://i.imgur.com/D6nryYj.png)

codecov-io: ## [Current coverage][1] is `82.62%`
> Merging **#1544** into **master** will not affect coverage as of [`2bbdb86`][3]



[1]: https://codecov.io/github/scrapy/scrapy?ref=2bbdb8678aeebaa06fa5f8337ff18f39d886e179
[2]: https://codecov.io/github/scrapy/scrapy/features/suggestions?ref=2bbdb8678aeebaa06fa5f8337ff18f39d886e179
[3]: https://codecov.io/github/scrapy/scrapy/commit/2bbdb8678aeebaa06fa5f8337ff18f39d886e179
[4]: https://codecov.io/github/scrapy/scrapy/compare/8dc400c66d2001a28cd3b717355e5e09a5da7b18...2bbdb8678aeebaa06fa5f8337ff18f39d886e179

> Powered by [Codecov](https://codecov.io). Updated on successful CI builds.
kmike: Looks good, given that conda package works. 
Is it OK not to use service_identity? Will it cause failed HTTPS connections for Windows users (see https://github.com/scrapy/scrapy/issues/1418)?
rolando: That's a valid concern. I have created this issue to address it: https://github.com/scrapinghub/scrapinghub-conda-recipes/issues/2
rolando: @kmike this PR restores `service_identity` as a dependency https://github.com/scrapinghub/scrapinghub-conda-recipes/pull/4 
kmike: @rolando is this PR ready to merge?
rolando: @kmike yes, it's good to merge.

Patch:
diff --git a/docs/faq.rst b/docs/faq.rst
index 3d2bd8d4d..2e61f44ee 100644
--- a/docs/faq.rst
+++ b/docs/faq.rst
@@ -144,7 +144,7 @@ I get "Filtered offsite request" messages. How can I fix them?
 Those messages (logged with ``DEBUG`` level) don't necessarily mean there is a
 problem, so you may not need to fix them.
 
-Those messages are thrown by the Offsite Spider Middleware, which is a spider
+Those message are thrown by the Offsite Spider Middleware, which is a spider
 middleware (enabled by default) whose purpose is to filter out requests to
 domains outside the ones covered by the spider.
 
diff --git a/docs/intro/install.rst b/docs/intro/install.rst
index 3adb4e6b0..abb94c006 100644
--- a/docs/intro/install.rst
+++ b/docs/intro/install.rst
@@ -24,9 +24,7 @@ The installation steps assume that you have the following things installed:
   where the Python installer ships it bundled.
 
 You can install Scrapy using pip (which is the canonical way to install Python
-packages).
-
-To install using pip::
+packages). To install using ``pip`` run::
 
    pip install Scrapy
 
@@ -35,6 +33,25 @@ To install using pip::
 Platform specific installation notes
 ====================================
 
+Anaconda
+--------
+
+.. note::
+
+  For Windows users, or if you have issues installing through `pip`, this is
+  the recommended way to install Scrapy.
+
+If you already have installed `Anaconda`_ or `Miniconda`_, the company
+`Scrapinghub`_ maintains official conda packages for Linux, Windows and OS X.
+
+To install Scrapy in Linux or OS X, use:
+
+  conda install -c scrapinghub scrapy 
+
+To install Scrapy in Windows, use:
+
+  conda install -c scrapinghub scrapy pywin32
+
 Windows
 -------
 
@@ -167,3 +184,6 @@ After any of these workarounds you should be able to install Scrapy::
 .. _homebrew: http://brew.sh/
 .. _zsh: http://www.zsh.org/
 .. _virtualenv: https://virtualenv.pypa.io/en/latest/
+.. _Scrapinghub: http://scrapinghub.com
+.. _Anaconda: http://docs.continuum.io/anaconda/index
+.. _Miniconda: http://conda.pydata.org/docs/install/quick.html
diff --git a/docs/topics/api.rst b/docs/topics/api.rst
index 42c0133c1..f54341eb8 100644
--- a/docs/topics/api.rst
+++ b/docs/topics/api.rst
@@ -140,14 +140,170 @@ Settings API
     For a detailed explanation on each settings sources, see:
     :ref:`topics-settings`.
 
-.. autofunction:: get_settings_priority
+.. class:: Settings(values={}, priority='project')
 
-.. autoclass:: Settings
-   :show-inheritance:
-   :members:
+    This object stores Scrapy settings for the configuration of internal
+    components, and can be used for any further customization.
 
-.. autoclass:: BaseSettings
-   :members:
+    After instantiation of this class, the new object will have the global
+    default settings described on :ref:`topics-settings-ref` already
+    populated.
+
+    Additional values can be passed on initialization with the ``values``
+    argument, and they would take the ``priority`` level.  If the latter
+    argument is a string, the priority name will be looked up in
+    :attr:`~scrapy.settings.SETTINGS_PRIORITIES`. Otherwise, a expecific
+    integer should be provided.
+
+    Once the object is created, new settings can be loaded or updated with the
+    :meth:`~scrapy.settings.Settings.set` method, and can be accessed with the
+    square bracket notation of dictionaries, or with the
+    :meth:`~scrapy.settings.Settings.get` method of the instance and its value
+    conversion variants.  When requesting a stored key, the value with the
+    highest priority will be retrieved.
+
+    .. method:: set(name, value, priority='project')
+
+       Store a key/value attribute with a given priority.
+
+       Settings should be populated *before* configuring the Crawler object
+       (through the :meth:`~scrapy.crawler.Crawler.configure` method),
+       otherwise they won't have any effect.
+
+       :param name: the setting name
+       :type name: string
+
+       :param value: the value to associate with the setting
+       :type value: any
+
+       :param priority: the priority of the setting. Should be a key of
+           :attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
+       :type priority: string or int
+
+    .. method:: setdict(values, priority='project')
+
+       Store key/value pairs with a given priority.
+
+       This is a helper function that calls
+       :meth:`~scrapy.settings.Settings.set` for every item of ``values``
+       with the provided ``priority``.
+
+       :param values: the settings names and values
+       :type values: dict
+
+       :param priority: the priority of the settings. Should be a key of
+           :attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
+       :type priority: string or int
+
+    .. method:: setmodule(module, priority='project')
+
+       Store settings from a module with a given priority.
+
+       This is a helper function that calls
+       :meth:`~scrapy.settings.Settings.set` for every globally declared
+       uppercase variable of ``module`` with the provided ``priority``.
+
+       :param module: the module or the path of the module
+       :type module: module object or string
+
+       :param priority: the priority of the settings. Should be a key of
+           :attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
+       :type priority: string or int
+
+    .. method:: get(name, default=None)
+
+       Get a setting value without affecting its original type.
+
+       :param name: the setting name
+       :type name: string
+
+       :param default: the value to return if no setting is found
+       :type default: any
+
+    .. method:: getbool(name, default=False)
+
+       Get a setting value as a boolean. For example, both ``1`` and ``'1'``, and
+       ``True`` return ``True``, while ``0``, ``'0'``, ``False`` and ``None``
+       return ``False````
+
+       For example, settings populated through environment variables set to ``'0'``
+       will return ``False`` when using this method.
+
+       :param name: the setting name
+       :type name: string
+
+       :param default: the value to return if no setting is found
+       :type default: any
+
+    .. method:: getint(name, default=0)
+
+       Get a setting value as an int
+
+       :param name: the setting name
+       :type name: string
+
+       :param default: the value to return if no setting is found
+       :type default: any
+
+    .. method:: getfloat(name, default=0.0)
+
+       Get a setting value as a float
+
+       :param name: the setting name
+       :type name: string
+
+       :param default: the value to return if no setting is found
+       :type default: any
+
+    .. method:: getlist(name, default=None)
+
+       Get a setting value as a list. If the setting original type is a list, a
+       copy of it will be returned. If it's a string it will be split by ",".
+
+       For example, settings populated through environment variables set to
+       ``'one,two'`` will return a list ['one', 'two'] when using this method.
+
+       :param name: the setting name
+       :type name: string
+
+       :param default: the value to return if no setting is found
+       :type default: any
+
+    .. method:: getdict(name, default=None)
+
+       Get a setting value as a dictionary. If the setting original type is a
+       dictionary, a copy of it will be returned. If it's a string it will
+       evaluated as a json dictionary.
+
+       :param name: the setting name
+       :type name: string
+
+       :param default: the value to return if no setting is found
+       :type default: any
+
+    .. method:: copy()
+
+       Make a deep copy of current settings.
+
+       This method returns a new instance of the :class:`Settings` class,
+       populated with the same values and their priorities.
+
+       Modifications to the new object won't be reflected on the original
+       settings.
+
+    .. method:: freeze()
+
+       Disable further changes to the current settings.
+
+       After calling this method, the present state of the settings will become
+       immutable. Trying to change values through the :meth:`~set` method and
+       its variants won't be possible and will be alerted.
+
+    .. method:: frozencopy()
+
+       Return an immutable copy of the current settings.
+
+       Alias for a :meth:`~freeze` call in the object returned by :meth:`copy`
 
 .. _topics-api-spiderloader:
 
diff --git a/docs/topics/broad-crawls.rst b/docs/topics/broad-crawls.rst
index 79f0b3b53..aaf46bc92 100644
--- a/docs/topics/broad-crawls.rst
+++ b/docs/topics/broad-crawls.rst
@@ -34,7 +34,7 @@ These are some common properties often found in broad crawls:
 
 As said above, Scrapy default settings are optimized for focused crawls, not
 broad crawls. However, due to its asynchronous architecture, Scrapy is very
-well suited for performing fast broad crawls. This page summarizes some things
+well suited for performing fast broad crawls. This page summarize some things
 you need to keep in mind when using Scrapy for doing broad crawls, along with
 concrete suggestions of Scrapy settings to tune in order to achieve an
 efficient broad crawl.
@@ -46,7 +46,7 @@ Concurrency is the number of requests that are processed in parallel. There is
 a global limit and a per-domain limit.
 
 The default global concurrency limit in Scrapy is not suitable for crawling
-many different domains in parallel, so you will want to increase it. How much
+many different  domains in parallel, so you will want to increase it. How much
 to increase it will depend on how much CPU you crawler will have available. A
 good starting point is ``100``, but the best way to find out is by doing some
 trials and identifying at what concurrency your Scrapy process gets CPU
diff --git a/docs/topics/downloader-middleware.rst b/docs/topics/downloader-middleware.rst
index cc0254d29..9122e5cb5 100644
--- a/docs/topics/downloader-middleware.rst
+++ b/docs/topics/downloader-middleware.rst
@@ -24,10 +24,10 @@ Here's an example::
     }
 
 The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
-:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant
-to be overridden) and then sorted by order to get the final sorted list of
-enabled middlewares: the first middleware is the one closer to the engine and
-the last is the one closer to the downloader.
+:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to
+be overridden) and then sorted by order to get the final sorted list of enabled
+middlewares: the first middleware is the one closer to the engine and the last
+is the one closer to the downloader.
 
 To decide which order to assign to your middleware see the
 :setting:`DOWNLOADER_MIDDLEWARES_BASE` setting and pick a value according to
diff --git a/docs/topics/extensions.rst b/docs/topics/extensions.rst
index 14528ba9a..fb5220e9d 100644
--- a/docs/topics/extensions.rst
+++ b/docs/topics/extensions.rst
@@ -17,7 +17,7 @@ Extensions use the :ref:`Scrapy settings <topics-settings>` to manage their
 settings, just like any other Scrapy code.
 
 It is customary for extensions to prefix their settings with their own name, to
-avoid collision with existing (and future) extensions. For example, a
+avoid collision with existing (and future) extensions. For example, an
 hypothetic extension to handle `Google Sitemaps`_ would use settings like
 `GOOGLESITEMAP_ENABLED`, `GOOGLESITEMAP_DEPTH`, and so on.
 
@@ -36,22 +36,23 @@ by a string: the full Python path to the extension's class name. For example::
 
     EXTENSIONS = {
         'scrapy.extensions.corestats.CoreStats': 500,
-        'scrapy.extensions.telnet.TelnetConsole': 500,
+        'scrapy.telnet.TelnetConsole': 500,
     }
 
 
 As you can see, the :setting:`EXTENSIONS` setting is a dict where the keys are
 the extension paths, and their values are the orders, which define the
-extension *loading* order. The :setting:`EXTENSIONS` setting is merged with the
-:setting:`EXTENSIONS_BASE` setting defined in Scrapy (and not meant to be
-overridden) and then sorted by order to get the final sorted list of enabled
-extensions.
+extension *loading* order. Extensions orders are not as important as middleware
+orders though, and they are typically irrelevant, ie. it doesn't matter in
+which order the extensions are loaded because they don't depend on each other
+[1].
 
-As extensions typically do not depend on each other, their loading order is
-irrelevant in most cases. This is why the :setting:`EXTENSIONS_BASE` setting
-defines all extensions with the same order (``0``). However, this feature can
-be exploited if you need to add an extension which depends on other extensions
-already loaded.
+However, this feature can be exploited if you need to add an extension which
+depends on other extensions already loaded.
+
+[1] This is is why the :setting:`EXTENSIONS_BASE` setting in Scrapy (which
+contains all built-in extensions enabled by default) defines all the extensions
+with the same order (``500``).
 
 Available, enabled and disabled extensions
 ==========================================
@@ -144,7 +145,7 @@ Here is the code of such extension::
             self.items_scraped += 1
             if self.items_scraped % self.item_count == 0:
                 logger.info("scraped %d items", self.items_scraped)
-
+                
 
 .. _topics-extensions-ref:
 
@@ -180,10 +181,10 @@ enabled (see :ref:`topics-stats`).
 Telnet console extension
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. module:: scrapy.extensions.telnet
+.. module:: scrapy.telnet
    :synopsis: Telnet console
 
-.. class:: scrapy.extensions.telnet.TelnetConsole
+.. class:: scrapy.telnet.TelnetConsole
 
 Provides a telnet console for getting into a Python interpreter inside the
 currently running Scrapy process, which can be very useful for debugging.
diff --git a/docs/topics/feed-exports.rst b/docs/topics/feed-exports.rst
index 03c6fb3fb..d9444e34a 100644
--- a/docs/topics/feed-exports.rst
+++ b/docs/topics/feed-exports.rst
@@ -285,14 +285,7 @@ Default::
         'ftp': 'scrapy.extensions.feedexport.FTPFeedStorage',
     }
 
-A dict containing the built-in feed storage backends supported by Scrapy. You
-can disable any of these backends by assigning ``None`` to their URI scheme in
-:setting:`FEED_STORAGES`. E.g., to disable the built-in FTP storage backend
-(without replacement), place this in your ``settings.py``::
-
-    FEED_STORAGES = {
-        'ftp': None,
-    }
+A dict containing the built-in feed storage backends supported by Scrapy.
 
 .. setting:: FEED_EXPORTERS
 
@@ -302,33 +295,26 @@ FEED_EXPORTERS
 Default:: ``{}``
 
 A dict containing additional exporters supported by your project. The keys are
-serialization formats and the values are paths to :ref:`Item exporter
-<topics-exporters>` classes.
+URI schemes and the values are paths to :ref:`Item exporter <topics-exporters>`
+classes.
 
 .. setting:: FEED_EXPORTERS_BASE
 
 FEED_EXPORTERS_BASE
 -------------------
+
 Default::
 
-    {
+    FEED_EXPORTERS_BASE = {
         'json': 'scrapy.exporters.JsonItemExporter',
         'jsonlines': 'scrapy.exporters.JsonLinesItemExporter',
-        'jl': 'scrapy.exporters.JsonLinesItemExporter',
         'csv': 'scrapy.exporters.CsvItemExporter',
         'xml': 'scrapy.exporters.XmlItemExporter',
         'marshal': 'scrapy.exporters.MarshalItemExporter',
-        'pickle': 'scrapy.exporters.PickleItemExporter',
     }
 
-A dict containing the built-in feed exporters supported by Scrapy. You can
-disable any of these exporters by assigning ``None`` to their serialization
-format in :setting:`FEED_EXPORTERS`. E.g., to disable the built-in CSV exporter
-(without replacement), place this in your ``settings.py``::
+A dict containing the built-in feed exporters supported by Scrapy.
 
-    FEED_EXPORTERS = {
-        'csv': None,
-    }
 
 .. _URI: http://en.wikipedia.org/wiki/Uniform_Resource_Identifier
 .. _Amazon S3: http://aws.amazon.com/s3/
diff --git a/docs/topics/item-pipeline.rst b/docs/topics/item-pipeline.rst
index 28969be61..f74400b4d 100644
--- a/docs/topics/item-pipeline.rst
+++ b/docs/topics/item-pipeline.rst
@@ -95,7 +95,7 @@ contain a price::
 Write items to a JSON file
 --------------------------
 
-The following pipeline stores all scraped items (from all spiders) into a
+The following pipeline stores all scraped items (from all spiders) into a a
 single ``items.jl`` file, containing one item per line serialized in JSON
 format::
 
diff --git a/docs/topics/items.rst b/docs/topics/items.rst
index 4a8f47e93..21ec0ed8c 100644
--- a/docs/topics/items.rst
+++ b/docs/topics/items.rst
@@ -61,7 +61,7 @@ the example above.
 You can specify any kind of metadata for each field. There is no restriction on
 the values accepted by :class:`Field` objects. For this same
 reason, there is no reference list of all available metadata keys. Each key
-defined in :class:`Field` objects could be used by a different component, and
+defined in :class:`Field` objects could be used by a different components, and
 only those components know about it. You can also define and use any other
 :class:`Field` key in your project too, for your own needs. The main goal of
 :class:`Field` objects is to provide a way to define all field metadata in one
diff --git a/docs/topics/leaks.rst b/docs/topics/leaks.rst
index 92590c180..735137ea2 100644
--- a/docs/topics/leaks.rst
+++ b/docs/topics/leaks.rst
@@ -97,7 +97,7 @@ subclasses):
 A real example
 --------------
 
-Let's see a concrete example of a hypothetical case of memory leaks.
+Let's see a concrete example of an hypothetical case of memory leaks.
 Suppose we have some spider with a line similar to this one::
 
     return Request("http://www.somenastyspider.com/product.php?pid=%d" % product_id,
diff --git a/docs/topics/media-pipeline.rst b/docs/topics/media-pipeline.rst
index 5ed6ce97d..4ee4f1758 100644
--- a/docs/topics/media-pipeline.rst
+++ b/docs/topics/media-pipeline.rst
@@ -7,7 +7,7 @@ Downloading and processing files and images
 .. currentmodule:: scrapy.pipelines.images
 
 Scrapy provides reusable :doc:`item pipelines </topics/item-pipeline>` for
-downloading files attached to a particular item (for example, when you scrape
+downloading fies attached to a particular item (for example, when you scrape
 products and also want to download their images locally). These pipelines share
 a bit of functionality and structure (we refer to them as media pipelines), but
 typically you'll either use the Files Pipeline or the Images Pipeline.
diff --git a/docs/topics/practices.rst b/docs/topics/practices.rst
index 60fe2267c..9ae34f423 100644
--- a/docs/topics/practices.rst
+++ b/docs/topics/practices.rst
@@ -228,7 +228,7 @@ with varying degrees of sophistication. Getting around those measures can be
 difficult and tricky, and may sometimes require special infrastructure. Please
 consider contacting `commercial support`_ if in doubt.
 
-Here are some tips to keep in mind when dealing with these kinds of sites:
+Here are some tips to keep in mind when dealing with these kind of sites:
 
 * rotate your user agent from a pool of well-known ones from browsers (google
   around to get a list of them)
diff --git a/docs/topics/selectors.rst b/docs/topics/selectors.rst
index 8dc82dfe5..273cae0f8 100644
--- a/docs/topics/selectors.rst
+++ b/docs/topics/selectors.rst
@@ -579,7 +579,7 @@ Built-in Selectors reference
     is used together with ``text``.
 
     If ``type`` is ``None`` and a ``response`` is passed, the selector type is
-    inferred from the response type as follows:
+    inferred from the response type as follow:
 
         * ``"html"`` for :class:`~scrapy.http.HtmlResponse` type
         * ``"xml"`` for :class:`~scrapy.http.XmlResponse` type
@@ -757,7 +757,7 @@ nodes can be accessed directly by their names::
      <Selector xpath='//link' data=u'<link xmlns="http://www.w3.org/2005/Atom'>,
      ...
 
-If you wonder why the namespace removal procedure isn't always called by default
+If you wonder why the namespace removal procedure isn't called always by default
 instead of having to call it manually, this is because of two reasons, which, in order
 of relevance, are:
 
diff --git a/docs/topics/settings.rst b/docs/topics/settings.rst
index aa0417e1a..484065406 100644
--- a/docs/topics/settings.rst
+++ b/docs/topics/settings.rst
@@ -369,7 +369,6 @@ Default::
         'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': 400,
         'scrapy.downloadermiddlewares.retry.RetryMiddleware': 500,
         'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware': 550,
-        'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware': 560,
         'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware': 580,
         'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 590,
         'scrapy.downloadermiddlewares.redirect.RedirectMiddleware': 600,
@@ -380,8 +379,7 @@ Default::
         'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': 900,
     }
 
-A dict containing the downloader middlewares enabled by default in Scrapy. Low
-orders are closer to the engine, high orders are closer to the downloader. You
+A dict containing the downloader middlewares enabled by default in Scrapy. You
 should never modify this setting in your project, modify
 :setting:`DOWNLOADER_MIDDLEWARES` instead.  For more info see
 :ref:`topics-downloader-middleware-setting`.
@@ -428,7 +426,7 @@ DOWNLOAD_HANDLERS
 Default: ``{}``
 
 A dict containing the request downloader handlers enabled in your project.
-See :setting:`DOWNLOAD_HANDLERS_BASE` for example format.
+See `DOWNLOAD_HANDLERS_BASE` for example format.
 
 .. setting:: DOWNLOAD_HANDLERS_BASE
 
@@ -439,23 +437,22 @@ Default::
 
     {
         'file': 'scrapy.core.downloader.handlers.file.FileDownloadHandler',
-        'http': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
-        'https': 'scrapy.core.downloader.handlers.http.HTTPDownloadHandler',
+        'http': 'scrapy.core.downloader.handlers.http.HttpDownloadHandler',
+        'https': 'scrapy.core.downloader.handlers.http.HttpDownloadHandler',
         's3': 'scrapy.core.downloader.handlers.s3.S3DownloadHandler',
-        'ftp': 'scrapy.core.downloader.handlers.ftp.FTPDownloadHandler',
     }
 
-
 A dict containing the request download handlers enabled by default in Scrapy.
 You should never modify this setting in your project, modify
 :setting:`DOWNLOAD_HANDLERS` instead.
 
-You can disable any of these download handlers by assigning ``None`` to their
-URI scheme in :setting:`DOWNLOAD_HANDLERS`. E.g., to disable the built-in FTP
-handler (without replacement), place this in your ``settings.py``::
+If you want to disable any of the above download handlers you must define them
+in your project's :setting:`DOWNLOAD_HANDLERS` setting and assign `None`
+as their value.  For example, if you want to disable the file download
+handler::
 
     DOWNLOAD_HANDLERS = {
-        'ftp': None,
+        'file': None,
     }
 
 .. setting:: DOWNLOAD_TIMEOUT
@@ -568,7 +565,7 @@ Default::
 
     {
         'scrapy.extensions.corestats.CoreStats': 0,
-        'scrapy.extensions.telnet.TelnetConsole': 0,
+        'scrapy.telnet.TelnetConsole': 0,
         'scrapy.extensions.memusage.MemoryUsage': 0,
         'scrapy.extensions.memdebug.MemoryDebugger': 0,
         'scrapy.extensions.closespider.CloseSpider': 0,
@@ -578,14 +575,13 @@ Default::
         'scrapy.extensions.throttle.AutoThrottle': 0,
     }
 
-A dict containing the extensions available by default in Scrapy, and their
-orders. This setting contains all stable built-in extensions. Keep in mind that
-some of them need to be enabled through a setting.
+The list of available extensions. Keep in mind that some of them need to
+be enabled through a setting. By default, this setting contains all stable
+built-in extensions.
 
 For more information See the :ref:`extensions user guide  <topics-extensions>`
 and the :ref:`list of available extensions <topics-extensions-ref>`.
 
-
 .. setting:: ITEM_PIPELINES
 
 ITEM_PIPELINES
@@ -593,9 +589,12 @@ ITEM_PIPELINES
 
 Default: ``{}``
 
-A dict containing the item pipelines to use, and their orders. Order values are
-arbitrary, but it is customary to define them in the 0-1000 range. Lower orders
-process before higher orders.
+A dict containing the item pipelines to use, and their orders. The dict is
+empty by default order values are arbitrary but it's customary to define them
+in the 0-1000 range.
+
+Lists are supported in :setting:`ITEM_PIPELINES` for backwards compatibility,
+but they are deprecated.
 
 Example::
 
@@ -639,7 +638,7 @@ LOG_FILE
 
 Default: ``None``
 
-File name to use for logging output. If ``None``, standard error will be used.
+File name to use for logging output. If None, standard error will be used.
 
 .. setting:: LOG_FORMAT
 
@@ -905,7 +904,7 @@ SPIDER_CONTRACTS
 
 Default:: ``{}``
 
-A dict containing the spider contracts enabled in your project, used for
+A dict containing the scrapy contracts enabled in your project, used for
 testing spiders. For more info see :ref:`topics-contracts`.
 
 .. setting:: SPIDER_CONTRACTS_BASE
@@ -925,14 +924,6 @@ A dict containing the scrapy contracts enabled by default in Scrapy. You should
 never modify this setting in your project, modify :setting:`SPIDER_CONTRACTS`
 instead. For more info see :ref:`topics-contracts`.
 
-You can disable any of these contracts by assigning ``None`` to their class
-path in :setting:`SPIDER_CONTRACTS`. E.g., to disable the built-in
-``ScrapesContract``, place this in your ``settings.py``::
-
-    SPIDER_CONTRACTS = {
-        'scrapy.contracts.default.ScrapesContract': None,
-    }
-
 .. setting:: SPIDER_LOADER_CLASS
 
 SPIDER_LOADER_CLASS
@@ -968,9 +959,10 @@ Default::
         'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,
     }
 
-A dict containing the spider middlewares enabled by default in Scrapy, and
-their orders. Low orders are closer to the engine, high orders are closer to
-the spider. For more info see :ref:`topics-spider-middleware-setting`.
+A dict containing the spider middlewares enabled by default in Scrapy. You
+should never modify this setting in your project, modify
+:setting:`SPIDER_MIDDLEWARES` instead. For more info see
+:ref:`topics-spider-middleware-setting`.
 
 .. setting:: SPIDER_MODULES
 
diff --git a/docs/topics/telnetconsole.rst b/docs/topics/telnetconsole.rst
index ce79c9f35..b1ff88771 100644
--- a/docs/topics/telnetconsole.rst
+++ b/docs/topics/telnetconsole.rst
@@ -4,7 +4,7 @@
 Telnet Console
 ==============
 
-.. module:: scrapy.extensions.telnet
+.. module:: scrapy.telnet
    :synopsis: The Telnet Console
 
 Scrapy comes with a built-in telnet console for inspecting and controlling a
diff --git a/scrapy/commands/check.py b/scrapy/commands/check.py
index b8a9ef989..017595f04 100644
--- a/scrapy/commands/check.py
+++ b/scrapy/commands/check.py
@@ -58,7 +58,10 @@ class Command(ScrapyCommand):
 
     def run(self, args, opts):
         # load contracts
-        contracts = build_component_list(self.settings.getwithbase('SPIDER_CONTRACTS'))
+        contracts = build_component_list(
+            self.settings['SPIDER_CONTRACTS_BASE'],
+            self.settings['SPIDER_CONTRACTS'],
+        )
         conman = ContractsManager(load_object(c) for c in contracts)
         runner = TextTestRunner(verbosity=2 if opts.verbose else 1)
         result = TextTestResult(runner.stream, runner.descriptions, runner.verbosity)
diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py
index 4b986bf9d..72df11476 100644
--- a/scrapy/commands/crawl.py
+++ b/scrapy/commands/crawl.py
@@ -1,7 +1,6 @@
 import os
 from scrapy.commands import ScrapyCommand
 from scrapy.utils.conf import arglist_to_dict
-from scrapy.utils.python import without_none_values
 from scrapy.exceptions import UsageError
 
 
@@ -35,9 +34,10 @@ class Command(ScrapyCommand):
                 self.settings.set('FEED_URI', 'stdout:', priority='cmdline')
             else:
                 self.settings.set('FEED_URI', opts.output, priority='cmdline')
-            feed_exporters = without_none_values(
-                self.settings.getwithbase('FEED_EXPORTERS'))
-            valid_output_formats = feed_exporters.keys()
+            valid_output_formats = (
+                list(self.settings.getdict('FEED_EXPORTERS').keys()) +
+                list(self.settings.getdict('FEED_EXPORTERS_BASE').keys())
+            )
             if not opts.output_format:
                 opts.output_format = os.path.splitext(opts.output)[1].replace(".", "")
             if opts.output_format not in valid_output_formats:
diff --git a/scrapy/commands/runspider.py b/scrapy/commands/runspider.py
index 1da09e4da..88f5a3015 100644
--- a/scrapy/commands/runspider.py
+++ b/scrapy/commands/runspider.py
@@ -6,7 +6,6 @@ from scrapy.utils.spider import iter_spider_classes
 from scrapy.commands import ScrapyCommand
 from scrapy.exceptions import UsageError
 from scrapy.utils.conf import arglist_to_dict
-from scrapy.utils.python import without_none_values
 
 
 def _import_file(filepath):
@@ -58,8 +57,10 @@ class Command(ScrapyCommand):
                 self.settings.set('FEED_URI', 'stdout:', priority='cmdline')
             else:
                 self.settings.set('FEED_URI', opts.output, priority='cmdline')
-            feed_exporters = without_none_values(self.settings.getwithbase('FEED_EXPORTERS'))
-            valid_output_formats = feed_exporters.keys()
+            valid_output_formats = (
+                list(self.settings.getdict('FEED_EXPORTERS').keys()) +
+                list(self.settings.getdict('FEED_EXPORTERS_BASE').keys())
+            )
             if not opts.output_format:
                 opts.output_format = os.path.splitext(opts.output)[1].replace(".", "")
             if opts.output_format not in valid_output_formats:
diff --git a/scrapy/core/downloader/handlers/__init__.py b/scrapy/core/downloader/handlers/__init__.py
index bc5cd742e..6c9514af6 100644
--- a/scrapy/core/downloader/handlers/__init__.py
+++ b/scrapy/core/downloader/handlers/__init__.py
@@ -6,7 +6,6 @@ import six
 from scrapy.exceptions import NotSupported, NotConfigured
 from scrapy.utils.httpobj import urlparse_cached
 from scrapy.utils.misc import load_object
-from scrapy.utils.python import without_none_values
 from scrapy import signals
 
 
@@ -20,9 +19,13 @@ class DownloadHandlers(object):
         self._schemes = {}  # stores acceptable schemes on instancing
         self._handlers = {}  # stores instanced handlers for schemes
         self._notconfigured = {}  # remembers failed handlers
-        handlers = without_none_values(
-            crawler.settings.getwithbase('DOWNLOAD_HANDLERS'))
+        handlers = crawler.settings.get('DOWNLOAD_HANDLERS_BASE')
+        handlers.update(crawler.settings.get('DOWNLOAD_HANDLERS', {}))
         for scheme, clspath in six.iteritems(handlers):
+            # Allow to disable a handler just like any other
+            # component (extension, middleware, etc).
+            if clspath is None:
+                continue
             self._schemes[scheme] = clspath
 
         crawler.signals.connect(self._close, signals.engine_stopped)
diff --git a/scrapy/core/downloader/middleware.py b/scrapy/core/downloader/middleware.py
index c3b23e284..9cd30c144 100644
--- a/scrapy/core/downloader/middleware.py
+++ b/scrapy/core/downloader/middleware.py
@@ -19,8 +19,8 @@ class DownloaderMiddlewareManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(
-            settings.getwithbase('DOWNLOADER_MIDDLEWARES'))
+        return build_component_list(settings['DOWNLOADER_MIDDLEWARES_BASE'], \
+            settings['DOWNLOADER_MIDDLEWARES'])
 
     def _add_middleware(self, mw):
         if hasattr(mw, 'process_request'):
diff --git a/scrapy/core/engine.py b/scrapy/core/engine.py
index eb2779b12..992327bfe 100644
--- a/scrapy/core/engine.py
+++ b/scrapy/core/engine.py
@@ -84,21 +84,6 @@ class ExecutionEngine(object):
         dfd = self._close_all_spiders()
         return dfd.addBoth(lambda _: self._finish_stopping_engine())
 
-    def close(self):
-        """Close the execution engine gracefully.
-
-        If it has already been started, stop it. In all cases, close all spiders
-        and the downloader.
-        """
-        if self.running:
-            # Will also close spiders and downloader
-            return self.stop()
-        elif self.open_spiders:
-            # Will also close downloader
-            return self._close_all_spiders()
-        else:
-            return defer.succeed(self.downloader.close())
-
     def pause(self):
         """Pause the execution engine"""
         self.paused = True
diff --git a/scrapy/core/spidermw.py b/scrapy/core/spidermw.py
index a206e4b0c..c1c5b10fc 100644
--- a/scrapy/core/spidermw.py
+++ b/scrapy/core/spidermw.py
@@ -18,7 +18,8 @@ class SpiderMiddlewareManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(settings.getwithbase('SPIDER_MIDDLEWARES'))
+        return build_component_list(settings['SPIDER_MIDDLEWARES_BASE'], \
+            settings['SPIDER_MIDDLEWARES'])
 
     def _add_middleware(self, mw):
         super(SpiderMiddlewareManager, self)._add_middleware(mw)
diff --git a/scrapy/crawler.py b/scrapy/crawler.py
index bdcfa9d0c..c7e3bb528 100644
--- a/scrapy/crawler.py
+++ b/scrapy/crawler.py
@@ -73,11 +73,8 @@ class Crawler(object):
             yield self.engine.open_spider(self.spider, start_requests)
             yield defer.maybeDeferred(self.engine.start)
         except Exception:
-            exc = defer.fail()
             self.crawling = False
-            if self.engine is not None:
-                yield self.engine.close()
-            yield exc
+            raise
 
     def _create_spider(self, *args, **kwargs):
         return self.spidercls.from_crawler(self, *args, **kwargs)
@@ -148,7 +145,9 @@ class CrawlerRunner(object):
 
         :param dict kwargs: keyword arguments to initialize the spider
         """
-        crawler = self.create_crawler(crawler_or_spidercls)
+        crawler = crawler_or_spidercls
+        if not isinstance(crawler_or_spidercls, Crawler):
+            crawler = self._create_crawler(crawler_or_spidercls)
         return self._crawl(crawler, *args, **kwargs)
 
     def _crawl(self, crawler, *args, **kwargs):
@@ -163,21 +162,6 @@ class CrawlerRunner(object):
 
         return d.addBoth(_done)
 
-    def create_crawler(self, crawler_or_spidercls):
-        """
-        Return a :class:`~scrapy.crawler.Crawler` object.
-
-        * If `crawler_or_spidercls` is a Crawler, it is returned as-is.
-        * If `crawler_or_spidercls` is a Spider subclass, a new Crawler
-          is constructed for it.
-        * If `crawler_or_spidercls` is a string, this function finds
-          a spider with this name in a Scrapy project (using spider loader),
-          then creates a Crawler instance for it.
-        """
-        if isinstance(crawler_or_spidercls, Crawler):
-            return crawler_or_spidercls
-        return self._create_crawler(crawler_or_spidercls)
-
     def _create_crawler(self, spidercls):
         if isinstance(spidercls, six.string_types):
             spidercls = self.spider_loader.load(spidercls)
diff --git a/scrapy/downloadermiddlewares/defaultheaders.py b/scrapy/downloadermiddlewares/defaultheaders.py
index 93fe97673..f1d2bd631 100644
--- a/scrapy/downloadermiddlewares/defaultheaders.py
+++ b/scrapy/downloadermiddlewares/defaultheaders.py
@@ -4,8 +4,6 @@ DefaultHeaders downloader middleware
 See documentation in docs/topics/downloader-middleware.rst
 """
 
-from scrapy.utils.python import without_none_values
-
 
 class DefaultHeadersMiddleware(object):
 
@@ -14,8 +12,7 @@ class DefaultHeadersMiddleware(object):
 
     @classmethod
     def from_crawler(cls, crawler):
-        headers = without_none_values(crawler.settings['DEFAULT_REQUEST_HEADERS'])
-        return cls(headers.items())
+        return cls(crawler.settings.get('DEFAULT_REQUEST_HEADERS').items())
 
     def process_request(self, request, spider):
         for k, v in self._headers:
diff --git a/scrapy/extension.py b/scrapy/extension.py
index e39e456fa..f68b1ba68 100644
--- a/scrapy/extension.py
+++ b/scrapy/extension.py
@@ -12,4 +12,5 @@ class ExtensionManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(settings.getwithbase('EXTENSIONS'))
+        return build_component_list(settings['EXTENSIONS_BASE'], \
+            settings['EXTENSIONS'])
diff --git a/scrapy/extensions/feedexport.py b/scrapy/extensions/feedexport.py
index daea551cb..7560e89d3 100644
--- a/scrapy/extensions/feedexport.py
+++ b/scrapy/extensions/feedexport.py
@@ -22,7 +22,6 @@ from scrapy.utils.ftp import ftp_makedirs_cwd
 from scrapy.exceptions import NotConfigured
 from scrapy.utils.misc import load_object
 from scrapy.utils.log import failure_to_exc_info
-from scrapy.utils.python import without_none_values
 
 logger = logging.getLogger(__name__)
 
@@ -196,7 +195,8 @@ class FeedExporter(object):
         return item
 
     def _load_components(self, setting_prefix):
-        conf = without_none_values(self.settings.getwithbase(setting_prefix))
+        conf = dict(self.settings['%s_BASE' % setting_prefix])
+        conf.update(self.settings[setting_prefix])
         d = {}
         for k, v in conf.items():
             try:
diff --git a/scrapy/extensions/telnet.py b/scrapy/extensions/telnet.py
deleted file mode 100644
index d9add1d97..000000000
--- a/scrapy/extensions/telnet.py
+++ /dev/null
@@ -1,88 +0,0 @@
-"""
-Scrapy Telnet Console extension
-
-See documentation in docs/topics/telnetconsole.rst
-"""
-
-import pprint
-import logging
-
-from twisted.internet import protocol
-try:
-    from twisted.conch import manhole, telnet
-    from twisted.conch.insults import insults
-    TWISTED_CONCH_AVAILABLE = True
-except ImportError:
-    TWISTED_CONCH_AVAILABLE = False
-
-from scrapy.exceptions import NotConfigured
-from scrapy import signals
-from scrapy.utils.trackref import print_live_refs
-from scrapy.utils.engine import print_engine_status
-from scrapy.utils.reactor import listen_tcp
-
-try:
-    import guppy
-    hpy = guppy.hpy()
-except ImportError:
-    hpy = None
-
-logger = logging.getLogger(__name__)
-
-# signal to update telnet variables
-# args: telnet_vars
-update_telnet_vars = object()
-
-
-class TelnetConsole(protocol.ServerFactory):
-
-    def __init__(self, crawler):
-        if not crawler.settings.getbool('TELNETCONSOLE_ENABLED'):
-            raise NotConfigured
-        if not TWISTED_CONCH_AVAILABLE:
-            raise NotConfigured
-        self.crawler = crawler
-        self.noisy = False
-        self.portrange = [int(x) for x in crawler.settings.getlist('TELNETCONSOLE_PORT')]
-        self.host = crawler.settings['TELNETCONSOLE_HOST']
-        self.crawler.signals.connect(self.start_listening, signals.engine_started)
-        self.crawler.signals.connect(self.stop_listening, signals.engine_stopped)
-
-    @classmethod
-    def from_crawler(cls, crawler):
-        return cls(crawler)
-
-    def start_listening(self):
-        self.port = listen_tcp(self.portrange, self.host, self)
-        h = self.port.getHost()
-        logger.debug("Telnet console listening on %(host)s:%(port)d",
-                     {'host': h.host, 'port': h.port},
-                     extra={'crawler': self.crawler})
-
-    def stop_listening(self):
-        self.port.stopListening()
-
-    def protocol(self):
-        telnet_vars = self._get_telnet_vars()
-        return telnet.TelnetTransport(telnet.TelnetBootstrapProtocol,
-            insults.ServerProtocol, manhole.Manhole, telnet_vars)
-
-    def _get_telnet_vars(self):
-        # Note: if you add entries here also update topics/telnetconsole.rst
-        telnet_vars = {
-            'engine': self.crawler.engine,
-            'spider': self.crawler.engine.spider,
-            'slot': self.crawler.engine.slot,
-            'crawler': self.crawler,
-            'extensions': self.crawler.extensions,
-            'stats': self.crawler.stats,
-            'settings': self.crawler.settings,
-            'est': lambda: print_engine_status(self.crawler.engine),
-            'p': pprint.pprint,
-            'prefs': print_live_refs,
-            'hpy': hpy,
-            'help': "This is Scrapy telnet console. For more info see: " \
-                "http://doc.scrapy.org/en/latest/topics/telnetconsole.html",
-        }
-        self.crawler.signals.send_catch_log(update_telnet_vars, telnet_vars=telnet_vars)
-        return telnet_vars
diff --git a/scrapy/http/request/form.py b/scrapy/http/request/form.py
index 4a9bd732e..a12a2fd07 100644
--- a/scrapy/http/request/form.py
+++ b/scrapy/http/request/form.py
@@ -11,7 +11,6 @@ from parsel.selector import create_root_node
 import six
 from scrapy.http.request import Request
 from scrapy.utils.python import to_bytes, is_listlike
-from scrapy.utils.response import get_base_url
 
 
 class FormRequest(Request):
@@ -45,7 +44,7 @@ class FormRequest(Request):
 
 def _get_form_url(form, url):
     if url is None:
-        return urljoin(form.base_url, form.action)
+        return form.action or form.base_url
     return urljoin(form.base_url, url)
 
 
@@ -59,7 +58,7 @@ def _urlencode(seq, enc):
 def _get_form(response, formname, formid, formnumber, formxpath):
     """Find the form element """
     text = response.body_as_unicode()
-    root = create_root_node(text, lxml.html.HTMLParser, base_url=get_base_url(response))
+    root = create_root_node(text, lxml.html.HTMLParser, base_url=response.url)
     forms = root.xpath('//form')
     if not forms:
         raise ValueError("No <form> element found in %s" % response)
diff --git a/scrapy/pipelines/__init__.py b/scrapy/pipelines/__init__.py
index 2ef8786d0..d433498f5 100644
--- a/scrapy/pipelines/__init__.py
+++ b/scrapy/pipelines/__init__.py
@@ -13,7 +13,15 @@ class ItemPipelineManager(MiddlewareManager):
 
     @classmethod
     def _get_mwlist_from_settings(cls, settings):
-        return build_component_list(settings.getwithbase('ITEM_PIPELINES'))
+        item_pipelines = settings['ITEM_PIPELINES']
+        if isinstance(item_pipelines, (tuple, list, set, frozenset)):
+            from scrapy.exceptions import ScrapyDeprecationWarning
+            import warnings
+            warnings.warn('ITEM_PIPELINES defined as a list or a set is deprecated, switch to a dict',
+                category=ScrapyDeprecationWarning, stacklevel=1)
+            # convert old ITEM_PIPELINE list to a dict with order 500
+            item_pipelines = dict(zip(item_pipelines, range(500, 500+len(item_pipelines))))
+        return build_component_list(settings['ITEM_PIPELINES_BASE'], item_pipelines)
 
     def _add_middleware(self, pipe):
         super(ItemPipelineManager, self)._add_middleware(pipe)
diff --git a/scrapy/settings/__init__.py b/scrapy/settings/__init__.py
index 342d2585e..af0d0dff1 100644
--- a/scrapy/settings/__init__.py
+++ b/scrapy/settings/__init__.py
@@ -20,18 +20,6 @@ SETTINGS_PRIORITIES = {
 }
 
 
-def get_settings_priority(priority):
-    """
-    Small helper function that looks up a given string priority in the
-    :attr:`~scrapy.settings.SETTINGS_PRIORITIES` dictionary and returns its
-    numerical value, or directly returns a given numerical priority.
-    """
-    if isinstance(priority, six.string_types):
-        return SETTINGS_PRIORITIES[priority]
-    else:
-        return priority
-
-
 class SettingsAttribute(object):
 
     """Class for storing data related to settings attributes.
@@ -42,16 +30,11 @@ class SettingsAttribute(object):
 
     def __init__(self, value, priority):
         self.value = value
-        if isinstance(self.value, BaseSettings):
-            self.priority = max(self.value.maxpriority(), priority)
-        else:
-            self.priority = priority
+        self.priority = priority
 
     def set(self, value, priority):
         """Sets value if priority is higher or equal than current priority."""
         if priority >= self.priority:
-            if isinstance(self.value, BaseSettings):
-                value = BaseSettings(value, priority=priority)
             self.value = value
             self.priority = priority
 
@@ -62,220 +45,64 @@ class SettingsAttribute(object):
     __repr__ = __str__
 
 
-class BaseSettings(MutableMapping):
-    """
-    Instances of this class behave like dictionaries, but store priorities
-    along with their ``(key, value)`` pairs, and can be frozen (i.e. marked
-    immutable).
-
-    Key-value entries can be passed on initialization with the ``values``
-    argument, and they would take the ``priority`` level (unless ``values`` is
-    already an instance of :class:`~scrapy.settings.BaseSettings`, in which
-    case the existing priority levels will be kept).  If the ``priority``
-    argument is a string, the priority name will be looked up in
-    :attr:`~scrapy.settings.SETTINGS_PRIORITIES`. Otherwise, a specific integer
-    should be provided.
-
-    Once the object is created, new settings can be loaded or updated with the
-    :meth:`~scrapy.settings.BaseSettings.set` method, and can be accessed with
-    the square bracket notation of dictionaries, or with the
-    :meth:`~scrapy.settings.BaseSettings.get` method of the instance and its
-    value conversion variants. When requesting a stored key, the value with the
-    highest priority will be retrieved.
-    """
+class Settings(object):
 
     def __init__(self, values=None, priority='project'):
         self.frozen = False
         self.attributes = {}
-        self.update(values, priority)
+        self.setmodule(default_settings, priority='default')
+        if values is not None:
+            self.setdict(values, priority)
 
     def __getitem__(self, opt_name):
-        if opt_name not in self:
-            return None
-        return self.attributes[opt_name].value
-
-    def __contains__(self, name):
-        return name in self.attributes
+        value = None
+        if opt_name in self.attributes:
+            value = self.attributes[opt_name].value
+        return value
 
     def get(self, name, default=None):
-        """
-        Get a setting value without affecting its original type.
-
-        :param name: the setting name
-        :type name: string
-
-        :param default: the value to return if no setting is found
-        :type default: any
-        """
         return self[name] if self[name] is not None else default
 
     def getbool(self, name, default=False):
         """
-        Get a setting value as a boolean.
-
-        ``1``, ``'1'``, and ``True`` return ``True``, while ``0``, ``'0'``,
-        ``False`` and ``None`` return ``False``.
-
-        For example, settings populated through environment variables set to
-        ``'0'`` will return ``False`` when using this method.
-
-        :param name: the setting name
-        :type name: string
-
-        :param default: the value to return if no setting is found
-        :type default: any
+        True is: 1, '1', True
+        False is: 0, '0', False, None
         """
         return bool(int(self.get(name, default)))
 
     def getint(self, name, default=0):
-        """
-        Get a setting value as an int.
-
-        :param name: the setting name
-        :type name: string
-
-        :param default: the value to return if no setting is found
-        :type default: any
-        """
         return int(self.get(name, default))
 
     def getfloat(self, name, default=0.0):
-        """
-        Get a setting value as a float.
-
-        :param name: the setting name
-        :type name: string
-
-        :param default: the value to return if no setting is found
-        :type default: any
-        """
         return float(self.get(name, default))
 
     def getlist(self, name, default=None):
-        """
-        Get a setting value as a list. If the setting original type is a list, a
-        copy of it will be returned. If it's a string it will be split by ",".
-
-        For example, settings populated through environment variables set to
-        ``'one,two'`` will return a list ['one', 'two'] when using this method.
-
-        :param name: the setting name
-        :type name: string
-
-        :param default: the value to return if no setting is found
-        :type default: any
-        """
         value = self.get(name, default or [])
         if isinstance(value, six.string_types):
             value = value.split(',')
         return list(value)
 
     def getdict(self, name, default=None):
-        """
-        Get a setting value as a dictionary. If the setting original type is a
-        dictionary, a copy of it will be returned. If it is a string it will be
-        evaluated as a JSON dictionary. In the case that it is a
-        :class:`~scrapy.settings.BaseSettings` instance itself, it will be
-        converted to a dictionary, containing all its current settings values
-        as they would be returned by :meth:`~scrapy.settings.BaseSettings.get`,
-        and losing all information about priority and mutability.
-
-        :param name: the setting name
-        :type name: string
-
-        :param default: the value to return if no setting is found
-        :type default: any
-        """
         value = self.get(name, default or {})
         if isinstance(value, six.string_types):
             value = json.loads(value)
         return dict(value)
 
-    def getwithbase(self, name):
-        """Get a composition of a dictionary-like setting and its `_BASE`
-        counterpart.
-
-        :param name: name of the dictionary-like setting
-        :type name: string
-        """
-        compbs = BaseSettings()
-        compbs.update(self[name + '_BASE'])
-        compbs.update(self[name])
-        return compbs
-
-    def getpriority(self, name):
-        """
-        Return the current numerical priority value of a setting, or ``None`` if
-        the given ``name`` does not exist.
-
-        :param name: the setting name
-        :type name: string
-        """
-        if name not in self:
-            return None
-        return self.attributes[name].priority
-
-    def maxpriority(self):
-        """
-        Return the numerical value of the highest priority present throughout
-        all settings, or the numerical value for ``default`` from
-        :attr:`~scrapy.settings.SETTINGS_PRIORITIES` if there are no settings
-        stored.
-        """
-        if len(self) > 0:
-            return max(self.getpriority(name) for name in self)
-        else:
-            return get_settings_priority('default')
-
-    def __setitem__(self, name, value):
-        self.set(name, value)
-
     def set(self, name, value, priority='project'):
-        """
-        Store a key/value attribute with a given priority.
-
-        Settings should be populated *before* configuring the Crawler object
-        (through the :meth:`~scrapy.crawler.Crawler.configure` method),
-        otherwise they won't have any effect.
-
-        :param name: the setting name
-        :type name: string
-
-        :param value: the value to associate with the setting
-        :type value: any
-
-        :param priority: the priority of the setting. Should be a key of
-            :attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
-        :type priority: string or int
-        """
         self._assert_mutability()
-        priority = get_settings_priority(priority)
-        if name not in self:
-            if isinstance(value, SettingsAttribute):
-                self.attributes[name] = value
-            else:
-                self.attributes[name] = SettingsAttribute(value, priority)
+        if isinstance(priority, six.string_types):
+            priority = SETTINGS_PRIORITIES[priority]
+        if name not in self.attributes:
+            self.attributes[name] = SettingsAttribute(value, priority)
         else:
             self.attributes[name].set(value, priority)
 
     def setdict(self, values, priority='project'):
-        self.update(values, priority)
+        self._assert_mutability()
+        for name, value in six.iteritems(values):
+            self.set(name, value, priority)
 
     def setmodule(self, module, priority='project'):
-        """
-        Store settings from a module with a given priority.
-
-        This is a helper function that calls
-        :meth:`~scrapy.settings.BaseSettings.set` for every globally declared
-        uppercase variable of ``module`` with the provided ``priority``.
-
-        :param module: the module or the path of the module
-        :type module: module object or string
-
-        :param priority: the priority of the settings. Should be a key of
-            :attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
-        :type priority: string or int
-        """
         self._assert_mutability()
         if isinstance(module, six.string_types):
             module = import_module(module)
@@ -283,97 +110,21 @@ class BaseSettings(MutableMapping):
             if key.isupper():
                 self.set(key, getattr(module, key), priority)
 
-    def update(self, values, priority='project'):
-        """
-        Store key/value pairs with a given priority.
-
-        This is a helper function that calls
-        :meth:`~scrapy.settings.BaseSettings.set` for every item of ``values``
-        with the provided ``priority``.
-
-        If ``values`` is a string, it is assumed to be JSON-encoded and parsed
-        into a dict with ``json.loads()`` first. If it is a
-        :class:`~scrapy.settings.BaseSettings` instance, the per-key priorities
-        will be used and the ``priority`` parameter ignored. This allows
-        inserting/updating settings with different priorities with a single
-        command.
-
-        :param values: the settings names and values
-        :type values: dict or string or :class:`~scrapy.settings.BaseSettings`
-
-        :param priority: the priority of the settings. Should be a key of
-            :attr:`~scrapy.settings.SETTINGS_PRIORITIES` or an integer
-        :type priority: string or int
-        """
-        self._assert_mutability()
-        if isinstance(values, six.string_types):
-            values = json.loads(values)
-        if values is not None:
-            if isinstance(values, BaseSettings):
-                for name, value in six.iteritems(values):
-                    self.set(name, value, values.getpriority(name))
-            else:
-                for name, value in six.iteritems(values):
-                    self.set(name, value, priority)
-
-    def delete(self, name, priority='project'):
-        self._assert_mutability()
-        priority = get_settings_priority(priority)
-        if priority >= self.getpriority(name):
-            del self.attributes[name]
-
-    def __delitem__(self, name):
-        self._assert_mutability()
-        del self.attributes[name]
-
     def _assert_mutability(self):
         if self.frozen:
             raise TypeError("Trying to modify an immutable Settings object")
 
     def copy(self):
-        """
-        Make a deep copy of current settings.
-
-        This method returns a new instance of the :class:`Settings` class,
-        populated with the same values and their priorities.
-
-        Modifications to the new object won't be reflected on the original
-        settings.
-        """
         return copy.deepcopy(self)
 
     def freeze(self):
-        """
-        Disable further changes to the current settings.
-
-        After calling this method, the present state of the settings will become
-        immutable. Trying to change values through the :meth:`~set` method and
-        its variants won't be possible and will be alerted.
-        """
         self.frozen = True
 
     def frozencopy(self):
-        """
-        Return an immutable copy of the current settings.
-
-        Alias for a :meth:`~freeze` call in the object returned by :meth:`copy`.
-        """
         copy = self.copy()
         copy.freeze()
         return copy
 
-    def __iter__(self):
-        return iter(self.attributes)
-
-    def __len__(self):
-        return len(self.attributes)
-
-    def __str__(self):
-        return str(self.attributes)
-
-    def __repr__(self):
-        return "<%s %s>" % (self.__class__.__name__, self.attributes)
-
     @property
     def overrides(self):
         warnings.warn("`Settings.overrides` attribute is deprecated and won't "
@@ -423,36 +174,11 @@ class _DictProxy(MutableMapping):
         return iter(self.o)
 
 
-class Settings(BaseSettings):
-    """
-    This object stores Scrapy settings for the configuration of internal
-    components, and can be used for any further customization.
-
-    It is a direct subclass and supports all methods of
-    :class:`~scrapy.settings.BaseSettings`. Additionally, after instantiation
-    of this class, the new object will have the global default settings
-    described on :ref:`topics-settings-ref` already populated.
-    """
-
-    def __init__(self, values=None, priority='project'):
-        # Do not pass kwarg values here. We don't want to promote user-defined
-        # dicts, and we want to update, not replace, default dicts with the
-        # values given by the user
-        super(Settings, self).__init__()
-        self.setmodule(default_settings, 'default')
-        # Promote default dictionaries to BaseSettings instances for per-key
-        # priorities
-        for name, val in six.iteritems(self):
-            if isinstance(val, dict):
-                self.set(name, BaseSettings(val, 'default'), 'default')
-        self.update(values, priority)
-
-
 class CrawlerSettings(Settings):
 
     def __init__(self, settings_module=None, **kw):
-        self.settings_module = settings_module
         Settings.__init__(self, **kw)
+        self.settings_module = settings_module
 
     def __getitem__(self, opt_name):
         if opt_name in self.overrides:
@@ -477,7 +203,6 @@ def iter_default_settings():
         if name.isupper():
             yield name, getattr(default_settings, name)
 
-
 def overridden_settings(settings):
     """Return a dict of the settings that have been overridden"""
     for name, defvalue in iter_default_settings():
diff --git a/scrapy/settings/default_settings.py b/scrapy/settings/default_settings.py
index 8435b0354..5d48ac1a7 100644
--- a/scrapy/settings/default_settings.py
+++ b/scrapy/settings/default_settings.py
@@ -120,7 +120,7 @@ EXTENSIONS = {}
 
 EXTENSIONS_BASE = {
     'scrapy.extensions.corestats.CoreStats': 0,
-    'scrapy.extensions.telnet.TelnetConsole': 0,
+    'scrapy.telnet.TelnetConsole': 0,
     'scrapy.extensions.memusage.MemoryUsage': 0,
     'scrapy.extensions.memdebug.MemoryDebugger': 0,
     'scrapy.extensions.closespider.CloseSpider': 0,
diff --git a/scrapy/telnet.py b/scrapy/telnet.py
index 70915f486..d9add1d97 100644
--- a/scrapy/telnet.py
+++ b/scrapy/telnet.py
@@ -1,7 +1,88 @@
-import warnings
-from scrapy.exceptions import ScrapyDeprecationWarning
-warnings.warn("Module `scrapy.telnet` is deprecated, "
-              "use `scrapy.extensions.telnet` instead",
-              ScrapyDeprecationWarning, stacklevel=2)
+"""
+Scrapy Telnet Console extension
 
-from scrapy.extensions.telnet import *
+See documentation in docs/topics/telnetconsole.rst
+"""
+
+import pprint
+import logging
+
+from twisted.internet import protocol
+try:
+    from twisted.conch import manhole, telnet
+    from twisted.conch.insults import insults
+    TWISTED_CONCH_AVAILABLE = True
+except ImportError:
+    TWISTED_CONCH_AVAILABLE = False
+
+from scrapy.exceptions import NotConfigured
+from scrapy import signals
+from scrapy.utils.trackref import print_live_refs
+from scrapy.utils.engine import print_engine_status
+from scrapy.utils.reactor import listen_tcp
+
+try:
+    import guppy
+    hpy = guppy.hpy()
+except ImportError:
+    hpy = None
+
+logger = logging.getLogger(__name__)
+
+# signal to update telnet variables
+# args: telnet_vars
+update_telnet_vars = object()
+
+
+class TelnetConsole(protocol.ServerFactory):
+
+    def __init__(self, crawler):
+        if not crawler.settings.getbool('TELNETCONSOLE_ENABLED'):
+            raise NotConfigured
+        if not TWISTED_CONCH_AVAILABLE:
+            raise NotConfigured
+        self.crawler = crawler
+        self.noisy = False
+        self.portrange = [int(x) for x in crawler.settings.getlist('TELNETCONSOLE_PORT')]
+        self.host = crawler.settings['TELNETCONSOLE_HOST']
+        self.crawler.signals.connect(self.start_listening, signals.engine_started)
+        self.crawler.signals.connect(self.stop_listening, signals.engine_stopped)
+
+    @classmethod
+    def from_crawler(cls, crawler):
+        return cls(crawler)
+
+    def start_listening(self):
+        self.port = listen_tcp(self.portrange, self.host, self)
+        h = self.port.getHost()
+        logger.debug("Telnet console listening on %(host)s:%(port)d",
+                     {'host': h.host, 'port': h.port},
+                     extra={'crawler': self.crawler})
+
+    def stop_listening(self):
+        self.port.stopListening()
+
+    def protocol(self):
+        telnet_vars = self._get_telnet_vars()
+        return telnet.TelnetTransport(telnet.TelnetBootstrapProtocol,
+            insults.ServerProtocol, manhole.Manhole, telnet_vars)
+
+    def _get_telnet_vars(self):
+        # Note: if you add entries here also update topics/telnetconsole.rst
+        telnet_vars = {
+            'engine': self.crawler.engine,
+            'spider': self.crawler.engine.spider,
+            'slot': self.crawler.engine.slot,
+            'crawler': self.crawler,
+            'extensions': self.crawler.extensions,
+            'stats': self.crawler.stats,
+            'settings': self.crawler.settings,
+            'est': lambda: print_engine_status(self.crawler.engine),
+            'p': pprint.pprint,
+            'prefs': print_live_refs,
+            'hpy': hpy,
+            'help': "This is Scrapy telnet console. For more info see: " \
+                "http://doc.scrapy.org/en/latest/topics/telnetconsole.html",
+        }
+        self.crawler.signals.send_catch_log(update_telnet_vars, telnet_vars=telnet_vars)
+        return telnet_vars
diff --git a/scrapy/templates/project/module/settings.py.tmpl b/scrapy/templates/project/module/settings.py.tmpl
index 822812c9a..2e660d7cf 100644
--- a/scrapy/templates/project/module/settings.py.tmpl
+++ b/scrapy/templates/project/module/settings.py.tmpl
@@ -56,7 +56,7 @@ NEWSPIDER_MODULE = '$project_name.spiders'
 # Enable or disable extensions
 # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html
 #EXTENSIONS = {
-#    'scrapy.extensions.telnet.TelnetConsole': None,
+#    'scrapy.telnet.TelnetConsole': None,
 #}
 
 # Configure item pipelines
diff --git a/scrapy/utils/conf.py b/scrapy/utils/conf.py
index e8af90f11..423ca43e2 100644
--- a/scrapy/utils/conf.py
+++ b/scrapy/utils/conf.py
@@ -5,46 +5,32 @@ from operator import itemgetter
 import six
 from six.moves.configparser import SafeConfigParser
 
-from scrapy.settings import BaseSettings
 from scrapy.utils.deprecate import update_classpath
-from scrapy.utils.python import without_none_values
 
 
-def build_component_list(compdict, custom=None, convert=update_classpath):
-    """Compose a component list from a { class: order } dictionary."""
+def build_component_list(base, custom, convert=update_classpath):
+    """Compose a component list based on a custom and base dict of components
+    (typically middlewares or extensions), unless custom is already a list, in
+    which case it's returned.
+    """
 
     def _check_components(complist):
         if len({convert(c) for c in complist}) != len(complist):
             raise ValueError('Some paths in {!r} convert to the same object, '
                              'please update your settings'.format(complist))
 
-    def _map_keys(compdict):
-        if isinstance(compdict, BaseSettings):
-            compbs = BaseSettings()
-            for k, v in six.iteritems(compdict):
-                prio = compdict.getpriority(k)
-                if compbs.getpriority(convert(k)) == prio:
-                    raise ValueError('Some paths in {!r} convert to the same '
-                                     'object, please update your settings'
-                                     ''.format(list(compdict.keys())))
-                else:
-                    compbs.set(convert(k), v, priority=prio)
-            return compbs
-        else:
-            _check_components(compdict)
-            return {convert(k): v for k, v in six.iteritems(compdict)}
-
-    # BEGIN Backwards compatibility for old (base, custom) call signature
     if isinstance(custom, (list, tuple)):
         _check_components(custom)
         return type(custom)(convert(c) for c in custom)
 
-    if custom is not None:
-        compdict.update(custom)
-    # END Backwards compatibility
+    def _map_keys(compdict):
+        _check_components(compdict)
+        return {convert(k): v for k, v in six.iteritems(compdict)}
 
-    compdict = without_none_values(_map_keys(compdict))
-    return [k for k, v in sorted(six.iteritems(compdict), key=itemgetter(1))]
+    compdict = _map_keys(base)
+    compdict.update(_map_keys(custom))
+    items = (x for x in six.iteritems(compdict) if x[1] is not None)
+    return [x[0] for x in sorted(items, key=itemgetter(1))]
 
 
 def arglist_to_dict(arglist):
diff --git a/scrapy/utils/deprecate.py b/scrapy/utils/deprecate.py
index 0fc33e0c4..37e94fae9 100644
--- a/scrapy/utils/deprecate.py
+++ b/scrapy/utils/deprecate.py
@@ -138,7 +138,6 @@ DEPRECATION_RULES = [
     ('scrapy.command.', 'scrapy.commands.'),
     ('scrapy.dupefilter.', 'scrapy.dupefilters.'),
     ('scrapy.linkextractor.', 'scrapy.linkextractors.'),
-    ('scrapy.telnet.', 'scrapy.extensions.telnet.'),
     ('scrapy.spider.', 'scrapy.spiders.'),
     ('scrapy.squeue.', 'scrapy.squeues.'),
     ('scrapy.statscol.', 'scrapy.statscollectors.'),
diff --git a/scrapy/utils/python.py b/scrapy/utils/python.py
index 1f9d02df5..d566783b2 100644
--- a/scrapy/utils/python.py
+++ b/scrapy/utils/python.py
@@ -324,15 +324,3 @@ def retry_on_eintr(function, *args, **kw):
         except IOError as e:
             if e.errno != errno.EINTR:
                 raise
-
-
-def without_none_values(iterable):
-    """Return a copy of `iterable` with all `None` entries removed.
-
-    If `iterable` is a mapping, return a dictionary where all pairs that have
-    value `None` have been removed.
-    """
-    try:
-        return {k: v for k, v in six.iteritems(iterable) if v is not None}
-    except AttributeError:
-        return type(iterable)((v for v in iterable if v is not None))


Test patch:
diff --git a/scrapy/utils/test.py b/scrapy/utils/test.py
index 51edfd353..bec9bdda9 100644
--- a/scrapy/utils/test.py
+++ b/scrapy/utils/test.py
@@ -26,10 +26,11 @@ def get_crawler(spidercls=None, settings_dict=None):
     priority.
     """
     from scrapy.crawler import CrawlerRunner
+    from scrapy.settings import Settings
     from scrapy.spiders import Spider
 
-    runner = CrawlerRunner(settings_dict)
-    return runner.create_crawler(spidercls or Spider)
+    runner = CrawlerRunner(Settings(settings_dict))
+    return runner._create_crawler(spidercls or Spider)
 
 def get_pythonpath():
     """Return a PYTHONPATH suitable to use in processes so that they find this
diff --git a/tests/test_cmdline/__init__.py b/tests/test_cmdline/__init__.py
index c2de4fbc8..1e2905e95 100644
--- a/tests/test_cmdline/__init__.py
+++ b/tests/test_cmdline/__init__.py
@@ -1,11 +1,9 @@
-import json
 import os
-import pstats
-import shutil
-import six
-from subprocess import Popen, PIPE
 import sys
+import shutil
+import pstats
 import tempfile
+from subprocess import Popen, PIPE
 import unittest
 try:
     from cStringIO import StringIO
@@ -56,16 +54,3 @@ class CmdlineTest(unittest.TestCase):
             self.assertIn('tottime', stats)
         finally:
             shutil.rmtree(path)
-
-    def test_override_dict_settings(self):
-        EXT_PATH = "tests.test_cmdline.extensions.DummyExtension"
-        EXTENSIONS = {EXT_PATH: 200}
-        settingsstr = self._execute('settings', '--get', 'EXTENSIONS', '-s',
-                                    'EXTENSIONS=' + json.dumps(EXTENSIONS))
-        # XXX: There's gotta be a smarter way to do this...
-        self.assertNotIn("...", settingsstr)
-        for char in ("'", "<", ">", 'u"'):
-            settingsstr = settingsstr.replace(char, '"')
-        settingsdict = json.loads(settingsstr)
-        six.assertCountEqual(self, settingsdict.keys(), EXTENSIONS.keys())
-        self.assertIn('value=200', settingsdict[EXT_PATH])
diff --git a/tests/test_cmdline/extensions.py b/tests/test_cmdline/extensions.py
index 72867eb56..4d347966a 100644
--- a/tests/test_cmdline/extensions.py
+++ b/tests/test_cmdline/extensions.py
@@ -8,8 +8,3 @@ class TestExtension(object):
     @classmethod
     def from_crawler(cls, crawler):
         return cls(crawler.settings)
-
-
-class DummyExtension(object):
-    pass
-
diff --git a/tests/test_crawl.py b/tests/test_crawl.py
index 814eb30d2..6d21acab0 100644
--- a/tests/test_crawl.py
+++ b/tests/test_crawl.py
@@ -6,12 +6,12 @@ from testfixtures import LogCapture
 from twisted.internet import defer
 from twisted.trial.unittest import TestCase
 
-from scrapy.http import Request
-from scrapy.crawler import CrawlerRunner
+from scrapy.utils.test import get_crawler
 from tests import mock
 from tests.spiders import FollowAllSpider, DelaySpider, SimpleSpider, \
     BrokenStartRequestsSpider, SingleRequestSpider, DuplicateStartRequestsSpider
 from tests.mockserver import MockServer
+from scrapy.http import Request
 
 
 class CrawlTestCase(TestCase):
@@ -19,14 +19,13 @@ class CrawlTestCase(TestCase):
     def setUp(self):
         self.mockserver = MockServer()
         self.mockserver.__enter__()
-        self.runner = CrawlerRunner()
 
     def tearDown(self):
         self.mockserver.__exit__(None, None, None)
 
     @defer.inlineCallbacks
     def test_follow_all(self):
-        crawler = self.runner.create_crawler(FollowAllSpider)
+        crawler = get_crawler(FollowAllSpider)
         yield crawler.crawl()
         self.assertEqual(len(crawler.spider.urls_visited), 11)  # 10 + start_url
 
@@ -42,7 +41,7 @@ class CrawlTestCase(TestCase):
     @defer.inlineCallbacks
     def _test_delay(self, delay, randomize):
         settings = {"DOWNLOAD_DELAY": delay, 'RANDOMIZE_DOWNLOAD_DELAY': randomize}
-        crawler = CrawlerRunner(settings).create_crawler(FollowAllSpider)
+        crawler = get_crawler(FollowAllSpider, settings)
         yield crawler.crawl(maxlatency=delay * 2)
         t = crawler.spider.times
         totaltime = t[-1] - t[0]
@@ -53,7 +52,7 @@ class CrawlTestCase(TestCase):
 
     @defer.inlineCallbacks
     def test_timeout_success(self):
-        crawler = self.runner.create_crawler(DelaySpider)
+        crawler = get_crawler(DelaySpider)
         yield crawler.crawl(n=0.5)
         self.assertTrue(crawler.spider.t1 > 0)
         self.assertTrue(crawler.spider.t2 > 0)
@@ -61,7 +60,7 @@ class CrawlTestCase(TestCase):
 
     @defer.inlineCallbacks
     def test_timeout_failure(self):
-        crawler = CrawlerRunner({"DOWNLOAD_TIMEOUT": 0.35}).create_crawler(DelaySpider)
+        crawler = get_crawler(DelaySpider, {"DOWNLOAD_TIMEOUT": 0.35})
         yield crawler.crawl(n=0.5)
         self.assertTrue(crawler.spider.t1 > 0)
         self.assertTrue(crawler.spider.t2 == 0)
@@ -76,14 +75,14 @@ class CrawlTestCase(TestCase):
 
     @defer.inlineCallbacks
     def test_retry_503(self):
-        crawler = self.runner.create_crawler(SimpleSpider)
+        crawler = get_crawler(SimpleSpider)
         with LogCapture() as l:
             yield crawler.crawl("http://localhost:8998/status?n=503")
         self._assert_retried(l)
 
     @defer.inlineCallbacks
     def test_retry_conn_failed(self):
-        crawler = self.runner.create_crawler(SimpleSpider)
+        crawler = get_crawler(SimpleSpider)
         with LogCapture() as l:
             yield crawler.crawl("http://localhost:65432/status?n=503")
         self._assert_retried(l)
@@ -92,7 +91,7 @@ class CrawlTestCase(TestCase):
     def test_retry_dns_error(self):
         with mock.patch('socket.gethostbyname',
                         side_effect=socket.gaierror(-5, 'No address associated with hostname')):
-            crawler = self.runner.create_crawler(SimpleSpider)
+            crawler = get_crawler(SimpleSpider)
             with LogCapture() as l:
                 yield crawler.crawl("http://example.com/")
             self._assert_retried(l)
@@ -100,7 +99,7 @@ class CrawlTestCase(TestCase):
     @defer.inlineCallbacks
     def test_start_requests_bug_before_yield(self):
         with LogCapture('scrapy', level=logging.ERROR) as l:
-            crawler = self.runner.create_crawler(BrokenStartRequestsSpider)
+            crawler = get_crawler(BrokenStartRequestsSpider)
             yield crawler.crawl(fail_before_yield=1)
 
         self.assertEqual(len(l.records), 1)
@@ -111,7 +110,7 @@ class CrawlTestCase(TestCase):
     @defer.inlineCallbacks
     def test_start_requests_bug_yielding(self):
         with LogCapture('scrapy', level=logging.ERROR) as l:
-            crawler = self.runner.create_crawler(BrokenStartRequestsSpider)
+            crawler = get_crawler(BrokenStartRequestsSpider)
             yield crawler.crawl(fail_yielding=1)
 
         self.assertEqual(len(l.records), 1)
@@ -122,7 +121,7 @@ class CrawlTestCase(TestCase):
     @defer.inlineCallbacks
     def test_start_requests_lazyness(self):
         settings = {"CONCURRENT_REQUESTS": 1}
-        crawler = CrawlerRunner(settings).create_crawler(BrokenStartRequestsSpider)
+        crawler = get_crawler(BrokenStartRequestsSpider, settings)
         yield crawler.crawl()
         #self.assertTrue(False, crawler.spider.seedsseen)
         #self.assertTrue(crawler.spider.seedsseen.index(None) < crawler.spider.seedsseen.index(99),
@@ -131,7 +130,7 @@ class CrawlTestCase(TestCase):
     @defer.inlineCallbacks
     def test_start_requests_dupes(self):
         settings = {"CONCURRENT_REQUESTS": 1}
-        crawler = CrawlerRunner(settings).create_crawler(DuplicateStartRequestsSpider)
+        crawler = get_crawler(DuplicateStartRequestsSpider, settings)
         yield crawler.crawl(dont_filter=True, distinct_urls=2, dupe_factor=3)
         self.assertEqual(crawler.spider.visited, 6)
 
@@ -160,7 +159,7 @@ Connection: close
 foo body
 with multiples lines
 '''})
-        crawler = self.runner.create_crawler(SimpleSpider)
+        crawler = get_crawler(SimpleSpider)
         with LogCapture() as l:
             yield crawler.crawl("http://localhost:8998/raw?{0}".format(query))
         self.assertEqual(str(l).count("Got response 200"), 1)
@@ -168,7 +167,7 @@ with multiples lines
     @defer.inlineCallbacks
     def test_retry_conn_lost(self):
         # connection lost after receiving data
-        crawler = self.runner.create_crawler(SimpleSpider)
+        crawler = get_crawler(SimpleSpider)
         with LogCapture() as l:
             yield crawler.crawl("http://localhost:8998/drop?abort=0")
         self._assert_retried(l)
@@ -176,7 +175,7 @@ with multiples lines
     @defer.inlineCallbacks
     def test_retry_conn_aborted(self):
         # connection lost before receiving data
-        crawler = self.runner.create_crawler(SimpleSpider)
+        crawler = get_crawler(SimpleSpider)
         with LogCapture() as l:
             yield crawler.crawl("http://localhost:8998/drop?abort=1")
         self._assert_retried(l)
@@ -195,7 +194,7 @@ with multiples lines
         req0.meta['next'] = req1
         req1.meta['next'] = req2
         req2.meta['next'] = req3
-        crawler = self.runner.create_crawler(SingleRequestSpider)
+        crawler = get_crawler(SingleRequestSpider)
         yield crawler.crawl(seed=req0)
         # basic asserts in case of weird communication errors
         self.assertIn('responses', crawler.spider.meta)
@@ -221,48 +220,9 @@ with multiples lines
         def cb(response):
             est.append(get_engine_status(crawler.engine))
 
-        crawler = self.runner.create_crawler(SingleRequestSpider)
+        crawler = get_crawler(SingleRequestSpider)
         yield crawler.crawl(seed='http://localhost:8998/', callback_func=cb)
         self.assertEqual(len(est), 1, est)
         s = dict(est[0])
         self.assertEqual(s['engine.spider.name'], crawler.spider.name)
         self.assertEqual(s['len(engine.scraper.slot.active)'], 1)
-
-    @defer.inlineCallbacks
-    def test_graceful_crawl_error_handling(self):
-        """
-        Test whether errors happening anywhere in Crawler.crawl() are properly
-        reported (and not somehow swallowed) after a graceful engine shutdown.
-        The errors should not come from within Scrapy's core but from within
-        spiders/middlewares/etc., e.g. raised in Spider.start_requests(),
-        SpiderMiddleware.process_start_requests(), etc.
-        """
-
-        class TestError(Exception):
-            pass
-
-        class FaultySpider(SimpleSpider):
-            def start_requests(self):
-                raise TestError
-
-        crawler = self.runner.create_crawler(FaultySpider)
-        yield self.assertFailure(crawler.crawl(), TestError)
-        self.assertFalse(crawler.crawling)
-
-    @defer.inlineCallbacks
-    def test_crawlerrunner_accepts_crawler(self):
-        crawler = self.runner.create_crawler(SimpleSpider)
-        with LogCapture() as log:
-            yield self.runner.crawl(crawler, "http://localhost:8998/status?n=200")
-        self.assertIn("Got response 200", str(log))
-
-    @defer.inlineCallbacks
-    def test_crawl_multiple(self):
-        self.runner.crawl(SimpleSpider, "http://localhost:8998/status?n=200")
-        self.runner.crawl(SimpleSpider, "http://localhost:8998/status?n=503")
-
-        with LogCapture() as log:
-            yield self.runner.join()
-
-        self._assert_retried(log)
-        self.assertIn("Got response 200", str(log))
diff --git a/tests/test_engine.py b/tests/test_engine.py
index dad921a60..e14957eae 100644
--- a/tests/test_engine.py
+++ b/tests/test_engine.py
@@ -19,7 +19,6 @@ from twisted.web import server, static, util
 from twisted.trial import unittest
 
 from scrapy import signals
-from scrapy.core.engine import ExecutionEngine
 from scrapy.utils.test import get_crawler
 from pydispatch import dispatcher
 from tests import tests_datadir
@@ -235,29 +234,6 @@ class EngineTest(unittest.TestCase):
         self.assertEqual({'spider': self.run.spider, 'reason': 'finished'},
                          self.run.signals_catched[signals.spider_closed])
 
-    @defer.inlineCallbacks
-    def test_close_downloader(self):
-        e = ExecutionEngine(get_crawler(TestSpider), lambda: None)
-        yield e.close()
-
-    @defer.inlineCallbacks
-    def test_close_spiders_downloader(self):
-        e = ExecutionEngine(get_crawler(TestSpider), lambda: None)
-        yield e.open_spider(TestSpider(), [])
-        self.assertEqual(len(e.open_spiders), 1)
-        yield e.close()
-        self.assertEqual(len(e.open_spiders), 0)
-
-    @defer.inlineCallbacks
-    def test_close_engine_spiders_downloader(self):
-        e = ExecutionEngine(get_crawler(TestSpider), lambda: None)
-        yield e.open_spider(TestSpider(), [])
-        e.start()
-        self.assertTrue(e.running)
-        yield e.close()
-        self.assertFalse(e.running)
-        self.assertEqual(len(e.open_spiders), 0)
-
 
 if __name__ == "__main__":
     if len(sys.argv) > 1 and sys.argv[1] == 'runserver':
diff --git a/tests/test_http_request.py b/tests/test_http_request.py
index 60fd855dd..ff0941961 100644
--- a/tests/test_http_request.py
+++ b/tests/test_http_request.py
@@ -801,25 +801,6 @@ class FormRequestTest(RequestTest):
         self.assertEqual(fs[b'test2'], [b'val2'])
         self.assertEqual(fs[b'button1'], [b''])
 
-    def test_html_base_form_action(self):
-        response = _buildresponse(
-            """
-            <html>
-                <head>
-                    <base href="http://b.com/">
-                </head>
-                <body>
-                    <form action="test_form">
-                    </form>
-                </body>
-            </html>
-            """,
-            url='http://a.com/'
-        )
-        req = self.request_class.from_response(response)
-        self.assertEqual(req.url, 'http://b.com/test_form')
-
-
 def _buildresponse(body, **kwargs):
     kwargs.setdefault('body', body)
     kwargs.setdefault('url', 'http://example.com')
diff --git a/tests/test_settings/__init__.py b/tests/test_settings/__init__.py
index 44b9b6df3..54b834aa0 100644
--- a/tests/test_settings/__init__.py
+++ b/tests/test_settings/__init__.py
@@ -2,21 +2,11 @@ import six
 import unittest
 import warnings
 
-from scrapy.settings import (BaseSettings, Settings, SettingsAttribute,
-                             CrawlerSettings, SETTINGS_PRIORITIES,
-                             get_settings_priority)
+from scrapy.settings import Settings, SettingsAttribute, CrawlerSettings
 from tests import mock
 from . import default_settings
 
 
-class SettingsGlobalFuncsTest(unittest.TestCase):
-
-    def test_get_settings_priority(self):
-        for prio_str, prio_num in six.iteritems(SETTINGS_PRIORITIES):
-            self.assertEqual(get_settings_priority(prio_str), prio_num)
-        self.assertEqual(get_settings_priority(99), 99)
-
-
 class SettingsAttributeTest(unittest.TestCase):
 
     def setUp(self):
@@ -37,37 +27,41 @@ class SettingsAttributeTest(unittest.TestCase):
         self.assertEqual(self.attribute.value, 'value')
         self.assertEqual(self.attribute.priority, 10)
 
-    def test_overwrite_basesettings(self):
-        original_dict = {'one': 10, 'two': 20}
-        original_settings = BaseSettings(original_dict, 0)
-        attribute = SettingsAttribute(original_settings, 0)
 
-        new_dict = {'three': 11, 'four': 21}
-        attribute.set(new_dict, 10)
-        self.assertIsInstance(attribute.value, BaseSettings)
-        six.assertCountEqual(self, attribute.value, new_dict)
-        six.assertCountEqual(self, original_settings, original_dict)
+class SettingsTest(unittest.TestCase):
 
-        new_settings = BaseSettings({'five': 12}, 0)
-        attribute.set(new_settings, 0)  # Insufficient priority
-        six.assertCountEqual(self, attribute.value, new_dict)
-        attribute.set(new_settings, 10)
-        six.assertCountEqual(self, attribute.value, new_settings)
+    if six.PY3:
+        assertItemsEqual = unittest.TestCase.assertCountEqual
 
-    def test_repr(self):
-        self.assertEqual(repr(self.attribute),
-                         "<SettingsAttribute value='value' priority=10>")
+    def setUp(self):
+        self.settings = Settings()
 
+    @mock.patch.dict('scrapy.settings.SETTINGS_PRIORITIES', {'default': 10})
+    @mock.patch('scrapy.settings.default_settings', default_settings)
+    def test_initial_defaults(self):
+        settings = Settings()
+        self.assertEqual(len(settings.attributes), 1)
+        self.assertIn('TEST_DEFAULT', settings.attributes)
 
-class BaseSettingsTest(unittest.TestCase):
+        attr = settings.attributes['TEST_DEFAULT']
+        self.assertIsInstance(attr, SettingsAttribute)
+        self.assertEqual(attr.value, 'defvalue')
+        self.assertEqual(attr.priority, 10)
 
-    if six.PY3:
-        assertItemsEqual = unittest.TestCase.assertCountEqual
+    @mock.patch.dict('scrapy.settings.SETTINGS_PRIORITIES', {})
+    @mock.patch('scrapy.settings.default_settings', {})
+    def test_initial_values(self):
+        settings = Settings({'TEST_OPTION': 'value'}, 10)
+        self.assertEqual(len(settings.attributes), 1)
+        self.assertIn('TEST_OPTION', settings.attributes)
 
-    def setUp(self):
-        self.settings = BaseSettings()
+        attr = settings.attributes['TEST_OPTION']
+        self.assertIsInstance(attr, SettingsAttribute)
+        self.assertEqual(attr.value, 'value')
+        self.assertEqual(attr.priority, 10)
 
     def test_set_new_attribute(self):
+        self.settings.attributes = {}
         self.settings.set('TEST_OPTION', 'value', 0)
         self.assertIn('TEST_OPTION', self.settings.attributes)
 
@@ -76,12 +70,6 @@ class BaseSettingsTest(unittest.TestCase):
         self.assertEqual(attr.value, 'value')
         self.assertEqual(attr.priority, 0)
 
-    def test_set_settingsattribute(self):
-        myattr = SettingsAttribute(0, 30)  # Note priority 30
-        self.settings.set('TEST_ATTR', myattr, 10)
-        self.assertEqual(self.settings.get('TEST_ATTR'), 0)
-        self.assertEqual(self.settings.getpriority('TEST_ATTR'), 30)
-
     def test_set_instance_identity_on_update(self):
         attr = SettingsAttribute('value', 0)
         self.settings.attributes = {'TEST_OPTION': attr}
@@ -91,11 +79,13 @@ class BaseSettingsTest(unittest.TestCase):
         self.assertIs(attr, self.settings.attributes['TEST_OPTION'])
 
     def test_set_calls_settings_attributes_methods_on_update(self):
-        attr = SettingsAttribute('value', 10)
-        with mock.patch.object(attr, '__setattr__') as mock_setattr, \
-                mock.patch.object(attr, 'set') as mock_set:
+        with mock.patch.object(SettingsAttribute, '__setattr__') as mock_setattr, \
+                mock.patch.object(SettingsAttribute, 'set') as mock_set:
 
+            attr = SettingsAttribute('value', 10)
             self.settings.attributes = {'TEST_OPTION': attr}
+            mock_set.reset_mock()
+            mock_setattr.reset_mock()
 
             for priority in (0, 10, 20):
                 self.settings.set('TEST_OPTION', 'othervalue', priority)
@@ -104,19 +94,6 @@ class BaseSettingsTest(unittest.TestCase):
                 mock_set.reset_mock()
                 mock_setattr.reset_mock()
 
-    def test_setitem(self):
-        settings = BaseSettings()
-        settings.set('key', 'a', 'default')
-        settings['key'] = 'b'
-        self.assertEqual(settings['key'], 'b')
-        self.assertEqual(settings.getpriority('key'), 20)
-        settings['key'] = 'c'
-        self.assertEqual(settings['key'], 'c')
-        settings['key2'] = 'x'
-        self.assertIn('key2', settings)
-        self.assertEqual(settings['key2'], 'x')
-        self.assertEqual(settings.getpriority('key2'), 20)
-
     def test_setdict_alias(self):
         with mock.patch.object(self.settings, 'set') as mock_set:
             self.settings.setdict({'TEST_1': 'value1', 'TEST_2': 'value2'}, 10)
@@ -141,8 +118,7 @@ class BaseSettingsTest(unittest.TestCase):
     def test_setmodule_alias(self):
         with mock.patch.object(self.settings, 'set') as mock_set:
             self.settings.setmodule(default_settings, 10)
-            mock_set.assert_any_call('TEST_DEFAULT', 'defvalue', 10)
-            mock_set.assert_any_call('TEST_DICT', {'key': 'val'}, 10)
+            mock_set.assert_called_with('TEST_DEFAULT', 'defvalue', 10)
 
     def test_setmodule_by_path(self):
         self.settings.attributes = {}
@@ -156,56 +132,11 @@ class BaseSettingsTest(unittest.TestCase):
         self.assertItemsEqual(six.iterkeys(self.settings.attributes),
                               six.iterkeys(ctrl_attributes))
 
-        for key in six.iterkeys(ctrl_attributes):
-            attr = self.settings.attributes[key]
-            ctrl_attr = ctrl_attributes[key]
+        for attr, ctrl_attr in zip(six.itervalues(self.settings.attributes),
+                                   six.itervalues(ctrl_attributes)):
             self.assertEqual(attr.value, ctrl_attr.value)
             self.assertEqual(attr.priority, ctrl_attr.priority)
 
-    def test_update(self):
-        settings = BaseSettings({'key_lowprio': 0}, priority=0)
-        settings.set('key_highprio', 10, priority=50)
-        custom_settings = BaseSettings({'key_lowprio': 1, 'key_highprio': 11},
-                                       priority=30)
-        custom_settings.set('newkey_one', None, priority=50)
-        custom_dict = {'key_lowprio': 2, 'key_highprio': 12, 'newkey_two': None}
-
-        settings.update(custom_dict, priority=20)
-        self.assertEqual(settings['key_lowprio'], 2)
-        self.assertEqual(settings.getpriority('key_lowprio'), 20)
-        self.assertEqual(settings['key_highprio'], 10)
-        self.assertIn('newkey_two', settings)
-        self.assertEqual(settings.getpriority('newkey_two'), 20)
-
-        settings.update(custom_settings)
-        self.assertEqual(settings['key_lowprio'], 1)
-        self.assertEqual(settings.getpriority('key_lowprio'), 30)
-        self.assertEqual(settings['key_highprio'], 10)
-        self.assertIn('newkey_one', settings)
-        self.assertEqual(settings.getpriority('newkey_one'), 50)
-
-        settings.update({'key_lowprio': 3}, priority=20)
-        self.assertEqual(settings['key_lowprio'], 1)
-
-    def test_update_jsonstring(self):
-        settings = BaseSettings({'number': 0, 'dict': BaseSettings({'key': 'val'})})
-        settings.update('{"number": 1, "newnumber": 2}')
-        self.assertEqual(settings['number'], 1)
-        self.assertEqual(settings['newnumber'], 2)
-        settings.set("dict", '{"key": "newval", "newkey": "newval2"}')
-        self.assertEqual(settings['dict']['key'], "newval")
-        self.assertEqual(settings['dict']['newkey'], "newval2")
-
-    def test_delete(self):
-        settings = BaseSettings({'key': None})
-        settings.set('key_highprio', None, priority=50)
-        settings.delete('key')
-        settings.delete('key_highprio')
-        self.assertNotIn('key', settings)
-        self.assertIn('key_highprio', settings)
-        del settings['key_highprio']
-        self.assertNotIn('key_highprio', settings)
-
     def test_get(self):
         test_configuration = {
             'TEST_ENABLED1': '1',
@@ -259,28 +190,6 @@ class BaseSettingsTest(unittest.TestCase):
         self.assertEqual(settings.getdict('TEST_DICT3', {'key1': 5}), {'key1': 5})
         self.assertRaises(ValueError, settings.getdict, 'TEST_LIST1')
 
-    def test_getpriority(self):
-        settings = BaseSettings({'key': 'value'}, priority=99)
-        self.assertEqual(settings.getpriority('key'), 99)
-        self.assertEqual(settings.getpriority('nonexistentkey'), None)
-
-    def test_getwithbase(self):
-        s = BaseSettings({'TEST_BASE': BaseSettings({1: 1, 2: 2}, 'project'),
-                          'TEST': BaseSettings({1: 10, 3: 30}, 'default'),
-                          'HASNOBASE': BaseSettings({3: 3000}, 'default')})
-        s['TEST'].set(2, 200, 'cmdline')
-        six.assertCountEqual(self, s.getwithbase('TEST'),
-                             {1: 1, 2: 200, 3: 30})
-        six.assertCountEqual(self, s.getwithbase('HASNOBASE'), s['HASNOBASE'])
-        self.assertEqual(s.getwithbase('NONEXISTENT'), {})
-
-    def test_maxpriority(self):
-        # Empty settings should return 'default'
-        self.assertEqual(self.settings.maxpriority(), 0)
-        self.settings.set('A', 0, 10)
-        self.settings.set('B', 0, 30)
-        self.assertEqual(self.settings.maxpriority(), 30)
-
     def test_copy(self):
         values = {
             'TEST_BOOL': True,
@@ -334,6 +243,7 @@ class BaseSettingsTest(unittest.TestCase):
             self.assertEqual(self.settings.get('FOO'), 'fez')
             self.assertEqual(self.settings.overrides.get('FOO'), 'fez')
 
+
     def test_deprecated_attribute_defaults(self):
         self.settings.set('BAR', 'fuz', priority='default')
         with warnings.catch_warnings(record=True) as w:
@@ -343,65 +253,6 @@ class BaseSettingsTest(unittest.TestCase):
             self.assertEqual(self.settings.defaults.get('BAR'), 'foo')
             self.assertIn('BAR', self.settings.defaults)
 
-    def test_repr(self):
-        settings = BaseSettings()
-        self.assertEqual(repr(settings), "<BaseSettings {}>")
-        attr = SettingsAttribute('testval', 15)
-        settings['testkey'] = attr
-        self.assertEqual(repr(settings),
-                         "<BaseSettings {'testkey': %s}>" % repr(attr))
-
-
-class SettingsTest(unittest.TestCase):
-
-    if six.PY3:
-        assertItemsEqual = unittest.TestCase.assertCountEqual
-
-    def setUp(self):
-        self.settings = Settings()
-
-    @mock.patch.dict('scrapy.settings.SETTINGS_PRIORITIES', {'default': 10})
-    @mock.patch('scrapy.settings.default_settings', default_settings)
-    def test_initial_defaults(self):
-        settings = Settings()
-        self.assertEqual(len(settings.attributes), 2)
-        self.assertIn('TEST_DEFAULT', settings.attributes)
-
-        attr = settings.attributes['TEST_DEFAULT']
-        self.assertIsInstance(attr, SettingsAttribute)
-        self.assertEqual(attr.value, 'defvalue')
-        self.assertEqual(attr.priority, 10)
-
-    @mock.patch.dict('scrapy.settings.SETTINGS_PRIORITIES', {})
-    @mock.patch('scrapy.settings.default_settings', {})
-    def test_initial_values(self):
-        settings = Settings({'TEST_OPTION': 'value'}, 10)
-        self.assertEqual(len(settings.attributes), 1)
-        self.assertIn('TEST_OPTION', settings.attributes)
-
-        attr = settings.attributes['TEST_OPTION']
-        self.assertIsInstance(attr, SettingsAttribute)
-        self.assertEqual(attr.value, 'value')
-        self.assertEqual(attr.priority, 10)
-
-    @mock.patch('scrapy.settings.default_settings', default_settings)
-    def test_autopromote_dicts(self):
-        settings = Settings()
-        mydict = settings.get('TEST_DICT')
-        self.assertIsInstance(mydict, BaseSettings)
-        self.assertIn('key', mydict)
-        self.assertEqual(mydict['key'], 'val')
-        self.assertEqual(mydict.getpriority('key'), 0)
-
-    @mock.patch('scrapy.settings.default_settings', default_settings)
-    def test_getdict_autodegrade_basesettings(self):
-        settings = Settings()
-        mydict = settings.getdict('TEST_DICT')
-        self.assertIsInstance(mydict, dict)
-        self.assertEqual(len(mydict), 1)
-        self.assertIn('key', mydict)
-        self.assertEqual(mydict['key'], 'val')
-
 
 class CrawlerSettingsTest(unittest.TestCase):
 
diff --git a/tests/test_settings/default_settings.py b/tests/test_settings/default_settings.py
index c24b5a9b9..23005d4c6 100644
--- a/tests/test_settings/default_settings.py
+++ b/tests/test_settings/default_settings.py
@@ -1,5 +1,2 @@
 
 TEST_DEFAULT = 'defvalue'
-
-TEST_DICT = {'key': 'val'}
-
diff --git a/tests/test_spiderloader/__init__.py b/tests/test_spiderloader/__init__.py
index 68dca2e98..7cb5e299b 100644
--- a/tests/test_spiderloader/__init__.py
+++ b/tests/test_spiderloader/__init__.py
@@ -8,12 +8,10 @@ from twisted.trial import unittest
 
 # ugly hack to avoid cyclic imports of scrapy.spiders when running this test
 # alone
-import scrapy
 from scrapy.interfaces import ISpiderLoader
 from scrapy.spiderloader import SpiderLoader
 from scrapy.settings import Settings
 from scrapy.http import Request
-from scrapy.crawler import CrawlerRunner
 
 module_dir = os.path.dirname(os.path.abspath(__file__))
 
@@ -78,14 +76,3 @@ class SpiderLoaderTest(unittest.TestCase):
         settings = Settings({'SPIDER_MODULES': [module]})
         self.spider_loader = SpiderLoader.from_settings(settings)
         assert len(self.spider_loader._spiders) == 0
-
-    def test_crawler_runner_loading(self):
-        module = 'tests.test_spiderloader.test_spiders.spider1'
-        runner = CrawlerRunner({'SPIDER_MODULES': [module]})
-
-        self.assertRaisesRegexp(KeyError, 'Spider not found',
-                                runner.create_crawler, 'spider2')
-
-        crawler = runner.create_crawler('spider1')
-        self.assertTrue(issubclass(crawler.spidercls, scrapy.Spider))
-        self.assertEqual(crawler.spidercls.name, 'spider1')
diff --git a/tests/test_utils_conf.py b/tests/test_utils_conf.py
index dab41ac8d..177d024fb 100644
--- a/tests/test_utils_conf.py
+++ b/tests/test_utils_conf.py
@@ -1,67 +1,40 @@
 import unittest
 
-from scrapy.settings import BaseSettings
 from scrapy.utils.conf import build_component_list, arglist_to_dict
 
 
 class BuildComponentListTest(unittest.TestCase):
 
     def test_build_dict(self):
-        d = {'one': 1, 'two': None, 'three': 8, 'four': 4}
-        self.assertEqual(build_component_list(d, convert=lambda x: x),
-                         ['one', 'four', 'three'])
-
-    def test_backwards_compatible_build_dict(self):
         base = {'one': 1, 'two': 2, 'three': 3, 'five': 5, 'six': None}
         custom = {'two': None, 'three': 8, 'four': 4}
-        self.assertEqual(build_component_list(base, custom,
-                                              convert=lambda x: x),
+        self.assertEqual(build_component_list(base, custom, lambda x: x),
                          ['one', 'four', 'five', 'three'])
 
     def test_return_list(self):
         custom = ['a', 'b', 'c']
-        self.assertEqual(build_component_list(None, custom,
-                                              convert=lambda x: x),
+        self.assertEqual(build_component_list(None, custom, lambda x: x),
                          custom)
 
     def test_map_dict(self):
         custom = {'one': 1, 'two': 2, 'three': 3}
-        self.assertEqual(build_component_list({}, custom,
-                                              convert=lambda x: x.upper()),
+        self.assertEqual(build_component_list({}, custom, lambda x: x.upper()),
                          ['ONE', 'TWO', 'THREE'])
 
     def test_map_list(self):
         custom = ['a', 'b', 'c']
-        self.assertEqual(build_component_list(None, custom,
-                                              lambda x: x.upper()),
+        self.assertEqual(build_component_list(None, custom, lambda x: x.upper()),
                          ['A', 'B', 'C'])
 
     def test_duplicate_components_in_dict(self):
         duplicate_dict = {'one': 1, 'two': 2, 'ONE': 4}
-        self.assertRaises(ValueError, build_component_list, {}, duplicate_dict,
-                          convert=lambda x: x.lower())
+        self.assertRaises(ValueError,
+                          build_component_list, {}, duplicate_dict, lambda x: x.lower())
 
     def test_duplicate_components_in_list(self):
         duplicate_list = ['a', 'b', 'a']
-        self.assertRaises(ValueError, build_component_list, None,
-                          duplicate_list, convert=lambda x: x)
-
-    def test_duplicate_components_in_basesettings(self):
-        # Higher priority takes precedence
-        duplicate_bs = BaseSettings({'one': 1, 'two': 2}, priority=0)
-        duplicate_bs.set('ONE', 4, priority=10)
-        self.assertEqual(build_component_list(duplicate_bs,
-                                              convert=lambda x: x.lower()),
-                         ['two', 'one'])
-        duplicate_bs.set('one', duplicate_bs['one'], priority=20)
-        self.assertEqual(build_component_list(duplicate_bs,
-                                              convert=lambda x: x.lower()),
-                         ['one', 'two'])
-        # Same priority raises ValueError
-        duplicate_bs.set('ONE', duplicate_bs['ONE'], priority=20)
-        self.assertRaises(ValueError, build_component_list, duplicate_bs,
-                          convert=lambda x: x.lower())
-
+        self.assertRaises(ValueError,
+                          build_component_list, None, duplicate_list, lambda x: x)
 
 class UtilsConfTestCase(unittest.TestCase):
 
diff --git a/tests/test_utils_python.py b/tests/test_utils_python.py
index 4f0834902..ca394ebf5 100644
--- a/tests/test_utils_python.py
+++ b/tests/test_utils_python.py
@@ -6,8 +6,7 @@ import six
 
 from scrapy.utils.python import (
     memoizemethod_noargs, isbinarytext, equal_attributes,
-    WeakKeyCache, stringify_dict, get_func_args, to_bytes, to_unicode,
-    without_none_values)
+    WeakKeyCache, stringify_dict, get_func_args, to_bytes, to_unicode)
 
 __doctests__ = ['scrapy.utils.python']
 
@@ -213,12 +212,5 @@ class UtilsPythonTestCase(unittest.TestCase):
         self.assertEqual(get_func_args(" ".join), [])
         self.assertEqual(get_func_args(operator.itemgetter(2)), [])
 
-    def test_without_none_values(self):
-        self.assertEqual(without_none_values([1, None, 3, 4]), [1, 3, 4])
-        self.assertEqual(without_none_values((1, None, 3, 4)), (1, 3, 4))
-        self.assertEqual(
-            without_none_values({'one': 1, 'none': None, 'three': 3, 'four': 4}),
-            {'one': 1, 'three': 3, 'four': 4})
-
 if __name__ == "__main__":
     unittest.main()