GSoC API cleanup

API cleanup task of the 2014 GSoC.

Tasks Status
------------

- [x] Settings
  - [x] Freeze and copy support
  - [x] Document
  - [x] Test
- [x] Spider
  - [x] Add from_crawler method
  - [x] Deprecate set_crawler
  - [x] Document
  - [x] Test
- [x] Crawler
  - [x] Instantiate with a required spider class (along the settings)
  - [x] Crawl method will create a spider object with the provided arguments
  - [x] Merge the configure method with crawl and init
  - [x] Merge the start method with crawl
  - [x] Redefine CrawlerProcess API
  - [x] Support creating crawlers in a already setup ioloop
  - [x] Allow simultaneous execution of crawlers
  - [x] Drop support for scrapy.project.crawler singleton
  - [x] Document
  - [x] Test
- [x] SpiderManager
  - [x] Instantiate with a settings instance
  - [x] Add load method to return a spider class given its name
  - [x] Document
  - [x] Test
- [x] Integration
  - [x] Update cmdline with the new API
  - [x] Update commands
  - [x] Update tests
  - [x] Update misc components

Obsolete features
-----------------

 * `SpiderManager.create`: Spiders are instantiated with their `from_crawler` method, so, since that method is implemented in the base spider, `create` is no longer needed.
 * `Crawler.install`, `Crawler.uninstall`, `scrapy.project` and `scrapy.stats`: Their support needed to be dropped to allow running multiple parallel crawling jobs. 
 * `Crawler.start`: Merged with `Crawler.crawl`, and its previous functionality it's not easily portable to the new API.
 * `Crawler._start_requests`: This attribute doesn't need to be set, since now it's created and used in the same method.
 * `CrawlerProcess` has been substantially modified, but since it's not a public class backward compatibility wasn't provided.

Changed features
----------------

 * `SpiderManager.__init__(spider_modules)` -> `SpiderManager.__init__(settings)`: The creation of the `SpiderManager` objects is given a settings instance rather than a list of modules to abstract custom implementations.
 * `Crawler.__init__(settings)` -> `Crawler.__init__(spidercls, settings)`: Each crawler it's bound to a spider class upon instantiation. The reasons for this design decision are described in the [GSoC 2014 Proposal](https://gist.github.com/Curita/83851743f0db0d5b825c).
 * `Crawler.crawl(spider, requests=None)` -> `Crawler.crawl(*args, **kwargs)`: `Crawler.crawl` and `Crawler.start` have been merged, so `Crawler.crawl` returns now a deferred. The arguments provided are used to initialize the crawler's spider class. This design decision is also explained in the previously cited document.
 *  `scrapy.shell.inspect_response(response, spider=None)` ->  `scrapy.shell.inspect_response(response, spider)`: Since we don't have the crawler singleton anymore, we have to explicitly state the crawler. The easiest way to implement this was to use the crawler attribute in the spider argument, so the later is now required.
I see it is kept for backwards compatibility, +1 to undocument it. more comments on scrapy/crawler.py file below.