[WIP] GSoC API cleanup

Progress on the 2014 GSoC, API cleanup task.

- [x] Spider
  - [x] Add from_crawler method
  - [x] Deprecate set_crawler
  - [x] Test
- [ ] Crawler
  - [x] Instantiate with a required spider class (along the settings)
  - [x] Crawl method will create a spider object with the provided arguments
  - [x] Support creating spiders from the different SpiderManager interfaces
  - [ ] Merge the configure method with init
  - [ ] Merge CrawlerProcess with Crawler
  - [ ] Document
  - [ ] Test
- [ ] SpiderManager
  - [x] Deprecate old interface
  - [x] Make init take no arguments
  - [x] Add load method to return a spider class given its name
  - [ ] Document
  - [ ] Test
- [ ] Settings
  - [x] Deprecate SPIDER_MANAGER_CLASS and SPIDER_MODULES
  - [x] Add [spiders] section to scrapy.cfg
  - [x] Add manager and modules settings to scrapy.cfg
  - [ ] Document options in scrapy.cfg
- [ ] Integration
  - [ ] Update cmdline with the new API
  - [ ] Update commands
  - [ ] Update tests
  - [ ] Update misc components

Backward Incompatible Changes:
- The deprecated SpiderManager implementation has been kept in another class and it's fully supported, but custom SpiderManagers that implement that interface by inheriting "SpiderManager" will have to subclass "DeprecatedSpiderManager" instead.