BaseSgmlLinkExtractor: Fixed unknown_endtag() so that it only set current_link=None when the end tag match the opening tag
BaseSgmlLinkExtractor: Added unit test of a link with an inner tag
BaseSgmlLinkExtractor: Fixed the missing space when the link has an inner tag
Disable smart strings in lxml XPath evaluations
Make lxml smart strings functionality customizable
Add testcase to check is default Selector doesnt return smart strings
Use assertTrue/False
Merge pull request #535 from redapple/xpath-smartstrings

Disable smart strings in lxml XPath evaluations
Ignore None's values when using the ItemLoader.
Unused re import and PEP8 minor edits.
Expose current crawler in the scrapy shell.
PEP8 minor edits.
RegexLinkExtractor: encode URL unicode value when creating Links
Updated shell docs with the crawler reference and fixed the actual shell output.

Also updated the shell example with a reproducible code example.
Updated the tutorial crawl output with latest output.
Merge pull request #561 from redapple/regexlx-encoding

RegexLinkExtractor: encode URL unicode value when creating Links
Merge pull request #556 from darkrho/item-loader-nones

Make `ItemLoader` ignore `None` values from processors.
replace unencodeable codepoints with html entities. fixes #562 and #285
Merge pull request #557 from darkrho/expose-crawler-in-shell

Expose current crawler in the scrapy shell.
Merge pull request #565 from dangra/562-sgmlinkextractor

replace unencodeable codepoints with html entities