Issue 1:
Sometimes it is needed to parse several variants of item from single page, where only a couple of fields differ. Obvious way to do this is to create base item with all common fields collected and then create new items (for example in a loop) using new ItemLoader instance with base item passed to it.

The problem I'm facing now is that ItemLoader.load_item() modifies base item - which is counterintuitive and can result in weird behavior (for example if variant items can have different fields - after those field were added to base item - they would appear in all loaded items).

In [5]: item = Item()

In [7]: item['url'] = 'foo'

In [8]: item
Out[8]: {'url': 'foo'}

In [9]: l = ItemLoader(item)

In [12]: l.add_value('category', 'bar')

In [13]: item
Out[13]: {'url': 'foo'}

In [14]: item_copy = l.load_item()

In [15]: item_copy
Out[15]: {'category': 'bar', 'url': 'foo'}

In [16]: item
Out[16]: {'category': 'bar', 'url': 'foo'}

In [17]: id(item)
Out[17]: 49468304

In [18]: id(item_copy)
Out[18]: 49468304

Now I'm using workaround like this to suppress such behavior:

loader = ItemLoader(selector=sel)
...
item = loader.load_item()
for variant in variants:
item_copy = item.copy()
loader = ItemLoader(item_copy)
...
yield loader.load_item()

What do you think about using item copy inside ItemLoader by default?

Issue 2:
Seen with Scrapy 14.0, Twisted 11.1.0, Python 2.6/2.7. After downgrade to twisted 11.0.0 the error does not show up.

In my case, happened in a long running spider which doesn't perform anything unusual, and after the error the crawler hangs up.

See the log below:

2011-11-23 09:32:40-0600 [projects] DEBUG: Crawled (200) <GET http://www.example.net/p/foo> (referer: http://www.example.net/p?page=51903&sort=users)
2011-11-23 09:32:40-0600 [projects] DEBUG: Crawled (200) <GET http://www.example.net/p/bar> (referer: http://www.example.net/p?page=51903&sort=users)
2011-11-23 09:38:10-0600 [projects] INFO: Crawled 89600 pages (at 42 pages/min), scraped 31075 items (at 16 items/min)
2011-11-23 09:38:11-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 45, in run
self.crawler.start()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/crawler.py", line 76, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/tcp.py", line 337, in failIfNotConnected
self.connector.connectionFailed(failure.Failure(err))
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1055, in connectionFailed
self.factory.clientConnectionFailed(self, reason)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/web/client.py", line 413, in clientConnectionFailed
self._disconnectedDeferred.callback(None)
exceptions.AttributeError: ScrapyHTTPClientFactory instance has no attribute '_disconnectedDeferred'

2011-11-23 09:38:13-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 45, in run
self.crawler.start()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/crawler.py", line 76, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/tcp.py", line 337, in failIfNotConnected
self.connector.connectionFailed(failure.Failure(err))
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1055, in connectionFailed
self.factory.clientConnectionFailed(self, reason)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/web/client.py", line 413, in clientConnectionFailed
self._disconnectedDeferred.callback(None)
exceptions.AttributeError: ScrapyHTTPClientFactory instance has no attribute '_disconnectedDeferred'

2011-11-23 09:38:13-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 45, in run
self.crawler.start()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/crawler.py", line 76, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/tcp.py", line 337, in failIfNotConnected
self.connector.connectionFailed(failure.Failure(err))
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1055, in connectionFailed
self.factory.clientConnectionFailed(self, reason)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/web/client.py", line 413, in clientConnectionFailed
self._disconnectedDeferred.callback(None)
exceptions.AttributeError: ScrapyHTTPClientFactory instance has no attribute '_disconnectedDeferred'

2011-11-23 09:38:13-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 45, in run
self.crawler.start()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/crawler.py", line 76, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/tcp.py", line 337, in failIfNotConnected
self.connector.connectionFailed(failure.Failure(err))
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1055, in connectionFailed
self.factory.clientConnectionFailed(self, reason)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/web/client.py", line 413, in clientConnectionFailed
self._disconnectedDeferred.callback(None)
exceptions.AttributeError: ScrapyHTTPClientFactory instance has no attribute '_disconnectedDeferred'

2011-11-23 09:38:13-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 45, in run
self.crawler.start()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/crawler.py", line 76, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/tcp.py", line 337, in failIfNotConnected
self.connector.connectionFailed(failure.Failure(err))
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1055, in connectionFailed
self.factory.clientConnectionFailed(self, reason)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/web/client.py", line 413, in clientConnectionFailed
self._disconnectedDeferred.callback(None)
exceptions.AttributeError: ScrapyHTTPClientFactory instance has no attribute '_disconnectedDeferred'

2011-11-23 09:38:13-0600 [-] Unhandled Error
Traceback (most recent call last):
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/commands/crawl.py", line 45, in run
self.crawler.start()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/scrapy/crawler.py", line 76, in start
reactor.run(installSignalHandlers=False) # blocking call
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1169, in run
self.mainLoop()
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1178, in mainLoop
self.runUntilCurrent()
--- <exception caught here> ---
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 800, in runUntilCurrent
call.func(*call.args, **call.kw)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/tcp.py", line 337, in failIfNotConnected
self.connector.connectionFailed(failure.Failure(err))
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/internet/base.py", line 1055, in connectionFailed
self.factory.clientConnectionFailed(self, reason)
File "/home/josemanuel/envs/global/lib/python2.7/site-packages/twisted/web/client.py", line 413, in clientConnectionFailed
self._disconnectedDeferred.callback(None)
exceptions.AttributeError: ScrapyHTTPClientFactory instance has no attribute '_disconnectedDeferred'

2011-11-23 09:38:58-0600 [projects] INFO: Crawled 89600 pages (at 0 pages/min), scraped 31075 items (at 0 items/min)
2011-11-23 09:39:58-0600 [projects] INFO: Crawled 89600 pages (at 0 pages/min), scraped 31075 items (at 0 items/min)
2011-11-23 09:40:58-0600 [projects] INFO: Crawled 89600 pages (at 0 pages/min), scraped 31075 items (at 0 items/min)