scrapy/scrapy — Issue #86: exceptions.AttributeError: 'dict' object has no attribute 'fields
Author: dfdeshom
Created: 2012-02-03 06:13:11+00:00
State: open

Hi,
we were running into problems where we would get the following exception when crawling urls:

Traceback (most recent call last):
[...]

field = item.fields[field_name]
exceptions.AttributeError: 'dict' object has no attribute 'fields'

After some digging, we found that the exception was caused by the newly-introduced item exporters, where the default is to export the item as json. This exception makese more sense:

ERROR: Error caught on signal handler: >
Traceback (most recent call last):
[...]
File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
return _iterencode(o, 0)
File "/usr/lib/python2.7/json/encoder.py", line 178, in default
raise TypeError(repr(o) + " is not JSON serializable")
exceptions.TypeError: datetime.datetime(2012, 2, 2, 0, 0, tzinfo=) is not JSON serializable

The item exporters extension ```scrapy.contrib.feedexport.FeedExporter``` should be either disabled by default or changed to a format that is guaranteed not to fail (like pickle), since not all items are json-serializable.