Repository: scrapy/scrapy

Issue:
dbm-based cache can become corrupted when spider is killed
Hi,

Scrapy uses shelve-based cache by default; this could lead to a bad experience with scrapy because shelve databases can become corrupted if spider is stopped forcefully.

What do you think about the following?
1. Check if it can be fixed - for example, by closing databases at certain points;
2. if it can't be fixed, then switch to an another cache backend.

This ticket could be related: https://github.com/scrapy/scrapy/issues/491 - I haven't looked at that ticket in details, but user mentions `close_spider` is not called on middleware - this could cause DB corruption for dbm backend. Any ideas?

File-based cache doesn't seem to have this issue. Another option is sqlite - in my experience it is much harder to get sqlite database corrupted, compared to dbm.



Discussion:
I found that LevelDB is very reliable and efficient to use as a cache backend. I haven't found any corruption issue when stopping the crawler (thing that I do a lot while developing spiders). But a sqlite-based backend could be a better default choice as it comes in the standard library.

IIRC, the file-based backend was the default cache backend a time ago.

@kmike did you manage to reproduce this consistently?

I'm a bit against changing the default backend, we had some performance issues before with large crawls and many files, and single file is certainly easier to manage (deleting could be immediate vs many minutes, S3 dot persistance in SH won't work well with FS backedn, etc)

Yes, this is reproducable, and I've faced this problem more that once myself. Based on our chat logs (searched for "cache"), I believe @andresp99999, @darkrho, @OlgaCh, @akhterwahab and @saulm also had this issue during 2013, and were vocal about it. Some quotes: "frustrating", "kind of annoying", "weird error" :)

This is an issue when spider is developed locally and cache is used to speedup re-runs; it is very common to stop spiders during development, and cache performance is not an issue - usually there is not a lot of data, and people tend to have SSDs on local machines.

I think we may enable bdb cache (or leveldb as Rolando suggested) at SH for production.


Resolving patch:
diff --git a/scrapy/contrib/httpcache.py b/scrapy/contrib/httpcache.py
--- a/scrapy/contrib/httpcache.py
+++ b/scrapy/contrib/httpcache.py
@@ -285,6 +285,69 @@ def _read_meta(self, spider, request):
             return pickle.load(f)
 
 
+class LeveldbCacheStorage(object):
+
+    def __init__(self, settings):
+        import leveldb
+        self._leveldb = leveldb
+        self.cachedir = data_path(settings['HTTPCACHE_DIR'], createdir=True)
+        self.expiration_secs = settings.getint('HTTPCACHE_EXPIRATION_SECS')
+        self.db = None
+
+    def open_spider(self, spider):
+        dbpath = os.path.join(self.cachedir, '%s.leveldb' % spider.name)
+        self.db = self._leveldb.LevelDB(dbpath)
+
+    def close_spider(self, spider):
+        del self.db
+
+    def retrieve_response(self, spider, request):
+        data = self._read_data(spider, request)
+        if data is None:
+            return  # not cached
+        url = data['url']
+        status = data['status']
+        headers = Headers(data['headers'])
+        body = data['body']
+        respcls = responsetypes.from_args(headers=headers, url=url)
+        response = respcls(url=url, headers=headers, status=status, body=body)
+        return response
+
+    def store_response(self, spider, request, response):
+        key = self._request_key(request)
+        data = {
+            'status': response.status,
+            'url': response.url,
+            'headers': dict(response.headers),
+            'body': response.body,
+        }
+        batch = self._leveldb.WriteBatch()
+        batch.Put('%s_data' % key, pickle.dumps(data, protocol=2))
+        batch.Put('%s_time' % key, str(time()))
+        self.db.Write(batch)
+
+    def _read_data(self, spider, request):
+        key = self._request_key(request)
+        try:
+            ts = self.db.Get('%s_time' % key)
+        except KeyError:
+            return  # not found or invalid entry
+
+        if 0 < self.expiration_secs < time() - float(ts):
+            return  # expired
+
+        try:
+            data = self.db.Get('%s_data' % key)
+        except KeyError:
+            return  # invalid entry
+        else:
+            return pickle.loads(data)
+
+    def _request_key(self, request):
+        return request_fingerprint(request)
+
+
+
 def parse_cachecontrol(header):
     """Parse Cache-Control header
 


Test patch:
diff --git a/scrapy/tests/test_downloadermiddleware_httpcache.py b/scrapy/tests/test_downloadermiddleware_httpcache.py
--- a/scrapy/tests/test_downloadermiddleware_httpcache.py
+++ b/scrapy/tests/test_downloadermiddleware_httpcache.py
@@ -5,6 +5,7 @@
 import unittest
 import email.utils
 from contextlib import contextmanager
+import pytest
 
 from scrapy.http import Response, HtmlResponse, Request
 from scrapy.spider import Spider
@@ -136,6 +137,12 @@ class FilesystemStorageTest(DefaultStorageTest):
     storage_class = 'scrapy.contrib.httpcache.FilesystemCacheStorage'
 
 
+class LeveldbStorageTest(DefaultStorageTest):
+
+    pytest.importorskip('leveldb')
+    storage_class = 'scrapy.contrib.httpcache.LeveldbCacheStorage'
+
+
 class DummyPolicyTest(_BaseTest):
 
     policy_class = 'scrapy.contrib.httpcache.DummyPolicy'