Repository: iterative/dvc

Issue:
Cannot VACUUM from within a transaction
First of all, thank you for building DVC. I've just discover it recently but I like the idea a lot! Thank you for all your hard work.

I'm running a command which outputs around 5 million images. DVC seems to fail while processing the hardlinks. 

```
$ dvc run -d code/radek_csv2png.py -o data/train-128/ -o data/test-128/ python code/radek_csv2png.py --sz 128 --pct 0.10   

...
...
...

Created 'hardlink': .dvc/cache/73/37f8c8f4c9f67d9070506f2f7338df -> data/test-128/9999996442242305.png 
Warning: Cleaning up state. This might take a while
Error: Unexpected error: cannot VACUUM from within a transaction
```

The interesting thing is that I've successfully executed the command in the past, while building the same amount of images but in half the size. I'm not sure if the size of the images might be what is creating issues for DVC.  I'll try to re-execute the command to see if it keeps failing.

Here are more info about my system.
```
$ cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.1 LTS"
NAME="Ubuntu"
VERSION="18.04.1 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.1 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
$ dvc --version
0.20.2
```

I've installed DVC though `pip`.




Discussion:
Hi @IamGianluca !

Looks like we have a bug while cleaning up our state file. Our current default entry limit is 10 000 000 entries, which is clearly not enough in your case. While we are considering increasing default limit and looking into vacuuming bug, you can change limit yourself in your dvc config with `dvc config state.row_limit 100000000`, which should make it work for now. Could you also tell us how big your state file is right now(i.e. `ls -lh .dvc/state`)?

Thanks,
Ruslan
One more thing, you are using python 3.6, right?

Resolving patch:
diff --git a/dvc/state.py b/dvc/state.py
--- a/dvc/state.py
+++ b/dvc/state.py
@@ -190,6 +190,12 @@ def load(self):
                 else:
                     raise
 
+    def _vacuum(self):
+        # NOTE: see https://bugs.python.org/issue28518
+        self.db.isolation_level = None
+        self._execute("VACUUM")
+        self.db.isolation_level = ''
+
     def dump(self):
         assert self.db is not None
 
@@ -213,7 +219,7 @@ def dump(self):
                                      self.STATE_TABLE,
                                      delete))
 
-            self._execute("VACUUM")
+            self._vacuum()
 
             cmd = "SELECT COUNT(*) FROM {}"
 


Test patch:
diff --git a/tests/test_state.py b/tests/test_state.py
--- a/tests/test_state.py
+++ b/tests/test_state.py
@@ -5,6 +5,7 @@
 from dvc.system import System
 from dvc.state import State
 from dvc.utils import file_md5
+from dvc.main import main
 
 from tests.basic_env import TestDvc
 
@@ -39,3 +40,20 @@ def test_update(self):
 
             entry_md5 = state.update(path)
             self.assertEqual(entry_md5, md5)
+
+
+class TestStateOverflow(TestDvc):
+    def test(self):
+        # NOTE: trying to add more entries than state can handle,
+        # to see if it will clean up and vacuum successfully
+        ret = main(['config', 'state.row_limit', '10'])
+        self.assertEqual(ret, 0)
+
+        dname = 'dir'
+        os.mkdir(dname)
+        for i in range(20):
+            with open(os.path.join(dname, str(i)), 'w+') as fobj:
+                fobj.write(str(i))
+
+        ret = main(['add', 'dir'])
+        self.assertEqual(ret, 0)