Repository: biopython/biopython

Issue:
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process
Recently (11 July 2023) the Windows 64 tests on AppVeyor have stopped working, last good build:

https://ci.appveyor.com/project/biopython/biopython/builds/47519949 - last good build

```
python -c "import sys; print(sys.version)"
3.10.12 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 19:01:18) [MSC v.1916 64 bit (AMD64)]
```

https://ci.appveyor.com/project/biopython/biopython/builds/47523213 - new failure

```
python -c "import sys; print(sys.version)"
3.11.4 | packaged by Anaconda, Inc. | (main, Jul  5 2023, 13:47:18) [MSC v.1916 64 bit (AMD64)]
```

This is likely due to the jump from Python 3.10 to 3.11, or perhaps from on of the Python dependencies.

(There were some unrelated failures in between, see 2287f70c0c54b893900442abd861735ac01c35bd for that)


```
======================================================================
ERROR: test_duplicate_id_load (common_BioSQL.DupLoadTest.test_duplicate_id_load)
Make sure can't import records with same ID (in one go).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 1011, in tearDown
    destroy_database()
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 234, in destroy_database
    os.remove(TESTDB)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmpnz4q3g65_BioSQL.db'
======================================================================
ERROR: test_duplicate_load (common_BioSQL.DupLoadTest.test_duplicate_load)
Make sure can't import a single record twice (in one go).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 1011, in tearDown
    destroy_database()
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 234, in destroy_database
    os.remove(TESTDB)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmpveckyte6_BioSQL.db'
======================================================================
ERROR: test_duplicate_load2 (common_BioSQL.DupLoadTest.test_duplicate_load2)
Make sure can't import a single record twice (in steps).
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 1011, in tearDown
    destroy_database()
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 234, in destroy_database
    os.remove(TESTDB)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmp5tuc5mu0_BioSQL.db'
======================================================================
ERROR: test_reload (common_BioSQL.InDepthLoadTest.test_reload)
Make sure can't reimport existing records.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 1250, in tearDown
    destroy_database()
  File "C:\projects\biopython\Tests\common_BioSQL.py", line 234, in destroy_database
    os.remove(TESTDB)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmp2gbx7v6e_BioSQL.db'
======================================================================
ERROR: test_bundle_without_target (test_MafIO_index.NewIndexTest.test_bundle_without_target)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\test_MafIO_index.py", line 155, in tearDown
    shutil.rmtree(self.tmpdir)
  File "C:\Py\Lib\shutil.py", line 759, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Py\Lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Py\Lib\shutil.py", line 620, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmpbkl11bxu\\database.sqlite3'
======================================================================
ERROR: test_good_big (test_MafIO_index.NewIndexTest.test_good_big)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\test_MafIO_index.py", line 155, in tearDown
    shutil.rmtree(self.tmpdir)
  File "C:\Py\Lib\shutil.py", line 759, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Py\Lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Py\Lib\shutil.py", line 620, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmp1_7klvpu\\database.sqlite3'
======================================================================
ERROR: test_good_small (test_MafIO_index.NewIndexTest.test_good_small)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\test_MafIO_index.py", line 155, in tearDown
    shutil.rmtree(self.tmpdir)
  File "C:\Py\Lib\shutil.py", line 759, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Py\Lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Py\Lib\shutil.py", line 620, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmpb36xwzlf\\database.sqlite3'
======================================================================
ERROR: test_length_coords_mismatch (test_MafIO_index.NewIndexTest.test_length_coords_mismatch)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\projects\biopython\Tests\test_MafIO_index.py", line 155, in tearDown
    shutil.rmtree(self.tmpdir)
  File "C:\Py\Lib\shutil.py", line [759](https://ci.appveyor.com/project/biopython/biopython/builds/47525001/job/42gbx250hgu75xk1#L759), in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Py\Lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Py\Lib\shutil.py", line 620, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\appveyor\\AppData\\Local\\Temp\\1\\tmpu4n11kx8\\database.sqlite3'
----------------------------------------------------------------------
```

i.e. Two classes of failure, removing a temporarly SQLite database in the BioSQL tests, and removing a temporary MAF index.

I've seen similar issues before on Windows where you cannot delete a file which still has an open handle (which is allowed on macOS and Linux). We are likely missing a couple of explicit file handle closes - something which made a lot of work for us with PyPy given its different GC model.

If this can easily be reproduced on a contributor or developer's Windows machine (i.e. not just using AppVeyor), that would be ideal for debugging this...


Discussion:
Help wanted: Windows users who can test this (e.g. am I right about the Python version being important?)
But all the errors were related to SQLite, aren't they? Could this be related/the reason:

https://til.simonwillison.net/python/os-remove-windows

I.e. explicitly closing the database connection?
Yes, a stale database connection is the most likely reason a handle is still open. e.g. this kind of thing (mostly related to PyPy testing on Windows if I recall correctly):

https://github.com/biopython/biopython/commit/8cbec18442eb0b588529693d7b127591873325f5
https://github.com/biopython/biopython/commit/b9de1b60826019867547662172c2f744fc67615d
https://github.com/biopython/biopython/commit/1027ba1eba9164607208a3e90c035510584249d4


Resolving patch:
diff --git a/Bio/AlignIO/MafIO.py b/Bio/AlignIO/MafIO.py
--- a/Bio/AlignIO/MafIO.py
+++ b/Bio/AlignIO/MafIO.py
@@ -281,14 +281,35 @@ def __init__(self, sqlite_file, maf_file, target_seqname):
         # if sqlite_file exists, use the existing db, otherwise index the file
         if os.path.isfile(sqlite_file):
             self._con = dbapi2.connect(sqlite_file)
-            self._record_count = self.__check_existing_db()
+            try:
+                self._record_count = self.__check_existing_db()
+            except ValueError as err:
+                self._maf_fp.close()
+                self._con.close()
+                raise err from None
         else:
             self._con = dbapi2.connect(sqlite_file)
-            self._record_count = self.__make_new_index()
+            try:
+                self._record_count = self.__make_new_index()
+            except ValueError as err:
+                self._maf_fp.close()
+                self._con.close()
+                raise err from None
 
         # lastly, setup a MafIterator pointing at the open maf_file
         self._mafiter = MafIterator(self._maf_fp)
 
+    def close(self):
+        """Close the file handle being used to read the data.
+
+        Once called, further use of the index won't work. The sole
+        purpose of this method is to allow explicit handle closure
+        - for example if you wish to delete the file, on Windows
+        you must first close all open handles to that file.
+        """
+        self._con.close()
+        self._record_count = 0
+
     def __check_existing_db(self):
         """Perform basic sanity checks upon loading an existing index (PRIVATE)."""
         try:


Test patch:
diff --git a/Tests/common_BioSQL.py b/Tests/common_BioSQL.py
--- a/Tests/common_BioSQL.py
+++ b/Tests/common_BioSQL.py
@@ -1008,9 +1008,9 @@ def setUp(self):
     def tearDown(self):
         self.server.rollback()
         self.server.close()
-        destroy_database()
         del self.db
         del self.server
+        destroy_database()
 
     def test_duplicate_load(self):
         """Make sure can't import a single record twice (in one go)."""
@@ -1247,9 +1247,9 @@ def setUp(self):
 
     def tearDown(self):
         self.server.close()
-        destroy_database()
         del self.db
         del self.server
+        destroy_database()
 
     def test_transfer(self):
         """Make sure can load record into another namespace."""
diff --git a/Tests/test_MafIO_index.py b/Tests/test_MafIO_index.py
--- a/Tests/test_MafIO_index.py
+++ b/Tests/test_MafIO_index.py
@@ -157,10 +157,12 @@ def tearDown(self):
         def test_good_small(self):
             idx = MafIndex(self.tmpfile, "MAF/ucsc_mm9_chr10.maf", "mm9.chr10")
             self.assertEqual(len(idx), 48)
+            idx.close()
 
         def test_good_big(self):
             idx = MafIndex(self.tmpfile, "MAF/ucsc_mm9_chr10_big.maf", "mm9.chr10")
             self.assertEqual(len(idx), 983)
+            idx.close()
 
         def test_bundle_without_target(self):
             self.assertRaises(
@@ -189,6 +191,9 @@ def setUp(self):
             )
             self.assertEqual(len(self.idx), 48)
 
+        def tearDown(self):
+            self.idx.close()
+
         def test_records_begin(self):
             rec1 = SeqRecord(
                 Seq(
@@ -349,7 +354,7 @@ def test_correct_retrieval_1(self):
 
             self.assertEqual(len(results), 4 + 4)
 
-            self.assertEqual({len(x) for x in results}, {4, 1, 9, 10, 4, 3, 5, 1})
+            self.assertEqual({len(x) for x in results}, {1, 3, 4, 5, 9, 10})
 
             # Code formatting note:
             # Expected start coordinates are grouped by alignment blocks