Repository: sunpy/sunpy

Issue:
TST: sqlalchemy.exc.InvalidRequestError DatabaseEntry has been deleted
### Describe the bug

```
_______________________________________________________________________________ test_clear_database ________________________________________________________________________________

filled_database = <Table length=1>
 id  observation_time_start observation_time_end ... starred download_time size
str1          str3   ...-------- ... ------- ------------- ----
   4                    N/A                  N/A ...      No           N/A  N/A

    def test_clear_database(filled_database):
        assert len(filled_database) == 10
        filled_database.clear()
        assert not filled_database
        assert filled_database.session.query(JSONDump).all() == []
        assert filled_database.session.query(FitsHeaderEntry).all() == []
        assert filled_database.session.query(FitsKeyComment).all() == []
        assert filled_database.session.query(Tag).all() == []
>       filled_database.undo()

.tox/py310-sunpy/lib/python3.10/site-packages/sunpy/database/tests/test_database.py:731: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/py310-sunpy/lib/python3.10/site-packages/sunpy/database/database.py:1087: in undo
    self._command_manager.undo(n)
.tox/py310-sunpy/lib/python3.10/site-packages/sunpy/database/commands.py:359: in undo
    command.undo()
.tox/py310-sunpy/lib/python3.10/site-packages/sunpy/database/commands.py:98: in undo
    operation.undo()
.tox/py310-sunpy/lib/python3.10/site-packages/sunpy/database/commands.py:161: in undo
    self.session.add(self.entry)
.tox/py310-sunpy/lib/python3.10/site-packages/sqlalchemy/orm/scoping.py:361: in add
    return self._proxied.add(instance, _warn=_warn)
.tox/py310-sunpy/lib/python3.10/site-packages/sqlalchemy/orm/session.py:3137: in add
    self._save_or_update_state(state)
.tox/py310-sunpy/lib/python3.10/site-packages/sqlalchemy/orm/session.py:3167: in _save_or_update_state
    self._save_or_update_impl(st_)
.tox/py310-sunpy/lib/python3.10/site-packages/sqlalchemy/orm/session.py:3804: in _save_or_update_impl
    self._update_impl(state)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <sqlalchemy.orm.session.Session object at 0x7feda710ee60>, state = <sqlalchemy.orm.state.InstanceState object at 0x7feda7736800>, revert_deletion = False

    def _update_impl(
        self, state: InstanceState[Any], revert_deletion: bool = False
    ) -> None:
        if state.key is None:
            raise sa_exc.InvalidRequestError(
                "Instance '%s' is not persisted" % state_str(state)
            )
    
        if state._deleted:
            if revert_deletion:
                if not state._attached:
                    return
                del state._deleted
            else:
>               raise sa_exc.InvalidRequestError(
                    "Instance '%s' has been deleted.  "
                    "Use the make_transient() "
                    "function to send this object back "
                    "to the transient state." % state_str(state)
                )
E               sqlalchemy.exc.InvalidRequestError: Instance '<DatabaseEntry at 0x7feda8cdc580>' has been deleted.  Use the make_transient() function to send this object back to the transient state.

.tox/py310-sunpy/lib/python3.10/site-packages/sqlalchemy/orm/session.py:3774: InvalidRequestError
```

### To Reproduce

Run sunpy test over at https://github.com/astropy/astropy-integration-testing/actions

Example log: https://github.com/astropy/astropy-integration-testing/actions/runs/3623315366/jobs/6109043995

### Screenshots

_No response_

### System Details

_No response_

### Installation method

pip


Discussion:
If possible, I would recommend skipping the sunpy database tests. 

I see that the version of SQLAlchemy is 2.0.0b3. Which we aren't currently testing against.
Testing with 2.0.0b4, the CI fails so its a problem for us to fix.
How does one do that in tox?

https://github.com/astropy/astropy-integration-testing/blob/bb61b25faa37884dcefa695ec8f4edec4a4e5090/tox.ini#L51
That is a good question.
I guess the quickest way is to add a `pip install -U SQLAlchemy<2` (not tested that this works) to the command list before running any other commands in the tox file.

I do wonder what is forcing it to install the beta version of SQLAlchemy.
We have the predeps on to grab all the pre-releases for integration testing of, well, pre-release. So maybe that's why.
Ah I missed that. 

I will try to fix this on the sunpy side soon hopefully as I assume SQLAlchemy 2 can't be that far away.
OK, I think pinning it for the sunpy job for our side should be good enough for now. Thanks for the pointers!
I can confirm that pinning to SQLAlchemy<2 fixed the integration testing for sunpy + astropy . Thanks!

Resolving patch:
diff --git a/sunpy/database/commands.py b/sunpy/database/commands.py
--- a/sunpy/database/commands.py
+++ b/sunpy/database/commands.py
@@ -4,6 +4,8 @@
 from sqlalchemy.exc import InvalidRequestError
 from sqlalchemy.orm import make_transient
 
+from sunpy.database.tables import DatabaseEntry
+
 __all__ = [
     'EmptyCommandStackError', 'NoSuchEntryError', 'NonRemovableTagError',
     'DatabaseOperation', 'AddEntry', 'RemoveEntry', 'EditEntry',
@@ -94,8 +96,20 @@ def __call__(self):
             operation()
 
     def undo(self):
+        # To prevent errors in SQLa 2.0 we have to first add all removed entries back in before we add any ~database.Tags.
+        for operation in self._operations:
+            if isinstance(operation, RemoveEntry):
+                # we also filter out ~database.commands.RemoveEntry where type is not ~database.tables.DatabaseEntry to do later.
+                if isinstance(operation.entry, DatabaseEntry):
+                    operation.undo()
         for operation in self._operations:
-            operation.undo()
+            if isinstance(operation, RemoveEntry):
+                if not isinstance(operation.entry, DatabaseEntry):
+                    # undo all ~database.commands.RemoveEntry where type is not ~database.tables.DatabaseEntry
+                    operation.undo()
+            else:
+                # and we also undo all other commands, e.g. ~database.commands.RemoveTag.
+                operation.undo()
 
     def __len__(self):
         return len(self._operations)
diff --git a/sunpy/database/tables.py b/sunpy/database/tables.py
--- a/sunpy/database/tables.py
+++ b/sunpy/database/tables.py
@@ -8,8 +8,7 @@
 
 import numpy as np
 from sqlalchemy import Boolean, Column, DateTime, Float, ForeignKey, Integer, String, Table
-from sqlalchemy.ext.declarative import declarative_base
-from sqlalchemy.orm import relationship
+from sqlalchemy.orm import declarative_base, relationship
 
 import astropy.table
 import astropy.units as u


Test patch:
diff --git a/sunpy/database/tests/test_database.py b/sunpy/database/tests/test_database.py
--- a/sunpy/database/tests/test_database.py
+++ b/sunpy/database/tests/test_database.py
@@ -730,6 +730,7 @@ def test_clear_database(filled_database):
     assert filled_database.session.query(Tag).all() == []
     filled_database.undo()
     assert len(filled_database) == 10
+    assert len(filled_database.session.query(Tag).all()) == 2
     filled_database.redo()
     assert not filled_database
     assert filled_database.session.query(JSONDump).all() == []