Repository: Archmonger/django-dbbackup

Issue:
Error in db restore: no such table
When I try to restore the database I receive

~~~
dbbackup/db/sqlite.py:74: UserWarning: Error in db restore: no such table: main.auth_group_permissions
  warnings.warn("Error in db restore: {}".format(err))
~~~


Discussion:
Hello @rgaiacs 
Could you check your backup file to look if the table is created before (or after) ?
Be backup starts with

~~~
CREATE TABLE "auth_group" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "nam
e" varchar(80) NOT NULL UNIQUE);
CREATE UNIQUE INDEX "auth_group_permissions_group_id_permission_id_0cd325b0_uniq
" ON "auth_group_permissions" ("group_id", "permission_id");
CREATE INDEX IF NOT EXISTS "auth_group_permissions_group_id_b120cbf9" ON "auth_g
roup_permissions" ("group_id");
CREATE INDEX IF NOT EXISTS "auth_group_permissions_permission_id_84c5c92e" ON "a
uth_group_permissions" ("permission_id");
~~~

and later it has

~~~
CREATE TABLE "auth_group_permissions" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "group_id" integer NOT NULL REFERENCES "auth_group" ("id"), "permission_id" integer NOT NULL REFERENCES "auth_permission" ("id"));
CREATE UNIQUE INDEX "auth_group_permissions_group_id_permission_id_0cd325b0_uniq" ON "auth_group_permissions" ("group_id", "permission_id");
CREATE INDEX IF NOT EXISTS "auth_group_permissions_group_id_b120cbf9" ON "auth_group_permissions" ("group_id");
CREATE INDEX IF NOT EXISTS "auth_group_permissions_permission_id_84c5c92e" ON "auth_group_permissions" ("permission_id");
~~~
@rgaiacs Hmmm,
Does the table is created after the INSERTs ?
the --clean option for postgresql needs to add --if-exists  

this is what causes this error https://github.com/django-dbbackup/django-dbbackup/blob/master/dbbackup/db/postgresql.py#L51

but it honestly is a good thing and you should simply add --if-exists when adding --clean


link to pg_dump docs: https://www.postgresql.org/docs/10/app-pgdump.html
I've been receiving the same warnings:

'UserWarning: Error in db restore: no such table: main.auth_permission
warnings.warn("Error in db restore: {}".format(err))'

and some anothers:

'UserWarning: Error in db restore: FOREIGN KEY constraint failed
  warnings.warn("Error in db restore: {}".format(err))'

'UserWarning: Error in db restore: index django_content_type_app_label_model_76bd3d3b_uniq already exists
  warnings.warn("Error in db restore: {}".format(err))'

Any suggestions?  ( I'm using SQLite3)

> I've been receiving the same warnings:
> 
> 'UserWarning: Error in db restore: no such table: main.auth_permission
> warnings.warn("Error in db restore: {}".format(err))'
> 
> and some anothers:
> 
> 'UserWarning: Error in db restore: FOREIGN KEY constraint failed
> warnings.warn("Error in db restore: {}".format(err))'
> 
> 'UserWarning: Error in db restore: index django_content_type_app_label_model_76bd3d3b_uniq already exists
> warnings.warn("Error in db restore: {}".format(err))'
> 
> Any suggestions? ( I'm using SQLite3)

Is your problem resolved since then? I have almost the same problem with SQLite3 as well and could not find any useful issue pointing out this particular problem.

If I run the `dbrestore` command twice, all of my records are restored with a bunch of more "UserWarning: Error in ...". But, I should have run it once, right? Or twice 🤔?
Having the same issue, where the tables are created in alphabetical order and this creates conflicts:

```
class ExpenseType(models.Model):
    name = models.CharField(max_length=15, unique=True)
    position = models.IntegerField(
        unique=True,
        help_text="Column position in grouped reports.",
    )


class Expense(models.Model):
    related_date = models.DateField(help_text="Date that the expense is accounted to.")
    value = models.DecimalField(max_digits=6, decimal_places=2)
    target = models.ForeignKey(to=ExpenseType, on_delete=models.PROTECT)
    paid_by = models.ForeignKey(
        to=UserModel,
        on_delete=models.PROTECT,
        null=True,
        blank=True,
    )
    comment = models.TextField(
        max_length=200,
        null=True,
        blank=True,
    )
```

The the models above, result in this in the dump:
```
CREATE TABLE IF NOT EXISTS "expenses_expense" (... "expenses_expensetype" ("id") DEFERRABLE INITIALLY DEFERRED);
CREATE TABLE IF NOT EXISTS "expenses_expensetype"...;
```

Which then results in UserWarning: Error in db restore: no such table: main.expenses_expensetype



Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bdb505d7..1322249c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -37,6 +37,9 @@ Don't forget to remove deprecated code on each major release!
 - Fix MediaRestore path corruption for files containing "media" in their paths.
 - Fix FTP storage restore issue where file objects without `fileno()` support caused `io.UnsupportedOperation` error during database restore operations.
 - Fix SQLite restore failing when multi-line `TextField` content contains semicolons.
+- Fix SQLite `no such table` errors.
+- Fix SQLite `UNIQUE constraint` errors.
+- Fix SQLite `index`/`trigger`/`view` `<NAME> already exists` errors.
 
 ### Security
 
diff --git a/dbbackup/db/sqlite.py b/dbbackup/db/sqlite.py
index d5ad1739..3f22e418 100644
--- a/dbbackup/db/sqlite.py
+++ b/dbbackup/db/sqlite.py
@@ -43,20 +43,26 @@ def _write_dump(self, fileobj):
             fileobj.write(f"{sql};\n".encode())
 
             table_name_ident = table_name.replace('"', '""')
-            res = cursor.execute(f'PRAGMA table_info("{table_name_ident}")')
-            column_names = [str(table_info[1]) for table_info in res.fetchall()]
-            q = """SELECT 'INSERT INTO "{0}" VALUES({1})' FROM "{0}";\n""".format(
+            cursor.execute(f'PRAGMA table_info("{table_name_ident}")')
+            column_names = [str(table_info[1]) for table_info in cursor.fetchall()]
+            q = """SELECT 'INSERT OR REPLACE INTO "{0}" VALUES({1})' FROM "{0}";\n""".format(
                 table_name_ident,
-                ",".join("""'||quote("{}")||'""".format(col.replace('"', '""')) for col in column_names),
+                ",".join(f"""'||quote("{col.replace('"', '""')}")||'""" for col in column_names),
             )
-            query_res = cursor.execute(q)
-            for row in query_res:
+            cursor.execute(q)
+            for row in cursor:
                 fileobj.write(f"{row[0]};\n".encode())
-            schema_res = cursor.execute(DUMP_ETC)
-            for name, _, sql in schema_res.fetchall():
-                if sql.startswith("CREATE INDEX"):
-                    sql = sql.replace("CREATE INDEX", "CREATE INDEX IF NOT EXISTS")
-                fileobj.write(f"{sql};\n".encode())
+
+        # Dump indexes, triggers, and views after all tables are created
+        cursor.execute(DUMP_ETC)
+        for name, _, sql in cursor.fetchall():
+            if sql.startswith("CREATE INDEX"):
+                sql = sql.replace("CREATE INDEX", "CREATE INDEX IF NOT EXISTS", 1)
+            elif sql.startswith("CREATE TRIGGER"):
+                sql = sql.replace("CREATE TRIGGER", "CREATE TRIGGER IF NOT EXISTS", 1)
+            elif sql.startswith("CREATE VIEW"):
+                sql = sql.replace("CREATE VIEW", "CREATE VIEW IF NOT EXISTS", 1)
+            fileobj.write(f"{sql};\n".encode())
         cursor.close()
 
     def create_dump(self):
@@ -69,13 +75,13 @@ def create_dump(self):
 
     def _is_sql_command_complete(self, sql_command_bytes):
         """
-        Check if an SQL command is complete by ensuring that any closing ");\n" 
+        Check if an SQL command is complete by ensuring that any closing ");\n"
         is not within a quoted string literal.
         """
         sql_str = sql_command_bytes.decode("UTF-8")
         if not sql_str.endswith(");\n"):
             return False
-            
+
         # Parse the SQL to check if we're inside a quoted string at the end
         in_quotes = False
         i = 0
@@ -91,7 +97,7 @@ def _is_sql_command_complete(self, sql_command_bytes):
                     i += 1
             else:
                 i += 1
-        
+
         # The command is complete if we're not inside quotes when we reach ");\n"
         return not in_quotes
 
@@ -115,9 +121,18 @@ def restore_dump(self, dump):
                 try:
                     cursor.execute(sql_command.decode("UTF-8"))
                 except (OperationalError, IntegrityError) as err:
-                    warnings.warn(f"Error in db restore: {err}")
+                    err_str = str(err)
+                    if not self._should_suppress_error(err_str):
+                        warnings.warn(f"Error in db restore: {err}")
+
                 sql_command = b""
 
+    @staticmethod
+    def _should_suppress_error(msg: str):
+        return (msg.startswith("index") or msg.startswith("trigger") or msg.startswith("view")) and msg.endswith(
+            "already exists"
+        )
+
 
 class SqliteCPConnector(BaseDBConnector):
     """
diff --git a/dbbackup/management/commands/dbrestore.py b/dbbackup/management/commands/dbrestore.py
index 8c5a84ef..a57c56ba 100644
--- a/dbbackup/management/commands/dbrestore.py
+++ b/dbbackup/management/commands/dbrestore.py
@@ -135,7 +135,9 @@ def _restore_backup(self):
                 input_file.fileno()
             except (AttributeError, io.UnsupportedOperation):
                 # File doesn't support fileno(), convert to SpooledTemporaryFile
-                self.logger.debug("Converting remote storage file to temporary file due to missing fileno() support required by subprocess")
+                self.logger.debug(
+                    "Converting remote storage file to temporary file due to missing fileno() support required by subprocess"
+                )
                 temp_file = utils.create_spooled_temporary_file(fileobj=input_file)
                 input_file.close()
                 input_file = temp_file
diff --git a/docs/src/databases.md b/docs/src/databases.md
index 93184c46..bc3a148d 100644
--- a/docs/src/databases.md
+++ b/docs/src/databases.md
@@ -105,7 +105,7 @@ SQLite uses by default `dbbackup.db.sqlite.SqliteConnector`.
 
 ### SqliteConnector
 
-It is in pure Python and copies the behavior of `.dump` command for creating a
+It is in pure Python and is similar to the Sqlite `.dump` command for creating a
 SQL dump.
 
 ### SqliteBackupConnector
@@ -114,7 +114,7 @@ The `dbbackup.db.sqlite.SqliteBackupConnector` makes a copy of the SQLite databa
 
 ### SqliteCPConnector
 
-You can also use `dbbackup.db.sqlite.SqliteCPConnector` for making a 
+You can also use `dbbackup.db.sqlite.SqliteCPConnector` for making a
 simple raw copy of your database file, like a snapshot.
 
 In-memory database aren't dumpable with it.


Test patch:
diff --git a/tests/test_checks.py b/tests/test_checks.py
index 279abea4..9887d140 100644
--- a/tests/test_checks.py
+++ b/tests/test_checks.py
@@ -9,7 +9,7 @@
     checks = None
 
 
-def test_func(*args, **kwargs):
+def foobar_func(*args, **kwargs):
     return "foo"
 
 
@@ -33,7 +33,7 @@ def test_hostname_storage(self):
         errors = checks.check_settings(DbbackupConfig)
         self.assertEqual(expected_errors, errors)
 
-    @patch("dbbackup.checks.settings.FILENAME_TEMPLATE", test_func)
+    @patch("dbbackup.checks.settings.FILENAME_TEMPLATE", foobar_func)
     def test_filename_template_is_callable(self):
         self.assertFalse(checks.check_settings(DbbackupConfig))
 
@@ -47,7 +47,7 @@ def test_filename_template_no_date(self):
         errors = checks.check_settings(DbbackupConfig)
         self.assertEqual(expected_errors, errors)
 
-    @patch("dbbackup.checks.settings.MEDIA_FILENAME_TEMPLATE", test_func)
+    @patch("dbbackup.checks.settings.MEDIA_FILENAME_TEMPLATE", foobar_func)
     def test_media_filename_template_is_callable(self):
         self.assertFalse(checks.check_settings(DbbackupConfig))
 
diff --git a/tests/test_connectors/test_sqlite.py b/tests/test_connectors/test_sqlite.py
index d21930fe..963e7d53 100644
--- a/tests/test_connectors/test_sqlite.py
+++ b/tests/test_connectors/test_sqlite.py
@@ -4,7 +4,7 @@
 from django.db import connection
 from django.test import TestCase
 
-from dbbackup.db.sqlite import SqliteConnector, SqliteCPConnector, SqliteBackupConnector
+from dbbackup.db.sqlite import SqliteBackupConnector, SqliteConnector, SqliteCPConnector
 from tests.testapp.models import CharModel, TextModel
 
 
@@ -29,9 +29,7 @@ def test_create_dump_with_unicode(self):
         self.assertTrue(dump.read())
 
     def test_create_dump_with_newline(self):
-        TextModel.objects.create(
-            field=f'INSERT ({"foo" * 5000}\nbar\n WHERE \nbaz IS\n "great" );\n'
-        )
+        TextModel.objects.create(field=f'INSERT ({"foo" * 5000}\nbar\n WHERE \nbaz IS\n "great" );\n')
 
         connector = SqliteConnector()
         dump = connector.create_dump()
@@ -46,7 +44,7 @@ def test_restore_dump(self):
     def test_restore_dump_with_multiline_js_content(self):
         """Test restore of objects with JavaScript/HTML content containing '); patterns"""
         # Create content that contains "); patterns that could confuse the restore logic
-        js_content = '''function showAlert() {
+        js_content = """function showAlert() {
     alert("Hello world!");
     console.log("Debug info");
     return true;
@@ -56,29 +54,80 @@ def test_restore_dump_with_multiline_js_content(self):
     document.addEventListener("DOMContentLoaded", function() {
         console.log("Ready!");
     });
-</script>'''
-        
+</script>"""
+
         # Create, backup, delete, restore cycle
         original_obj = TextModel.objects.create(field=js_content)
         original_id = original_obj.id
-        
+
         connector = SqliteConnector()
         dump = connector.create_dump()
-        
+
         # Delete the original
         original_obj.delete()
         self.assertFalse(TextModel.objects.filter(id=original_id).exists())
-        
+
         # Restore and verify
         dump.seek(0)
         connector.restore_dump(dump)
-        
+
         restored_objects = TextModel.objects.filter(id=original_id)
         self.assertTrue(restored_objects.exists(), "Object should be restored")
-        
+
         restored_obj = restored_objects.first()
         self.assertEqual(restored_obj.field, js_content, "Content should match exactly")
 
+    def test_restore_dump_may_warn_for_already_exists(self):
+        """Test that restore may produce warnings for already existing objects"""
+        import warnings
+
+        # Create some test data
+        CharModel.objects.create(field="test1")
+        TextModel.objects.create(field="test content")
+
+        connector = SqliteConnector()
+        dump = connector.create_dump()
+
+        # Clear all data but keep schema (tables/indexes still exist)
+        CharModel.objects.all().delete()
+        TextModel.objects.all().delete()
+
+        # Restore may produce warnings for already existing schema objects
+        dump.seek(0)
+        with warnings.catch_warnings(record=True) as warning_list:
+            warnings.simplefilter("always")  # Capture all warnings
+            connector.restore_dump(dump)
+
+        # Verify data was restored despite any warnings
+        self.assertTrue(CharModel.objects.filter(field="test1").exists())
+        self.assertTrue(TextModel.objects.filter(field="test content").exists())
+
+    def test_restore_dump_warns_only_for_serious_errors(self):
+        """Test that restore only warns for serious errors like 'no such table'"""
+        import warnings
+        from io import BytesIO
+
+        # Create a malformed dump with reference to non-existent table
+        bad_dump = BytesIO()
+        bad_dump.write(b"INSERT INTO nonexistent_table VALUES(1, 'test');\n")
+        bad_dump.seek(0)
+
+        connector = SqliteConnector()
+
+        with warnings.catch_warnings(record=True) as warning_list:
+            warnings.simplefilter("always")
+            connector.restore_dump(bad_dump)
+
+        # Should warn about the serious error
+        dbbackup_warnings = [w for w in warning_list if "dbbackup" in str(w.filename)]
+        self.assertTrue(len(dbbackup_warnings) > 0, "Should warn about 'no such table' error")
+
+        warning_messages = [str(w.message) for w in dbbackup_warnings]
+        self.assertTrue(
+            any("no such table" in msg.lower() for msg in warning_messages),
+            f"Should warn about 'no such table', got: {warning_messages}",
+        )
+
     def test_create_dump_with_virtual_tables(self):
         with connection.cursor() as c:
             c.execute("CREATE VIRTUAL TABLE lookup USING fts5(field)")
@@ -87,6 +136,87 @@ def test_create_dump_with_virtual_tables(self):
         dump = connector.create_dump()
         self.assertTrue(dump.read())
 
+    def test_restore_dump_unique_conflict_updates_row(self):
+        """
+        Test that restoring a dump updates an existing row when a UNIQUE constraint conflict occurs,
+        verifying that INSERT OR REPLACE is used to restore the original data.
+        """
+        # Create initial row
+        obj = CharModel.objects.create(field="original")
+        obj_id = obj.id
+        # Backup
+        connector = SqliteConnector()
+        dump = connector.create_dump()
+        # Modify the existing row so that restoring will cause UNIQUE constraint failure on primary key
+        CharModel.objects.filter(id=obj_id).update(field="changed")
+        # Restore should update row back to original content via INSERT OR REPLACE retry
+        dump.seek(0)
+        connector.restore_dump(dump)
+        self.assertEqual(CharModel.objects.get(id=obj_id).field, "original")
+
+    def test_schema_objects_use_if_not_exists(self):
+        """Test that indexes, triggers, and views use IF NOT EXISTS syntax"""
+        from io import BytesIO
+
+        # Create a test table, index, trigger, and view
+        with connection.cursor() as c:
+            c.execute("CREATE TABLE test_schema (id INTEGER PRIMARY KEY, value TEXT)")
+            c.execute("CREATE INDEX test_idx ON test_schema(value)")
+            c.execute(
+                "CREATE TRIGGER test_trigger AFTER INSERT ON test_schema BEGIN UPDATE test_schema SET value = NEW.value || '_triggered' WHERE id = NEW.id; END"
+            )
+            c.execute("CREATE VIEW test_view AS SELECT * FROM test_schema WHERE value IS NOT NULL")
+
+        connector = SqliteConnector()
+        dump_file = BytesIO()
+        connector._write_dump(dump_file)
+
+        # Check that the dump contains IF NOT EXISTS for all schema objects
+        dump_content = dump_file.getvalue().decode("utf-8")
+
+        self.assertIn("CREATE INDEX IF NOT EXISTS", dump_content, "Indexes should use IF NOT EXISTS")
+        self.assertIn("CREATE TRIGGER IF NOT EXISTS", dump_content, "Triggers should use IF NOT EXISTS")
+        self.assertIn("CREATE VIEW IF NOT EXISTS", dump_content, "Views should use IF NOT EXISTS")
+
+    def test_restore_warns_about_already_exists_errors(self):
+        """Test that restore warns about 'already exists' errors"""
+        import warnings
+        from io import BytesIO
+
+        # Create dump content that would trigger "already exists" errors - WITHOUT IF NOT EXISTS
+        dump_content = """
+CREATE TABLE test_exists (id INTEGER PRIMARY KEY);
+CREATE INDEX test_exists_idx ON test_exists(id);
+CREATE TRIGGER test_exists_trigger AFTER INSERT ON test_exists BEGIN UPDATE test_exists SET id = NEW.id; END;
+CREATE VIEW test_exists_view AS SELECT * FROM test_exists;
+        """.strip()
+
+        dump_file = BytesIO(dump_content.encode("utf-8"))
+        connector = SqliteConnector()
+
+        # First create the schema objects to ensure "already exists" situations
+        with connection.cursor() as c:
+            c.execute("CREATE TABLE test_exists (id INTEGER PRIMARY KEY)")
+            c.execute("CREATE INDEX test_exists_idx ON test_exists(id)")
+            c.execute(
+                "CREATE TRIGGER test_exists_trigger AFTER INSERT ON test_exists BEGIN UPDATE test_exists SET id = NEW.id; END"
+            )
+            c.execute("CREATE VIEW test_exists_view AS SELECT * FROM test_exists")
+
+        with warnings.catch_warnings(record=True) as warning_list:
+            warnings.simplefilter("always")
+            connector.restore_dump(dump_file)
+
+        # Filter warnings from this package
+        dbbackup_warnings = [w for w in warning_list if "dbbackup" in str(w.filename)]
+
+        # Should warn about "already exists" errors now that filtering is removed
+        self.assertGreater(len(dbbackup_warnings), 0, "Should have warnings for 'already exists' errors")
+
+        # Verify we get warnings about "already exists"
+        already_exists_warnings = [w for w in dbbackup_warnings if "already exists" in str(w.message).lower()]
+        self.assertGreater(len(already_exists_warnings), 0, "Should warn about 'already exists' errors")
+
 
 @patch("dbbackup.db.sqlite.open", mock_open(read_data=b"foo"), create=True)
 class SqliteCPConnectorTest(TestCase):