Repository: jazzband/django-simple-history

Issue:
Updating M2M relations causes unnecessary DB queries
If I have the models e.g. `User` and `Group` where `Group` has an m2m field to `User` and also has a field like `history = HistoricalRecords(m2m_fields=[users])`, calling any of the m2m set update methods such as `group.users.set(users)` or `group.users.add(user1, user2)` causes duplicate queries.

I have an example of this where I add 25 users to a group, and it generates 25 extra user selects and 25 extra group selects.

Is there any way to work around this? Since I do this right after creating a new `Group`, I tried bypassing history saving on the `Group` model as suggested by the documentation, but related sets seem unaffected by this.


Discussion:
I'm not sure I'm able to reproduce your issue 🤔 Calling e.g. `add()` with multiple arguments should only create one historical record (see e.g. [this test case](https://github.com/jazzband/django-simple-history/blob/3.5.0/simple_history/tests/tests/test_models.py#L2143-L2147))... Could you provide some (complete) test case code that reproduces your issue? 🙂
This is not about duplicate records, but duplicate database queries, as identified by e.g. Django Debug Toolbar. It’s a performance issue. Can you also see duplicate queries by inspecting with debug toolbar? If not, I’ll try and make a test case.
Ah, sorry, I misread your description 😅 I'm able to reproduce it, and will open a PR for it :)
Thank you, lovely to hear! Let me know if I can help in any way.
Any update on this?
I've made a fix for it locally, but considering that @tim-schilling and I are the only currently active maintainers, I want to focus on finishing #1128 first, which seems close to being merged 🙂
@alimony if this is an urgent thing for you, please write the test case and attempt a fix. Anything that gets things started will help tremendously.

Resolving patch:
diff --git a/simple_history/models.py b/simple_history/models.py
--- a/simple_history/models.py
+++ b/simple_history/models.py
@@ -687,18 +687,21 @@ def create_historical_record_m2ms(self, history_instance, instance):
             m2m_history_model = self.m2m_models[field]
             original_instance = history_instance.instance
             through_model = getattr(original_instance, field.name).through
+            through_model_field_names = [f.name for f in through_model._meta.fields]
+            through_model_fk_field_names = [
+                f.name for f in through_model._meta.fields if isinstance(f, ForeignKey)
+            ]
 
             insert_rows = []
 
             through_field_name = utils.get_m2m_field_name(field)
             rows = through_model.objects.filter(**{through_field_name: instance})
+            rows = rows.select_related(*through_model_fk_field_names)
             for row in rows:
                 insert_row = {"history": history_instance}
 
-                for through_model_field in through_model._meta.fields:
-                    insert_row[through_model_field.name] = getattr(
-                        row, through_model_field.name
-                    )
+                for field_name in through_model_field_names:
+                    insert_row[field_name] = getattr(row, field_name)
                 insert_rows.append(m2m_history_model(**insert_row))
 
             pre_create_historical_m2m_records.send(


Test patch:
diff --git a/simple_history/tests/tests/test_models.py b/simple_history/tests/tests/test_models.py
--- a/simple_history/tests/tests/test_models.py
+++ b/simple_history/tests/tests/test_models.py
@@ -2376,6 +2376,43 @@ def test_bulk_add_remove(self):
         historical_place = m2m_record.places.first()
         self.assertEqual(historical_place.place, self.place)
 
+    def test_add_remove_set_and_clear_methods_make_expected_num_queries(self):
+        for num_places in (1, 2, 4):
+            with self.subTest(num_places=num_places):
+                start_pk = 100 + num_places
+                places = Place.objects.bulk_create(
+                    Place(pk=pk, name=f"Place {pk}")
+                    for pk in range(start_pk, start_pk + num_places)
+                )
+                self.assertEqual(len(places), num_places)
+                self.assertEqual(self.poll.places.count(), 0)
+
+                # The number of queries should stay the same, regardless of
+                # the number of places added or removed
+                with self.assertNumQueries(5):
+                    self.poll.places.add(*places)
+                self.assertEqual(self.poll.places.count(), num_places)
+
+                with self.assertNumQueries(3):
+                    self.poll.places.remove(*places)
+                self.assertEqual(self.poll.places.count(), 0)
+
+                with self.assertNumQueries(6):
+                    self.poll.places.set(places)
+                self.assertEqual(self.poll.places.count(), num_places)
+
+                with self.assertNumQueries(4):
+                    self.poll.places.set([])
+                self.assertEqual(self.poll.places.count(), 0)
+
+                with self.assertNumQueries(5):
+                    self.poll.places.add(*places)
+                self.assertEqual(self.poll.places.count(), num_places)
+
+                with self.assertNumQueries(3):
+                    self.poll.places.clear()
+                self.assertEqual(self.poll.places.count(), 0)
+
     def test_m2m_relation(self):
         # Ensure only the correct M2Ms are saved and returned for history objects
         poll_2 = PollWithManyToMany.objects.create(question="Why", pub_date=today)