Repository: encode/django-rest-framework

Issue:
ModelSerializer fields does not get updated correctly when signals are connected to some fields
## Checklist

- [x] I have verified that that issue exists against the `master` branch of Django REST framework.
- [x] I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
- [x] This is not a usage question. (Those should be directed to the [discussion group](https://groups.google.com/forum/#!forum/django-rest-framework) instead.)
- [x] This cannot be dealt with as a third party library. (We prefer new functionality to be [in the form of third party libraries](https://www.django-rest-framework.org/community/third-party-packages/#about-third-party-packages) where possible.)
- [x] I have reduced the issue to the simplest possible case.
- [ ] I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

## Steps to reproduce

Having model Ticket

``` Python
class Ticket(models.Model):
    tags = models.ManyToManyField(to='Tag', ...)
    changed_by_signal = models.CharField(...)
    # ... other fields ...
```

then some custom signal which is listening to Ticket.tags changes

``` Python
@receiver(m2m_changed, sender=Ticket.tags.through)
def process_tags_changed(sender, ticket, tag_set, **kwargs):
    ticket.changed_by_signal = 'Some value'
    ticket.save()
```

and TicketModelSerializer

``` Python
class TicketModelSerializer(serializers.ModelSerializer):
    class Meta:
        model = Ticket
        fields = ('tags', 'changed_by_signal')
```

then when you save serializer with new `tags` and `changed_by_signal` **left unchanged** (e. g. `None`), you would expect the value of `changed_by_signal` to be 'Some value' as the signal sets, but the value is `None`. Why? Because `ModelSerializer.update` method does not update fields in correct way.

``` Python
class ModelSerializer(Serializer):
    ...

    def update(self, instance, validated_data):
        raise_errors_on_nested_writes('update', self, validated_data)
        info = model_meta.get_field_info(instance)

        # Simply set each attribute on the instance, and then save it.
        # Note that unlike `.create()` we don't need to treat many-to-many
        # relationships as being a special case. During updates we already
        # have an instance pk for the relationships to be associated with.
        for attr, value in validated_data.items():  # <- 1.
            if attr in info.relations and info.relations[attr].to_many:
                field = getattr(instance, attr)
                field.set(value)
            else:
                setattr(instance, attr, value)
        instance.save()

        return instance
```

1. ModelSerializer updates the data from validated_data dict which is not sorted in any way. So when `tags` gets updated before `changed_by_signal`, then the value which is set by signal gets overriden.

## Proposed solution

I suppose changing the ModelSerializer method to

1. save non m2m fields (does not trigger any signals and does not change the instance as a side effect)
2. save m2m fields (can trigger side signals and change instance by side effects)

``` Python
class ModelSerializer(Serializer):
    ...

    def update(self, instance, validated_data):
        raise_errors_on_nested_writes('update', self, validated_data)
        info = model_meta.get_field_info(instance)

        # Simply set each attribute on the instance, and then save it.
        # Note that unlike `.create()` we don't need to treat many-to-many
        # relationships as being a special case. During updates we already
        # have an instance pk for the relationships to be associated with.
        m2m_fields = []
        for attr, value in validated_data.items():
            if attr not in info.relations or not info.relations[attr].to_many:
                setattr(instance, attr, value)
            else:
                m2m_fields.append((attr, value))

        for attr, value in m2m_fields:
            field = getattr(instance, attr)
            field.set(value)

        instance.save()

        return instance
```

Which stores m2m fields in a list that is processed after every other field has been changed. This is my temporary override for the TicketModelSerializer. I would consider making a pull request with changes that fix this issue.

ModelSerializer fields does not get updated correctly when signals are connected to some fields
## Checklist

- [x] I have verified that that issue exists against the `master` branch of Django REST framework.
- [x] I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
- [x] This is not a usage question. (Those should be directed to the [discussion group](https://groups.google.com/forum/#!forum/django-rest-framework) instead.)
- [x] This cannot be dealt with as a third party library. (We prefer new functionality to be [in the form of third party libraries](https://www.django-rest-framework.org/community/third-party-packages/#about-third-party-packages) where possible.)
- [x] I have reduced the issue to the simplest possible case.
- [ ] I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

## Steps to reproduce

Having model Ticket

``` Python
class Ticket(models.Model):
    tags = models.ManyToManyField(to='Tag', ...)
    changed_by_signal = models.CharField(...)
    # ... other fields ...
```

then some custom signal which is listening to Ticket.tags changes

``` Python
@receiver(m2m_changed, sender=Ticket.tags.through)
def process_tags_changed(sender, ticket, tag_set, **kwargs):
    ticket.changed_by_signal = 'Some value'
    ticket.save()
```

and TicketModelSerializer

``` Python
class TicketModelSerializer(serializers.ModelSerializer):
    class Meta:
        model = Ticket
        fields = ('tags', 'changed_by_signal')
```

then when you save serializer with new `tags` and `changed_by_signal` **left unchanged** (e. g. `None`), you would expect the value of `changed_by_signal` to be 'Some value' as the signal sets, but the value is `None`. Why? Because `ModelSerializer.update` method does not update fields in correct way.

``` Python
class ModelSerializer(Serializer):
    ...

    def update(self, instance, validated_data):
        raise_errors_on_nested_writes('update', self, validated_data)
        info = model_meta.get_field_info(instance)

        # Simply set each attribute on the instance, and then save it.
        # Note that unlike `.create()` we don't need to treat many-to-many
        # relationships as being a special case. During updates we already
        # have an instance pk for the relationships to be associated with.
        for attr, value in validated_data.items():  # <- 1.
            if attr in info.relations and info.relations[attr].to_many:
                field = getattr(instance, attr)
                field.set(value)
            else:
                setattr(instance, attr, value)
        instance.save()

        return instance
```

1. ModelSerializer updates the data from validated_data dict which is not sorted in any way. So when `tags` gets updated before `changed_by_signal`, then the value which is set by signal gets overriden.

## Proposed solution

I suppose changing the ModelSerializer method to

1. save non m2m fields (does not trigger any signals and does not change the instance as a side effect)
2. save m2m fields (can trigger side signals and change instance by side effects)

``` Python
class ModelSerializer(Serializer):
    ...

    def update(self, instance, validated_data):
        raise_errors_on_nested_writes('update', self, validated_data)
        info = model_meta.get_field_info(instance)

        # Simply set each attribute on the instance, and then save it.
        # Note that unlike `.create()` we don't need to treat many-to-many
        # relationships as being a special case. During updates we already
        # have an instance pk for the relationships to be associated with.
        m2m_fields = []
        for attr, value in validated_data.items():
            if attr not in info.relations or not info.relations[attr].to_many:
                setattr(instance, attr, value)
            else:
                m2m_fields.append((attr, value))

        for attr, value in m2m_fields:
            field = getattr(instance, attr)
            field.set(value)

        instance.save()

        return instance
```

Which stores m2m fields in a list that is processed after every other field has been changed. This is my temporary override for the TicketModelSerializer. I would consider making a pull request with changes that fix this issue.



Discussion:
I think we're likely going to pass on this, as an out-of-scope usage.

We could *consider* a pull request that addressed it, but it seems like a pretty fundamental change, and it's not obvious that it wouldn't have side effects.

If you're really invested in seeing it happen you're welcome to attempt a pull request for it, to see if you're able to make the proposed change without affecting anything already existing in the test suite.
I think we're likely going to pass on this, as an out-of-scope usage.

We could *consider* a pull request that addressed it, but it seems like a pretty fundamental change, and it's not obvious that it wouldn't have side effects.

If you're really invested in seeing it happen you're welcome to attempt a pull request for it, to see if you're able to make the proposed change without affecting anything already existing in the test suite.

Resolving patch:
diff --git a/rest_framework/serializers.py b/rest_framework/serializers.py
--- a/rest_framework/serializers.py
+++ b/rest_framework/serializers.py
@@ -973,14 +973,22 @@ def update(self, instance, validated_data):
         # Note that unlike `.create()` we don't need to treat many-to-many
         # relationships as being a special case. During updates we already
         # have an instance pk for the relationships to be associated with.
+        m2m_fields = []
         for attr, value in validated_data.items():
             if attr in info.relations and info.relations[attr].to_many:
-                field = getattr(instance, attr)
-                field.set(value)
+                m2m_fields.append((attr, value))
             else:
                 setattr(instance, attr, value)
+
         instance.save()
 
+        # Note that many-to-many fields are set after updating instance.
+        # Setting m2m fields triggers signals which could potentialy change
+        # updated instance and we do not want it to collide with .update()
+        for attr, value in m2m_fields:
+            field = getattr(instance, attr)
+            field.set(value)
+
         return instance
 
     # Determine the fields to apply...


Test patch:
diff --git a/tests/test_model_serializer.py b/tests/test_model_serializer.py
--- a/tests/test_model_serializer.py
+++ b/tests/test_model_serializer.py
@@ -18,6 +18,8 @@
     MaxValueValidator, MinLengthValidator, MinValueValidator
 )
 from django.db import models
+from django.db.models.signals import m2m_changed
+from django.dispatch import receiver
 from django.test import TestCase
 
 from rest_framework import serializers
@@ -1251,7 +1253,6 @@ class Meta:
 
 
 class Issue6110Test(TestCase):
-
     def test_model_serializer_custom_manager(self):
         instance = Issue6110ModelSerializer().create({'name': 'test_name'})
         self.assertEqual(instance.name, 'test_name')
@@ -1260,3 +1261,43 @@ def test_model_serializer_custom_manager_error_message(self):
         msginitial = ('Got a `TypeError` when calling `Issue6110TestModel.all_objects.create()`.')
         with self.assertRaisesMessage(TypeError, msginitial):
             Issue6110ModelSerializer().create({'wrong_param': 'wrong_param'})
+
+
+class Issue6751Model(models.Model):
+    many_to_many = models.ManyToManyField(ManyToManyTargetModel, related_name='+')
+    char_field = models.CharField(max_length=100)
+    char_field2 = models.CharField(max_length=100)
+
+
+@receiver(m2m_changed, sender=Issue6751Model.many_to_many.through)
+def process_issue6751model_m2m_changed(action, instance, **_):
+    if action == 'post_add':
+        instance.char_field = 'value changed by signal'
+        instance.save()
+
+
+class Issue6751Test(TestCase):
+    def test_model_serializer_save_m2m_after_instance(self):
+        class TestSerializer(serializers.ModelSerializer):
+            class Meta:
+                model = Issue6751Model
+                fields = (
+                    'many_to_many',
+                    'char_field',
+                )
+
+        instance = Issue6751Model.objects.create(char_field='initial value')
+        m2m_target = ManyToManyTargetModel.objects.create(name='target')
+
+        serializer = TestSerializer(
+            instance=instance,
+            data={
+                'many_to_many': (m2m_target.id,),
+                'char_field': 'will be changed by signal',
+            }
+        )
+
+        serializer.is_valid()
+        serializer.save()
+
+        self.assertEqual(instance.char_field, 'value changed by signal')