Repository: encode/django-rest-framework

Issue:
Uniqueness validators should not be run for excluded (read_only) fields
get_uniqueness_extra_kwargs and get_unique_together_validators both rely on Django's model unique_together specification. However, if read_only is set for both they should not be expected to be validated as normal. This reflects the behavior of django forms (exclude the field in form and it doesn't validated against this). This allows unique_together fields to be autogenerated in the save method of the model.

ListSerializer doesn't handle unique_together constraints
## Checklist
- [X] I have verified that that issue exists against the `master` branch of Django REST framework.
- [X] I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
- [X] This is not a usage question. (Those should be directed to the [discussion group](https://groups.google.com/forum/#!forum/django-rest-framework) instead.)
- [X] This cannot be dealt with as a third party library. (We prefer new functionality to be [in the form of third party libraries](http://www.django-rest-framework.org/topics/third-party-resources/#about-third-party-packages) where possible.)
- [X] I have reduced the issue to the simplest possible case.
- [ ] I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

As referenced in
https://github.com/tomchristie/django-rest-framework/pull/2575
https://github.com/miki725/django-rest-framework-bulk/issues/30

ListSerializer attempts to apply unique together validation on the queryset passed to it, but some part of the code is passed the full queryset, where it attempts to look up attributes on what it expects to be individual instances.
## Steps to reproduce

I've modified a test case found in the above links, and run them against master. I'm not sure what the file should be named, and if it follows your quality standards, so I have not submitted  a PR with it.

``` python
from django.db import models
from django.test import TestCase

from rest_framework import serializers


class ManyUniqueTogetherModel(models.Model):
    foo = models.IntegerField()
    bar = models.IntegerField()
    class Meta:
        unique_together = ("foo", "bar")


class ManyUniqueTogetherSerializer(serializers.ModelSerializer):
    class Meta:
        model = ManyUniqueTogetherModel


class ManyUniqueTogetherTestCase(TestCase):

    def test_partial_false(self):
        obj = ManyUniqueTogetherModel.objects.create(foo=1, bar=2)
        serializer = ManyUniqueTogetherSerializer(
            instance=ManyUniqueTogetherModel.objects.all(),
            data=[{
                "id": obj.pk,
                "foo": 5,
                "bar": 6,
            }],
            many=True
        )
        self.assertTrue(serializer.is_valid())

    def test_partial_true(self):
        obj = ManyUniqueTogetherModel.objects.create(foo=1, bar=2)
        serializer = ManyUniqueTogetherSerializer(
            instance=ManyUniqueTogetherModel.objects.all(),
            data=[{
                "id": obj.pk,
                "foo": 5,
            }],
            partial=True,
            many=True
        )
        self.assertTrue(serializer.is_valid())
```
## Expected behavior

The code path looks up properties on individual instances in the queryset, is_valid() returns either True or False depending on the actual data, the tests pass.
## Actual behavior

```
====================================================================== FAILURES ======================================================================
___________________________________________________ ManyUniqueTogetherTestCase.test_partial_false ____________________________________________________
tests/test_unique_together.py:35: in test_partial_false
    self.assertTrue(serializer.is_valid())
rest_framework/serializers.py:213: in is_valid
    self._validated_data = self.run_validation(self.initial_data)
rest_framework/serializers.py:557: in run_validation
    value = self.to_internal_value(data)
rest_framework/serializers.py:593: in to_internal_value
    validated = self.child.run_validation(item)
rest_framework/serializers.py:409: in run_validation
    self.run_validators(value)
rest_framework/fields.py:499: in run_validators
    validator(value)
rest_framework/validators.py:142: in __call__
    queryset = self.exclude_current_instance(attrs, queryset)
rest_framework/validators.py:135: in exclude_current_instance
    return queryset.exclude(pk=self.instance.pk)
E   AttributeError: 'QuerySet' object has no attribute 'pk'
____________________________________________________ ManyUniqueTogetherTestCase.test_partial_true ____________________________________________________
tests/test_unique_together.py:50: in test_partial_true
    self.assertTrue(serializer.is_valid())
rest_framework/serializers.py:213: in is_valid
    self._validated_data = self.run_validation(self.initial_data)
rest_framework/serializers.py:557: in run_validation
    value = self.to_internal_value(data)
rest_framework/serializers.py:593: in to_internal_value
    validated = self.child.run_validation(item)
rest_framework/serializers.py:409: in run_validation
    self.run_validators(value)
rest_framework/fields.py:499: in run_validators
    validator(value)
rest_framework/validators.py:141: in __call__
    queryset = self.filter_queryset(attrs, queryset)
rest_framework/validators.py:120: in filter_queryset
    attrs[field_name] = getattr(self.instance, field_name)
E   AttributeError: 'QuerySet' object has no attribute 'bar'
```



Discussion:
Presumably if _either_ is read_only, then we shouldn't validate them? (Ie. same as excluding one?)

This is how django does it: https://github.com/django/django/blob/master/django/forms/models.py#L344 (build exclusion list)

Then here: https://github.com/django/django/blob/a10b4c010ab2cdaa6ba8bfaec3e3540299ea77be/django/db/models/base.py#L926
                for name in check:
                    # If this is an excluded field, don't add this check.
                    if name in exclude:
                        break

So It looks like you are correct, a uniquetogether check is not done if ANY of the fields are missing

I can write a PR for this as long as the design is accepted. I just hate writing stuff that won't get approved.

I'm broadly in favor of this, but I don't think we can make any definite decisions without seeing an example fix and test case.

Labeling this as a valid bug to address tho.

Uniqueness validators should also not be run if required=False.  I have a serializer (Room) with required=False set on a nested user serializer, but it fails with a "This field is required" because user is part of a unique_together requirement on the Room model.  Is there a workaround for now to exclude this validation on the serializer?  For now, I've removed the unique_together requirement, but that's obviously less than ideal.

This use case is more complex. You want to run the uniqueness constraint when both have a value though not if only one is provided.

+1.
Both for read_only=True or required=False
When both have a value: you want uniqueness. When only one or none have a value: no constraints.

Behaviour today is that they are both blocked as required when uniqueness is defined, even when the fields are read_only=True or required=False.
(I presume the same should happen if only 1 of the fields is required and the other not?)

`required=False` will be "overridden" by the `UniqueTogetherValidator`.
I start feeling there are too many possible use cases. What if one field is not required, what if one is read only, what if we have both constraints and what if....
I'm not even sure what I should think about a `UniqueTogetherValidator` with one field having a read only constraint on top of it.
Let's rather use a custom `UniqueTogetherValidator` or use business logic level validation and make the documentation more clear about what the `UniqueTogetherValidator` does.
@tomchristie thoughts on that one ?

It comes down to decide which constraint takes precedence?
If both are not required but have uniqueness, they might both be blank (thus technically violating the uniqueness if this happens more than once)

http://stackoverflow.com/questions/5772176/django-unique-together-and-blank-true

Checking on the behaviour of Django, it seems like the uniqueness takes precedence, as it is enforced on DB level.

What I would at least suggest is to handle the exception more gracefully. Now if you have a field that you put required=False but with uniqueness, you get the message "this field is required".

> It comes down to decide which constraint takes precedence?

It comes down to how to handle the most common use cases and easily allow developers to deal with edge cases. To me, it's very similar to the work done on writable nested serializers.

Raising warnings if conflicting constraints are found could be a nice way to make the developers aware that they might be hitting a wall.

I strongly recommend following Django's behavior. Partially for dev sanity, but mainly so unique_together fields can be autogenerated in the save method of the model.

If they are both read_only, then there is no point in validating because there is nothing to be sent from the wire. However, having validating fail, makes it so you cannot autogenerate values, which is very problematic.

> unique_together fields can be autogenerated in the save method of the model.

There's no way DRF can be aware of auto generated data at save time. This basically means the API will raise 500 errors because the unique_together constraint may be broken if the dev forgot to set the data.
This falls back to my previous comment.

> There's no way DRF can be aware of auto generated data at save time. 

Exactly. That's why DRF can't be imposing itself when you declare that you will be managing those fields yourself.

If you disagree, give me one scenario where read_only is set on a field, but DRF itself will save that field based on user input.

@ntucker any create operation on a model that has a unique_together constraint with a read_only field on one of them.

Note that I've been performing something similar but handled it outside of generic DRF.
Reason is, even if `UniqueTogetherConstraint` would work with QuerySet it would still fail in some cases such as:

```
data=[{
                "id": id1,
                "foo": 5,
                "bar": 6,
            }, {
                "id": id2,
                "foo": 5,
                "bar": 6,
            },],
```

I suspect there are some other specific use cases that would prove hard to solve with validators in their current forms.

Could it be related to the issue I'm having ?

I got 

```
author = models.ForeignKey(User)
created_at = models.DateTimeField(auto_now_add=True)

class Meta:
    get_latest_by = "created_at"
    ordering = ['-created_at']
    unique_together = [
        ["author", "created_at"]
    ]
```

in my Model. User is a cms.models.User object.

For some reason, Swagger is throwing an error when trying to list available API endpoints `'CreateOnlyDefault' object has no attribute 'is_update'`

```
Traceback:
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  132.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  58.         return view_func(*args, **kwargs)
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/django/views/generic/base.py" in view
  71.             return self.dispatch(request, *args, **kwargs)
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/rest_framework/views.py" in dispatch
  466.             response = self.handle_exception(exc)
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/rest_framework/views.py" in dispatch
  463.             response = handler(request, *args, **kwargs)
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/rest_framework_swagger/views.py" in get
  164.             'models': generator.get_models(apis),
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/rest_framework_swagger/docgenerator.py" in get_models
  139.             data = self._get_serializer_fields(serializer)
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/rest_framework_swagger/docgenerator.py" in _get_serializer_fields
  328.                 'defaultValue': get_default_value(field),
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/rest_framework_swagger/introspectors.py" in get_default_value
  54.         default_value = default_value()
File "/Users/louis/Documents/WebProjects/Python/SiteTestlib/python2.7/site-packages/rest_framework/fields.py" in __call__
  225.         if self.is_update:
```

I know Django REST Swagger is not part of that project, but the call is made from Django REST Framework if I understand the stack trace correctly.

Related: #2996.


Resolving patch:
diff --git a/rest_framework/serializers.py b/rest_framework/serializers.py
--- a/rest_framework/serializers.py
+++ b/rest_framework/serializers.py
@@ -1243,6 +1243,11 @@ def get_extra_kwargs(self):
 
         read_only_fields = getattr(self.Meta, 'read_only_fields', None)
         if read_only_fields is not None:
+            if not isinstance(read_only_fields, (list, tuple)):
+                raise TypeError(
+                    'The `read_only_fields` option must be a list or tuple. '
+                    'Got %s.' % type(read_only_fields).__name__
+                )
             for field_name in read_only_fields:
                 kwargs = extra_kwargs.get(field_name, {})
                 kwargs['read_only'] = True
@@ -1258,6 +1263,9 @@ def get_uniqueness_extra_kwargs(self, field_names, declared_fields, extra_kwargs
 
         ('dict of updated extra kwargs', 'mapping of hidden fields')
         """
+        if getattr(self.Meta, 'validators', None) is not None:
+            return (extra_kwargs, {})
+
         model = getattr(self.Meta, 'model')
         model_fields = self._get_model_fields(
             field_names, declared_fields, extra_kwargs
@@ -1308,7 +1316,7 @@ def get_uniqueness_extra_kwargs(self, field_names, declared_fields, extra_kwargs
                 else:
                     uniqueness_extra_kwargs[unique_constraint_name] = {'default': default}
             elif default is not empty:
-                # The corresponding field is not present in the,
+                # The corresponding field is not present in the
                 # serializer. We have a default to use for it, so
                 # add in a hidden field that populates it.
                 hidden_fields[unique_constraint_name] = HiddenField(default=default)
@@ -1390,6 +1398,7 @@ def get_unique_together_validators(self):
         field_names = {
             field.source for field in self.fields.values()
             if (field.source != '*') and ('.' not in field.source)
+            and not field.read_only
         }
 
         # Note that we make sure to check `unique_together` both on the


Test patch:
diff --git a/tests/test_model_serializer.py b/tests/test_model_serializer.py
--- a/tests/test_model_serializer.py
+++ b/tests/test_model_serializer.py
@@ -521,8 +521,6 @@ class Meta:
                 one_to_one = NestedSerializer(read_only=True):
                     url = HyperlinkedIdentityField(view_name='onetoonetargetmodel-detail')
                     name = CharField(max_length=100)
-                class Meta:
-                    validators = [<UniqueTogetherValidator(queryset=UniqueTogetherModel.objects.all(), fields=('foreign_key', 'one_to_one'))>]
         """)
         if six.PY2:
             # This case is also too awkward to resolve fully across both py2
diff --git a/tests/test_validators.py b/tests/test_validators.py
--- a/tests/test_validators.py
+++ b/tests/test_validators.py
@@ -239,6 +239,45 @@ class Meta:
         """)
         assert repr(serializer) == expected
 
+    def test_ignore_read_only_fields(self):
+        """
+        When serializer fields are read only, then uniqueness
+        validators should not be added for that field.
+        """
+        class ReadOnlyFieldSerializer(serializers.ModelSerializer):
+            class Meta:
+                model = UniquenessTogetherModel
+                fields = ('id', 'race_name', 'position')
+                read_only_fields = ('race_name',)
+
+        serializer = ReadOnlyFieldSerializer()
+        expected = dedent("""
+            ReadOnlyFieldSerializer():
+                id = IntegerField(label='ID', read_only=True)
+                race_name = CharField(read_only=True)
+                position = IntegerField(required=True)
+        """)
+        assert repr(serializer) == expected
+
+    def test_allow_explict_override(self):
+        """
+        Ensure validators can be explicitly removed..
+        """
+        class NoValidatorsSerializer(serializers.ModelSerializer):
+            class Meta:
+                model = UniquenessTogetherModel
+                fields = ('id', 'race_name', 'position')
+                validators = []
+
+        serializer = NoValidatorsSerializer()
+        expected = dedent("""
+            NoValidatorsSerializer():
+                id = IntegerField(label='ID', read_only=True)
+                race_name = CharField(max_length=100)
+                position = IntegerField()
+        """)
+        assert repr(serializer) == expected
+
     def test_ignore_validation_for_null_fields(self):
         # None values that are on fields which are part of the uniqueness
         # constraint cause the instance to ignore uniqueness validation.