Repository: pydantic/pydantic

Issue:
Sequence constraint to enforce only unique items
### Checks

* [x] I added a descriptive title to this issue
* [x] I have searched (google, github) for similar issues and couldn't find anything
* [x] I have read and followed [the docs](https://pydantic-docs.helpmanual.io/) and still think this feature/change is needed
* [x] After submitting this, I commit to one of:
    * Look through open issues and helped at least one other person
    * Hit the "watch" button on this repo to receive notifications and I commit to help at least 2 people that ask questions in the future
    * Implement a Pull Request for a confirmed bug

<!-- Sorry to sound so draconian, but every second saved replying to issues is time spend improving pydantic :-) -->

# Feature Request

Output of `python -c "import pydantic.utils; print(pydantic.utils.version_info())"`:
```
             pydantic version: 1.6.1
            pydantic compiled: False
                 install path: /Users/yuriikarabas/my-projects/pydantic/pydantic
               python version: 3.6.8 (v3.6.8:3c6b436a57, Dec 24 2018, 02:04:31)  [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)]
                     platform: Darwin-19.6.0-x86_64-i386-64bit
     optional deps. installed: ['typing-extensions', 'email-validator', 'devtools']

```
<!-- or if you're using pydantic prior to v1.3, manually include: OS, python version and pydantic version -->

<!-- Please read the [docs](https://pydantic-docs.helpmanual.io/) and search through issues to
confirm your feature hasn't been asked for before, or already implemented. -->

<!-- Where possible please include a self-contained code snippet describing your feature request: -->

The idea of this feature is to add the ability to validate that sequence types (like `list`, `tuple`, or `set`) contains only unique items.

I think the best API for this feature will be something like this:
```python
from typing import List
from pydantic import BaseModel, EmailStr, Field

class User(BaseModel):
    email: EmailStr


class Form(BaseModel):
    users: List[User] = Field(unique=True)
```

In case when data contains duplicates, they will be available using `errors` method.
For instance:
```python
try:
    f = Form(users=[{'email': 'some@email.com'}] * 2)
except ValidationError as e:
    print(e.errors())
```
Will print:
```python
[
    {
        'loc': ('users',),
        'msg': "value contains duplicate items [User(email='some@email.com')]",
        'type': 'type_error.sequencenotunique',
        'ctx': {'duplicates': [User(email='some@email.com')]}
    },
]
```


Also, as an option, we can add a new generic type `Unique`. 
Usage will look like:
```python
from pydantic import BaseModel, EmailStr, Unique

class User(BaseModel):
    email: EmailStr


class Form(BaseModel):
    users: Unique[User]
```


I have already implemented this feature in my fork. [link to commit](https://github.com/uriyyo/pydantic/commit/bb67a2e9f15da0d826ba48da117230284182c1b3)


Comments:
As a workaround, maybe you could set the attribute type to a set? Then when a list gets passed in, it gets deduplicated automatically. Then you can override `json()` and `dict()` to convert it back to a list on the way out.
@mdavis-xyz The idea of this feature is to validate that sequences have only unique items.

Code below will show the idea:
```python
from typing import Set

from pydantic import BaseModel, Field


class FooForm(BaseModel):
    tags: Set[int]


foo = FooForm(tags=[1, 1, 2])  # tags attribute will be {1, 2}


class BarForm(BaseModel):
    tags: Set[int] = Field(unique=True)


bar = BarForm(tags=[1, 1, 2])  # will fail, because sequence has duplicate items
```

If you have any questions, feel free to ask.
This is an interesting issue that clarifies some of the friction points with validation and casting within Pydantic. 

As a workaround, it's possible to add a validator with 'pre=True' to get the desired behavior. 

```python
from typing import Set, Sequence
from pydantic import BaseModel, Field, validator

def custom_to_set(xs: Sequence[int]) -> Set[int]:
    items = set([])
    for item in xs:
        if item in items:
            raise ValueError(f"Duplicate item {item}")
        else:
            items.add(item)
    return items


class Record(BaseModel):
    tags: Set[int] = Field(...)

    @validator('tags', pre=True)
    def validate_unique_tags(cls, value):
        # raise ValueError or a set
        return custom_to_set(value)

def example() -> int:

    t1 = ['1', '2'] # Notice how liberal Pydantic is. This is not a validation error
    t2 = [1, 3, 1] # This now raises
    for t in [t1, t2]:
        print(Record(tags=t))
    return 0
```


Is this issue a bandaid for the fundamental loose casting/coercion model that Pydantic has adopted?

https://github.com/samuelcolvin/pydantic/issues/1098

Your proposal does seem reasonable and clarifies the default behavior  of `Set[T]` in a "strict" mode should raise a core validation error if the input sequence has any duplicates. However, this is not how Pydantic has approached validation.  Perhaps it would be better to wait until Pydantic has a coherent "strict" model?

With regards to the proposal, it's a bit counterintuitive to define a `Set[int]` and also set `Field(unique=True)` to get the validation to work "correctly".  Similarly, `List[T] = Field(unique=True)` is counterintuitive because it's not clear why a `Set[T]` isn't being used.  This case might be better solved by a custom validation/casting function with `pre=True`? 


@mpkocher  This issue came up when I faced with an issue which I thought was a bug.

I had a model and one of the fields had a type `Set[str]`, the model looked like:
```python
class Form(BaseModel):
    tags: Set[int]
```
When one of the scripts tried to instantiate `Form` with duplicated items it simply cast sequence to a set, which was the wrong behavior for me script. After investigating I found that this is expected behavior, that why I create this PR)

I agree with you that `List[T] = Field(unique=True)` can be a little bit confusing.

Regarding #1098 it looks great and maybe it's better to return to this feature after `Strict Configuration` will be implemented.
I'm running into similar friction points where the overly liberal casting yields surprising results. 

Patch:
diff --git a/pydantic/errors.py b/pydantic/errors.py
--- a/pydantic/errors.py
+++ b/pydantic/errors.py
@@ -48,6 +48,7 @@
     'TupleLengthError',
     'ListMinLengthError',
     'ListMaxLengthError',
+    'ListUniqueItemsError',
     'SetMinLengthError',
     'SetMaxLengthError',
     'FrozenSetMinLengthError',
@@ -324,6 +325,11 @@ def __init__(self, *, limit_value: int) -> None:
         super().__init__(limit_value=limit_value)
 
 
+class ListUniqueItemsError(PydanticValueError):
+    code = 'list.unique_items'
+    msg_template = 'the list has duplicated items'
+
+
 class SetMinLengthError(PydanticValueError):
     code = 'set.min_items'
     msg_template = 'ensure this value has at least {limit_value} items'
diff --git a/pydantic/fields.py b/pydantic/fields.py
--- a/pydantic/fields.py
+++ b/pydantic/fields.py
@@ -103,6 +103,7 @@ class FieldInfo(Representation):
         'multiple_of',
         'min_items',
         'max_items',
+        'unique_items',
         'min_length',
         'max_length',
         'allow_mutation',
@@ -123,6 +124,7 @@ class FieldInfo(Representation):
         'multiple_of': None,
         'min_items': None,
         'max_items': None,
+        'unique_items': None,
         'allow_mutation': True,
     }
 
@@ -143,6 +145,7 @@ def __init__(self, default: Any = Undefined, **kwargs: Any) -> None:
         self.multiple_of = kwargs.pop('multiple_of', None)
         self.min_items = kwargs.pop('min_items', None)
         self.max_items = kwargs.pop('max_items', None)
+        self.unique_items = kwargs.pop('unique_items', None)
         self.min_length = kwargs.pop('min_length', None)
         self.max_length = kwargs.pop('max_length', None)
         self.allow_mutation = kwargs.pop('allow_mutation', True)
@@ -208,6 +211,7 @@ def Field(
     multiple_of: float = None,
     min_items: int = None,
     max_items: int = None,
+    unique_items: bool = None,
     min_length: int = None,
     max_length: int = None,
     allow_mutation: bool = True,
@@ -241,6 +245,12 @@ def Field(
       schema will have a ``maximum`` validation keyword
     :param multiple_of: only applies to numbers, requires the field to be "a multiple of". The
       schema will have a ``multipleOf`` validation keyword
+    :param min_items: only applies to lists, requires the field to have a minimum number of
+      elements. The schema will have a ``minItems`` validation keyword
+    :param max_items: only applies to lists, requires the field to have a maximum number of
+      elements. The schema will have a ``maxItems`` validation keyword
+    :param max_items: only applies to lists, requires the field not to have duplicated
+      elements. The schema will have a ``uniqueItems`` validation keyword
     :param min_length: only applies to strings, requires the field to have a minimum length. The
       schema will have a ``maximum`` validation keyword
     :param max_length: only applies to strings, requires the field to have a maximum length. The
@@ -268,6 +278,7 @@ def Field(
         multiple_of=multiple_of,
         min_items=min_items,
         max_items=max_items,
+        unique_items=unique_items,
         min_length=min_length,
         max_length=max_length,
         allow_mutation=allow_mutation,
diff --git a/pydantic/schema.py b/pydantic/schema.py
--- a/pydantic/schema.py
+++ b/pydantic/schema.py
@@ -953,11 +953,9 @@ def get_annotation_from_field_info(
     :return: the same ``annotation`` if unmodified or a new annotation with validation in place
     """
     constraints = field_info.get_constraints()
-
     used_constraints: Set[str] = set()
     if constraints:
         annotation, used_constraints = get_annotation_with_constraints(annotation, field_info)
-
     if validate_assignment:
         used_constraints.add('allow_mutation')
 
@@ -1001,9 +999,18 @@ def go(type_: Any) -> Type[Any]:
             if is_union(origin):
                 return Union[tuple(go(a) for a in args)]  # type: ignore
 
-            if issubclass(origin, List) and (field_info.min_items is not None or field_info.max_items is not None):
-                used_constraints.update({'min_items', 'max_items'})
-                return conlist(go(args[0]), min_items=field_info.min_items, max_items=field_info.max_items)
+            if issubclass(origin, List) and (
+                field_info.min_items is not None
+                or field_info.max_items is not None
+                or field_info.unique_items is not None
+            ):
+                used_constraints.update({'min_items', 'max_items', 'unique_items'})
+                return conlist(
+                    go(args[0]),
+                    min_items=field_info.min_items,
+                    max_items=field_info.max_items,
+                    unique_items=field_info.unique_items,
+                )
 
             if issubclass(origin, Set) and (field_info.min_items is not None or field_info.max_items is not None):
                 used_constraints.update({'min_items', 'max_items'})
diff --git a/pydantic/types.py b/pydantic/types.py
--- a/pydantic/types.py
+++ b/pydantic/types.py
@@ -540,15 +540,18 @@ class ConstrainedList(list):  # type: ignore
 
     min_items: Optional[int] = None
     max_items: Optional[int] = None
+    unique_items: Optional[bool] = None
     item_type: Type[T]  # type: ignore
 
     @classmethod
     def __get_validators__(cls) -> 'CallableGenerator':
         yield cls.list_length_validator
+        if cls.unique_items:
+            yield cls.unique_items_validator
 
     @classmethod
     def __modify_schema__(cls, field_schema: Dict[str, Any]) -> None:
-        update_not_none(field_schema, minItems=cls.min_items, maxItems=cls.max_items)
+        update_not_none(field_schema, minItems=cls.min_items, maxItems=cls.max_items, uniqueItems=cls.unique_items)
 
     @classmethod
     def list_length_validator(cls, v: 'Optional[List[T]]') -> 'Optional[List[T]]':
@@ -566,10 +569,22 @@ def list_length_validator(cls, v: 'Optional[List[T]]') -> 'Optional[List[T]]':
 
         return v
 
+    @classmethod
+    def unique_items_validator(cls, v: 'List[T]') -> 'List[T]':
+        for i, value in enumerate(v, start=1):
+            if value in v[i:]:
+                raise errors.ListUniqueItemsError()
+
+        return v
 
-def conlist(item_type: Type[T], *, min_items: int = None, max_items: int = None) -> Type[List[T]]:
+
+def conlist(
+    item_type: Type[T], *, min_items: int = None, max_items: int = None, unique_items: bool = None
+) -> Type[List[T]]:
     # __args__ is needed to conform to typing generics api
-    namespace = {'min_items': min_items, 'max_items': max_items, 'item_type': item_type, '__args__': (item_type,)}
+    namespace = dict(
+        min_items=min_items, max_items=max_items, unique_items=unique_items, item_type=item_type, __args__=(item_type,)
+    )
     # We use new_class to be able to deal with Generic types
     return new_class('ConstrainedListValue', (ConstrainedList,), {}, lambda ns: ns.update(namespace))
 


Test patch:
diff --git a/tests/test_types.py b/tests/test_types.py
--- a/tests/test_types.py
+++ b/tests/test_types.py
@@ -216,6 +216,39 @@ class ConListModelMin(BaseModel):
     ]
 
 
+def test_constrained_list_not_unique_hashable_items():
+    class ConListModelUnique(BaseModel):
+        v: conlist(int, unique_items=True)
+
+    with pytest.raises(ValidationError) as exc_info:
+        ConListModelUnique(v=[1, 1, 2, 2, 2, 3])
+    assert exc_info.value.errors() == [
+        {
+            'loc': ('v',),
+            'msg': 'the list has duplicated items',
+            'type': 'value_error.list.unique_items',
+        }
+    ]
+
+
+def test_constrained_list_not_unique_unhashable_items():
+    class ConListModelUnique(BaseModel):
+        v: conlist(Set[int], unique_items=True)
+
+    m = ConListModelUnique(v=[{1}, {2}, {3}])
+    assert m.v == [{1}, {2}, {3}]
+
+    with pytest.raises(ValidationError) as exc_info:
+        ConListModelUnique(v=[{1}, {1}, {2}, {2}, {2}, {3}])
+    assert exc_info.value.errors() == [
+        {
+            'loc': ('v',),
+            'msg': 'the list has duplicated items',
+            'type': 'value_error.list.unique_items',
+        }
+    ]
+
+
 def test_constrained_list_optional():
     class Model(BaseModel):
         req: Optional[conlist(str, min_items=1)] = ...
@@ -296,8 +329,8 @@ class ConListModel(BaseModel):
 
 def test_conlist():
     class Model(BaseModel):
-        foo: List[int] = Field(..., min_items=2, max_items=4)
-        bar: conlist(str, min_items=1, max_items=4) = None
+        foo: List[int] = Field(..., min_items=2, max_items=4, unique_items=True)
+        bar: conlist(str, min_items=1, max_items=4, unique_items=False) = None
 
     assert Model(foo=[1, 2], bar=['spoon']).dict() == {'foo': [1, 2], 'bar': ['spoon']}
 
@@ -307,12 +340,29 @@ class Model(BaseModel):
     with pytest.raises(ValidationError, match='ensure this value has at most 4 items'):
         Model(foo=list(range(5)))
 
+    with pytest.raises(ValidationError, match='the list has duplicated items'):
+        Model(foo=[1, 1, 2, 2])
+
     assert Model.schema() == {
         'title': 'Model',
         'type': 'object',
         'properties': {
-            'foo': {'title': 'Foo', 'type': 'array', 'items': {'type': 'integer'}, 'minItems': 2, 'maxItems': 4},
-            'bar': {'title': 'Bar', 'type': 'array', 'items': {'type': 'string'}, 'minItems': 1, 'maxItems': 4},
+            'foo': {
+                'title': 'Foo',
+                'type': 'array',
+                'items': {'type': 'integer'},
+                'minItems': 2,
+                'maxItems': 4,
+                'uniqueItems': True,
+            },
+            'bar': {
+                'title': 'Bar',
+                'type': 'array',
+                'items': {'type': 'string'},
+                'minItems': 1,
+                'maxItems': 4,
+                'uniqueItems': False,
+            },
         },
         'required': ['foo'],
     }