Repository: pydantic/pydantic

Issue:
Optional fields loose some validation
# Bug

* OS: **Ubuntu 18.04**
* Python version: **3.7.3**
* Pydantic versions: **0.30(pypi), 1.0a1 (master)**

Example code:

```py
from pydantic import BaseModel, Schema
from typing import Optional

class MyModel(BaseModel):
    my_int: Optional[int] = Schema(..., ge=3)

MyModel(my_int=2)
```
I would expect:
```
pydantic.error_wrappers.ValidationError: 1 validation error
my_int
  ensure this value is greater than or equal to 3 (type=value_error.number.not_ge; limit_value=3)
```
but `MyModel` gets instantiated just fine.

If I make `my_int` not optional, the behaviour is as expected.


Comments:
@vdwees I've traced the source of this issue to [this line](https://github.com/samuelcolvin/pydantic/blob/5015a7e48bc869adf99b78eb38075951549e9ea7/pydantic/schema.py#L833).

The problem here is that `Optional[X]` is not considered by python to be a `type`. I definitely don't think the way it is currently handled is ideal, but it might be somewhat involved to fix this properly.

For now, here's a workaround:
```python
from pydantic import BaseModel, Schema, ConstrainedInt
from typing import Optional

class ConInt3(ConstrainedInt):
    ge = 3

class MyModel(BaseModel):
    my_int: Optional[ConInt3] = Schema(...)

print(MyModel.schema())
# {'title': 'MyModel', 'type': 'object', 'properties': {'my_int': {'title': 'My_Int', 'minimum': 3, 'type': 'integer'}}}

print(MyModel(my_int=2))
"""
pydantic.error_wrappers.ValidationError: 2 validation errors for MyModel
my_int
  ensure this value is greater than or equal to 3 (type=value_error.number.not_ge; limit_value=3)
my_int
  value is not none (type=type_error.none.allowed)
"""
```

-------

@samuelcolvin I'm not sure whether it would be better to modify the `get_annotation_from_schema` function to deal with the whole `typing` introspection rigamarole (like in, e.g., `Field._populate_sub_fields`), or to just require more careful handling (like in the "workaround" I provided above). Thoughts?
Thanks for reporting. Definitely looks like a bug, I'll look into in and see what I can do.
Firstly thank you both for your contributions to this library, and +1 for resolving this issue if possible.

@dmontagu I have tried your workaround but I believe it has uncovered a second bug - if you look at your code snippet the schema now raises two issues when I would expect one (i.e. I assume it should not complain that the optional value is not none).

For a simpler example (using Pydantic 0.32.2)on MacOSX:

```
>>> from pydantic import BaseModel, PositiveInt
>>> from typing import Optional
>>> class ExampleModel(BaseModel):
...     foo: Optional[PositiveInt]
...
>>> ExampleModel(foo=0)
pydantic.error_wrappers.ValidationError: 2 validation errors for ExampleModel
foo
  ensure this value is greater than 0 (type=value_error.number.not_gt; limit_value=0)
foo
  value is not none (type=type_error.none.allowed)
```
hi @dannymilsom I'm not clear what second bug you've uncovered? Please can you elaborate.
@dannymilsom what's happening is that it's showing you, for each option in the union, why it can't be that option. The first option was PositiveInt, which fails. The second is None, which also fails (obviously).

This is because `Optional[PositiveInt]` is basically the same as `Union[PositiveInt, None]`.

@samuelcolvin I feel like maybe some of this stuff has changed in v1 anyway; not sure whether the error messages still look the same? Either way, I don't think this is a buggy error message, just a slightly confusing one.
@dmontagu yes, you're right on all: it's not an error and it's clearer in v1.

It's also not related in anyway to this issue.

Patch:
diff --git a/docs/examples/constrained_types.py b/docs/examples/constrained_types.py
--- a/docs/examples/constrained_types.py
+++ b/docs/examples/constrained_types.py
@@ -12,6 +12,7 @@
     conint,
     conlist,
     constr,
+    Field,
 )
 
 class Model(BaseModel):
@@ -40,25 +41,4 @@ class Model(BaseModel):
     decimal_max_digits_and_places: condecimal(max_digits=2, decimal_places=2)
     mod_decimal: condecimal(multiple_of=Decimal('0.25'))
 
-m = Model(
-    short_bytes=b'foo',
-    strip_bytes=b'   bar',
-    short_str='foo',
-    regex_str='apple pie',
-    strip_str='   bar',
-    big_int=1001,
-    mod_int=155,
-    pos_int=1,
-    neg_int=-1,
-    big_float=1002.1,
-    mod_float=1.5,
-    pos_float=2.2,
-    neg_float=-2.3,
-    unit_interval=0.5,
-    short_list=[1, 2],
-    decimal_positive='21.12',
-    decimal_negative='-21.12',
-    decimal_max_digits_and_places='0.99',
-    mod_decimal='2.75',
-)
-print(m.dict())
+    bigger_int: int = Field(..., gt=10000)
diff --git a/docs/examples/unenforced_constraints.py b/docs/examples/unenforced_constraints.py
new file mode 100644
--- /dev/null
+++ b/docs/examples/unenforced_constraints.py
@@ -0,0 +1,26 @@
+from pydantic import BaseModel, Field, PositiveInt
+
+try:
+    # this won't work since PositiveInt takes precedence over the
+    # constraints defined in Field meaning they're ignored
+    class Model(BaseModel):
+        foo: PositiveInt = Field(..., lt=10)
+except ValueError as e:
+    print(e)
+
+# but you can set the schema attribute directly:
+# (Note: here exclusiveMaximum will not be enforce)
+class Model(BaseModel):
+    foo: PositiveInt = Field(..., exclusiveMaximum=10)
+
+print(Model.schema())
+
+# if you find yourself needing this, an alternative is to declare
+# the constraints in Field (or you could use conint())
+# here both constraints will be enforced:
+class Model(BaseModel):
+    # Here both constraints will be applied and the schema
+    # will be generated correctly
+    foo: int = Field(..., gt=0, lt=10)
+
+print(Model.schema())
diff --git a/pydantic/fields.py b/pydantic/fields.py
--- a/pydantic/fields.py
+++ b/pydantic/fields.py
@@ -23,7 +23,7 @@
 from .error_wrappers import ErrorWrapper
 from .errors import NoneIsNotAllowedError
 from .types import Json, JsonWrapper
-from .typing import AnyType, Callable, ForwardRef, display_as_type, is_literal_type
+from .typing import AnyType, Callable, ForwardRef, NoneType, display_as_type, is_literal_type
 from .utils import PyObjectStr, Representation, lenient_issubclass, sequence_like
 from .validators import constant_validator, dict_validator, find_validators, validate_json
 
@@ -33,7 +33,6 @@
     Literal = None  # type: ignore
 
 Required: Any = Ellipsis
-NoneType = type(None)
 
 if TYPE_CHECKING:
     from .class_validators import ValidatorsList  # noqa: F401
@@ -256,7 +255,7 @@ def infer(
             field_info = FieldInfo(value, **field_info_from_config)
         field_info.alias = field_info.alias or field_info_from_config.get('alias')
         required = value == Required
-        annotation = get_annotation_from_field_info(annotation, field_info)
+        annotation = get_annotation_from_field_info(annotation, field_info, name)
         return cls(
             name=name,
             type_=annotation,
diff --git a/pydantic/schema.py b/pydantic/schema.py
--- a/pydantic/schema.py
+++ b/pydantic/schema.py
@@ -6,12 +6,38 @@
 from enum import Enum
 from ipaddress import IPv4Address, IPv4Interface, IPv4Network, IPv6Address, IPv6Interface, IPv6Network
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional, Sequence, Set, Tuple, Type, TypeVar, Union, cast
+from typing import (
+    TYPE_CHECKING,
+    Any,
+    Callable,
+    Dict,
+    FrozenSet,
+    List,
+    Optional,
+    Sequence,
+    Set,
+    Tuple,
+    Type,
+    TypeVar,
+    Union,
+    cast,
+)
 from uuid import UUID
 
 from .class_validators import ROOT_KEY
 from .color import Color
-from .fields import SHAPE_LIST, SHAPE_MAPPING, SHAPE_SET, SHAPE_SINGLETON, SHAPE_TUPLE, FieldInfo, ModelField
+from .fields import (
+    SHAPE_FROZENSET,
+    SHAPE_LIST,
+    SHAPE_MAPPING,
+    SHAPE_SEQUENCE,
+    SHAPE_SET,
+    SHAPE_SINGLETON,
+    SHAPE_TUPLE,
+    SHAPE_TUPLE_ELLIPSIS,
+    FieldInfo,
+    ModelField,
+)
 from .json import pydantic_encoder
 from .networks import AnyUrl, EmailStr, IPvAnyAddress, IPvAnyInterface, IPvAnyNetwork, NameEmail
 from .types import (
@@ -30,13 +56,21 @@
     SecretBytes,
     SecretStr,
     StrictBool,
+    conbytes,
     condecimal,
     confloat,
     conint,
-    conlist,
     constr,
 )
-from .typing import Literal, is_callable_type, is_literal_type, is_new_type, literal_values, new_type_supertype
+from .typing import (
+    ForwardRef,
+    Literal,
+    is_callable_type,
+    is_literal_type,
+    is_new_type,
+    literal_values,
+    new_type_supertype,
+)
 from .utils import lenient_issubclass
 
 if TYPE_CHECKING:
@@ -352,14 +386,14 @@ def field_type_schema(
     definitions = {}
     nested_models: Set[str] = set()
     ref_prefix = ref_prefix or default_prefix
-    if field.shape == SHAPE_LIST:
+    if field.shape in {SHAPE_LIST, SHAPE_TUPLE_ELLIPSIS, SHAPE_SEQUENCE}:
         f_schema, f_definitions, f_nested_models = field_singleton_schema(
             field, by_alias=by_alias, model_name_map=model_name_map, ref_prefix=ref_prefix, known_models=known_models
         )
         definitions.update(f_definitions)
         nested_models.update(f_nested_models)
         return {'type': 'array', 'items': f_schema}, definitions, nested_models
-    elif field.shape == SHAPE_SET:
+    elif field.shape in {SHAPE_SET, SHAPE_FROZENSET}:
         f_schema, f_definitions, f_nested_models = field_singleton_schema(
             field, by_alias=by_alias, model_name_map=model_name_map, ref_prefix=ref_prefix, known_models=known_models
         )
@@ -723,31 +757,64 @@ def encode_default(dft: Any) -> Any:
 _map_types_constraint: Dict[Any, Callable[..., type]] = {int: conint, float: confloat, Decimal: condecimal}
 
 
-def get_annotation_from_field_info(annotation: Any, field_info: FieldInfo) -> Type[Any]:
+def get_annotation_from_field_info(annotation: Any, field_info: FieldInfo, field_name: str) -> Type[Any]:  # noqa: C901
     """
     Get an annotation with validation implemented for numbers and strings based on the field_info.
 
     :param annotation: an annotation from a field specification, as ``str``, ``ConstrainedStr``
     :param field_info: an instance of FieldInfo, possibly with declarations for validations and JSON Schema
+    :param field_name: name of the field for use in error messages
     :return: the same ``annotation`` if unmodified or a new annotation with validation in place
     """
-    if isinstance(annotation, type):
+    constraints = {f for f in validation_attribute_to_schema_keyword if getattr(field_info, f) is not None}
+    if not constraints:
+        return annotation
+    used_constraints: Set[str] = set()
+
+    def go(type_: Any) -> Type[Any]:
+        if is_literal_type(annotation) or isinstance(type_, ForwardRef) or lenient_issubclass(type_, ConstrainedList):
+            return type_
+        origin = getattr(type_, '__origin__', None)
+        if origin is not None:
+            args: Tuple[Any, ...] = type_.__args__
+            if any(isinstance(a, ForwardRef) for a in args):
+                # forward refs cause infinite recursion below
+                return type_
+
+            if origin is Union:
+                return Union[tuple(go(a) for a in args)]
+
+            # conlist isn't working properly with schema #913
+            # if issubclass(origin, List):
+            #     used_constraints.update({'min_items', 'max_items'})
+            #     return conlist(go(args[0]), min_items=field_info.min_items, max_items=field_info.max_items)
+
+            for t in (Tuple, List, Set, FrozenSet, Sequence):
+                if issubclass(origin, t):  # type: ignore
+                    return t[tuple(go(a) for a in args)]  # type: ignore
+
+            if issubclass(origin, Dict):
+                return Dict[args[0], go(args[1])]  # type: ignore
+
         attrs: Optional[Tuple[str, ...]] = None
         constraint_func: Optional[Callable[..., type]] = None
-        if issubclass(annotation, str) and not issubclass(annotation, (EmailStr, AnyUrl, ConstrainedStr)):
-            attrs = ('max_length', 'min_length', 'regex')
-            constraint_func = constr
-        elif lenient_issubclass(annotation, numeric_types) and not issubclass(
-            annotation, (ConstrainedInt, ConstrainedFloat, ConstrainedDecimal, ConstrainedList, bool)
-        ):
-            # Is numeric type
-            attrs = ('gt', 'lt', 'ge', 'le', 'multiple_of')
-            numeric_type = next(t for t in numeric_types if issubclass(annotation, t))  # pragma: no branch
-            constraint_func = _map_types_constraint[numeric_type]
-        elif issubclass(annotation, ConstrainedList):
-            attrs = ('min_items', 'max_items')
-            constraint_func = conlist
+        if isinstance(type_, type):
+            if issubclass(type_, str) and not issubclass(type_, (EmailStr, AnyUrl, ConstrainedStr)):
+                attrs = ('max_length', 'min_length', 'regex')
+                constraint_func = constr
+            elif issubclass(type_, bytes):
+                attrs = ('max_length', 'min_length', 'regex')
+                constraint_func = conbytes
+            elif issubclass(type_, numeric_types) and not issubclass(
+                type_, (ConstrainedInt, ConstrainedFloat, ConstrainedDecimal, ConstrainedList, bool)
+            ):
+                # Is numeric type
+                attrs = ('gt', 'lt', 'ge', 'le', 'multiple_of')
+                numeric_type = next(t for t in numeric_types if issubclass(type_, t))  # pragma: no branch
+                constraint_func = _map_types_constraint[numeric_type]
+
         if attrs:
+            used_constraints.update(set(attrs))
             kwargs = {
                 attr_name: attr
                 for attr_name, attr in ((attr_name, getattr(field_info, attr_name)) for attr_name in attrs)
@@ -756,7 +823,19 @@ def get_annotation_from_field_info(annotation: Any, field_info: FieldInfo) -> Ty
             if kwargs:
                 constraint_func = cast(Callable[..., type], constraint_func)
                 return constraint_func(**kwargs)
-    return annotation
+        return type_
+
+    ans = go(annotation)
+
+    unused_constraints = constraints - used_constraints
+    if unused_constraints:
+        raise ValueError(
+            f'On field "{field_name}" the following field constraints are set but not enforced: '
+            f'{", ".join(unused_constraints)}. '
+            f'\nFor more details see https://pydantic-docs.helpmanual.io/usage/schema/#unenforced-field-constraints'
+        )
+
+    return ans
 
 
 class SkipField(Exception):
diff --git a/pydantic/typing.py b/pydantic/typing.py
--- a/pydantic/typing.py
+++ b/pydantic/typing.py
@@ -72,6 +72,7 @@ def evaluate_forwardref(type_, globalns, localns):  # type: ignore
     'Callable',
     'AnyCallable',
     'AnyType',
+    'NoneType',
     'display_as_type',
     'resolve_annotations',
     'is_callable_type',
@@ -96,6 +97,7 @@ def evaluate_forwardref(type_, globalns, localns):  # type: ignore
 
 
 AnyType = Type[Any]
+NoneType = type(None)
 
 
 def display_as_type(v: AnyType) -> str:


Test patch:
diff --git a/tests/test_edge_cases.py b/tests/test_edge_cases.py
--- a/tests/test_edge_cases.py
+++ b/tests/test_edge_cases.py
@@ -19,7 +19,7 @@
     validate_model,
     validator,
 )
-from pydantic.fields import Schema
+from pydantic.fields import Field, Schema
 
 
 def test_str_bytes():
@@ -1122,6 +1122,22 @@ def alias_generator(cls, f_name):
     assert Child.__fields__['x'].alias == 'x2'
 
 
+def test_optional_field_constraints():
+    class MyModel(BaseModel):
+        my_int: Optional[int] = Field(..., ge=3)
+
+    with pytest.raises(ValidationError) as exc_info:
+        MyModel(my_int=2)
+    assert exc_info.value.errors() == [
+        {
+            'loc': ('my_int',),
+            'msg': 'ensure this value is greater than or equal to 3',
+            'type': 'value_error.number.not_ge',
+            'ctx': {'limit_value': 3},
+        }
+    ]
+
+
 def test_field_str_shape():
     class Model(BaseModel):
         a: List[int]
diff --git a/tests/test_forward_ref.py b/tests/test_forward_ref.py
--- a/tests/test_forward_ref.py
+++ b/tests/test_forward_ref.py
@@ -419,3 +419,23 @@ class Account(BaseModel):
             },
         },
     }
+
+
+def test_forward_ref_with_field(create_module):
+    create_module(
+        """
+from typing import List
+from pydantic import BaseModel, Field
+from pydantic.typing import ForwardRef
+
+Foo = ForwardRef('Foo')
+
+try:
+    class Foo(BaseModel):
+        c: List[Foo] = Field(..., gt=0)
+except ValueError:
+    pass
+else:
+    raise AssertionError('error not raised')
+    """
+    )
diff --git a/tests/test_schema.py b/tests/test_schema.py
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -11,7 +11,7 @@
 
 import pytest
 
-from pydantic import BaseModel, Extra, Field, ValidationError, validator
+from pydantic import BaseModel, Extra, Field, ValidationError, conlist, validator
 from pydantic.color import Color
 from pydantic.networks import AnyUrl, EmailStr, IPvAnyAddress, IPvAnyInterface, IPvAnyNetwork, NameEmail, stricturl
 from pydantic.schema import (
@@ -1092,7 +1092,7 @@ class UserModel(BaseModel):
     'kwargs,type_,expected_extra',
     [
         ({'max_length': 5}, str, {'type': 'string', 'maxLength': 5}),
-        ({'max_length': 5}, constr(max_length=6), {'type': 'string', 'maxLength': 6}),
+        ({}, constr(max_length=6), {'type': 'string', 'maxLength': 6}),
         ({'min_length': 2}, str, {'type': 'string', 'minLength': 2}),
         ({'max_length': 5}, bytes, {'type': 'string', 'maxLength': 5, 'format': 'binary'}),
         ({'regex': '^foo$'}, str, {'type': 'string', 'pattern': '^foo$'}),
@@ -1128,41 +1128,35 @@ class Foo(BaseModel):
 
 
 @pytest.mark.parametrize(
-    'kwargs,type_,expected',
+    'kwargs,type_',
     [
-        ({'max_length': 5}, int, {'type': 'integer'}),
-        ({'min_length': 2}, float, {'type': 'number'}),
-        ({'max_length': 5}, Decimal, {'type': 'number'}),
-        ({'regex': '^foo$'}, int, {'type': 'integer'}),
-        ({'gt': 2}, str, {'type': 'string'}),
-        ({'lt': 5}, bytes, {'type': 'string', 'format': 'binary'}),
-        ({'ge': 2}, str, {'type': 'string'}),
-        ({'le': 5}, bool, {'type': 'boolean'}),
+        ({'max_length': 5}, int),
+        ({'min_length': 2}, float),
+        ({'max_length': 5}, Decimal),
+        ({'regex': '^foo$'}, int),
+        ({'gt': 2}, str),
+        ({'lt': 5}, bytes),
+        ({'ge': 2}, str),
+        ({'le': 5}, bool),
+        ({'gt': 0}, Callable),
+        ({'gt': 0}, Callable[[int], int]),
+        ({'gt': 0}, conlist(int, min_items=4)),
     ],
 )
-def test_not_constraints_schema(kwargs, type_, expected):
-    class Foo(BaseModel):
-        a: type_ = Field('foo', title='A title', description='A description', **kwargs)
-
-    base_schema = {
-        'title': 'Foo',
-        'type': 'object',
-        'properties': {'a': {'title': 'A title', 'description': 'A description', 'default': 'foo'}},
-    }
+def test_unenforced_constraints_schema(kwargs, type_):
+    with pytest.raises(ValueError, match='On field "a" the following field constraints are set but not enforced'):
 
-    base_schema['properties']['a'].update(expected)
-    assert Foo.schema() == base_schema
+        class Foo(BaseModel):
+            a: type_ = Field('foo', title='A title', description='A description', **kwargs)
 
 
 @pytest.mark.parametrize(
     'kwargs,type_,value',
     [
         ({'max_length': 5}, str, 'foo'),
-        ({'max_length': 5}, constr(max_length=6), 'foo'),
         ({'min_length': 2}, str, 'foo'),
         ({'max_length': 5}, bytes, b'foo'),
         ({'regex': '^foo$'}, str, 'foo'),
-        ({'max_length': 5}, bool, True),
         ({'gt': 2}, int, 3),
         ({'lt': 5}, int, 3),
         ({'ge': 2}, int, 3),
@@ -1508,3 +1502,95 @@ class Config:
         'required': ['a'],
         'additionalProperties': False,
     }
+
+
+@pytest.mark.parametrize(
+    'annotation,kwargs,field_schema',
+    [
+        (int, dict(gt=0), {'title': 'A', 'exclusiveMinimum': 0, 'type': 'integer'}),
+        (Optional[int], dict(gt=0), {'title': 'A', 'exclusiveMinimum': 0, 'type': 'integer'}),
+        (
+            Tuple[int, ...],
+            dict(gt=0),
+            {'title': 'A', 'exclusiveMinimum': 0, 'type': 'array', 'items': {'exclusiveMinimum': 0, 'type': 'integer'}},
+        ),
+        (
+            Tuple[int, int, int],
+            dict(gt=0),
+            {
+                'title': 'A',
+                'type': 'array',
+                'items': [
+                    {'exclusiveMinimum': 0, 'type': 'integer'},
+                    {'exclusiveMinimum': 0, 'type': 'integer'},
+                    {'exclusiveMinimum': 0, 'type': 'integer'},
+                ],
+            },
+        ),
+        (
+            Union[int, float],
+            dict(gt=0),
+            {
+                'title': 'A',
+                'anyOf': [{'exclusiveMinimum': 0, 'type': 'integer'}, {'exclusiveMinimum': 0, 'type': 'number'}],
+            },
+        ),
+        (
+            List[int],
+            dict(gt=0),
+            {'title': 'A', 'exclusiveMinimum': 0, 'type': 'array', 'items': {'exclusiveMinimum': 0, 'type': 'integer'}},
+        ),
+        (
+            Dict[str, int],
+            dict(gt=0),
+            {
+                'title': 'A',
+                'exclusiveMinimum': 0,
+                'type': 'object',
+                'additionalProperties': {'exclusiveMinimum': 0, 'type': 'integer'},
+            },
+        ),
+        (
+            Union[str, int],
+            dict(gt=0, max_length=5),
+            {'title': 'A', 'anyOf': [{'maxLength': 5, 'type': 'string'}, {'exclusiveMinimum': 0, 'type': 'integer'}]},
+        ),
+    ],
+)
+def test_enforced_constraints(annotation, kwargs, field_schema):
+    class Model(BaseModel):
+        a: annotation = Field(..., **kwargs)
+
+    schema = Model.schema()
+    # debug(schema['properties']['a'])
+    assert schema['properties']['a'] == field_schema
+
+
+def test_real_vs_phony_constraints():
+    class Model1(BaseModel):
+        foo: int = Field(..., gt=123)
+
+        class Config:
+            title = 'Test Model'
+
+    class Model2(BaseModel):
+        foo: int = Field(..., exclusiveMinimum=123)
+
+        class Config:
+            title = 'Test Model'
+
+    with pytest.raises(ValidationError, match='ensure this value is greater than 123'):
+        Model1(foo=122)
+
+    assert Model2(foo=122).dict() == {'foo': 122}
+
+    assert (
+        Model1.schema()
+        == Model2.schema()
+        == {
+            'title': 'Test Model',
+            'type': 'object',
+            'properties': {'foo': {'title': 'Foo', 'exclusiveMinimum': 123, 'type': 'integer'}},
+            'required': ['foo'],
+        }
+    )