Repository: pydantic/pydantic

Issue:
Failed to create schema with optional fields
Just tried to use schema creation and it's awesome, but I found, I think, bug with optional fields.

Here is my model:
```
from typing import Optional
from pydantic import BaseModel

class TestModel(BaseModel):
    foo: Optional[int] = None

print(TestModel.schema_json(indent=2))
```

Output:
```
Traceback (most recent call last):
  File "test_pydantic_schema.py", line 9, in <module>
    print(TestModel.schema_json(indent=2))
  File "/pydantic/pydantic/main.py", line 288, in schema_json
    return json.dumps(cls.schema(by_alias=by_alias), default=pydantic_encoder, **dumps_kwargs)
  File "/pydantic/pydantic/main.py", line 278, in schema
    s['properties'] = {f.alias: f.schema(by_alias) for f in cls.__fields__.values()}
  File "/pydantic/pydantic/main.py", line 278, in <dictcomp>
    s['properties'] = {f.alias: f.schema(by_alias) for f in cls.__fields__.values()}
  File "/pydantic/pydantic/fields.py", line 146, in schema
    if issubclass(self.type_, Enum):
TypeError: issubclass() arg 1 must be a class
```

Same issue with `Union`:
```
from typing import Union
from pydantic import BaseModel

class TestModel(BaseModel):
    foo: Union[str, int]

print(TestModel.schema_json(indent=2))
```

I think `pydantic` should cover cases like this, I don't know what is the best way to solve issue with `Optional`, but in case of `Union` `oneOf` or `anyOf` rule can be used http://json-schema.org/latest/json-schema-validation.html#rfc.section.6.7.


Discussion:
yes.

I guess `Optional` is just equivalent of `Union[None, Foobar]`.

As discussed on #129 I'm not that bothered by `json-schema` or it's equivalents, but yes, something like that makes sense.

Perhaps we just have `"type": ["int", "float"]` and an array for `type` just means "union"?
For JSON Schemas, "union" means:
```
{
    "type": "object",
    "properties": {
        "foo": {
            "anyOf": [
                { "type": "int },
                { "type": "float" },
            ]
        }
    }
}
```
Also I don't think that we need to use `anyOf` for `Optional`, I think better to convert this case to property with `default=null`.
I'm still not sure how to display unions, I don't think we should return multiple instances of the entire field definition, eg. if we have

```json
  "Gender": {
    "type": "int",
    "title": "Gender",
    "required": false,
  }
```

and the value can also be a string, I think

```json
  "Gender": {
    "type": ["int", "str"],
    "title": "Gender",
    "required": false,
  }
```
or
```json
  "Gender": {
    "type": {"anyOf": ["int", "str"]},
    "title": "Gender",
    "required": false,
  }
```
is clearer than
```json
  "Gender": {
    "anyOf": [
      {
        "type": "int",
        "title": "Gender",
        "required": false,
      },
      {
        "type": "str",
        "title": "Gender",
        "required": false,
      },
    ]
  }
```

But that get's more complicated in strange situations like `Union[str, FoobarModel]`

What do you think?
ok, after looking at #218 I propose the following way to deal with this:

`type` becomes a dictionary. 

For "simple" fields, eg. just an int or float, type like:

```py
'type': {
  'item_type': 'float',
}
```

For unions we'd have:

```py
'type': {
  'mode': 'any_of',
  'item_type': ['str', 'float'],
}
```

For things like `List[int]` we'd have:

```py
'type': {
  'item_type': 'int',
  'shape': 'list',
}
```

or `List[Union[int, str]]` we'd have:

```py
'type': {
  'mode': 'any_of',
  'item_type': ['int', 'str'],
  'shape': 'list',
}
```

For `Dict[str, int]` we'd have:

```py
'type': {
  'item_type': 'int',
  'key_type': 'str',
  'shape': 'dict',
}
```

(we could add `key_mode` if the key is a union, but that's a very rare case)

The names `mode`, `item_type` and `shape` are not great (suggestions very welcome), but I think this is the best way to go.

Questions:
* are their any case's I've missed?
* do we do `type: {'item_type': 'int', ...}` or just add all these items to the main dict?
* what do we do when implement tuples #12, I guess something like `properties` on sub models?
* what do we do with `Union[int, FoobarModel]` - I guess we have `type` as above, then `properties`. What do we do with `Union[FooModel, BarModel]` - maybe this is just banned?

@Gr1N @x1ah
May missed `enum` type ?

In my opinion, we should carefully change the field structure. and `typing.Union` 's field info looks good.

```python
from typing import Union
from pydantic import BaseModel


class Foo(BaseModel):
    ubar = Union[int, str]

print(Foo.__fields__['r'].info)
# {'type': 'typing.Union[int, str]',
#  'required': True,
#  'sub_fields': [<Field r_int: type='int', required=True, validators=['int_validator']>,
#   <Field r_str: type='str', required=True, validators=['not_none_validator', 'str_validator','anystr_strip_whitespace', 'anystr_length_validator']>]}
```

and similar for `List[int, str]`, we get

```python
{'type': 'typing.List[int]',
 'required': True,
 'sub_fields': [<Field r_int: type='int', ...>]}
```

by this way, this problem will be solved without changing the original fields struct, just put into `sub_fields`.

What do you think? @samuelcolvin 
Just copying python's syntax is ok with me, it'll make it slightly harder for people to read programmatically, but it'll be more concise and extendible.

The case of enums is already implemented via `choices`.
> are their any case's I've missed?

What about `Optional[any type]`?

And one question, why we want to invent new format? Why not use JSON Schema spec? I understand that spec in some cases can be very explicit, but it covers all needs to my mind. Also it's good to follow specification in cases if we want to generate API specs from `pydantic` models and give them to another team who works, for example, with different language, and if generated specs follow JSON schema spec we can use them to generate code for any language we want.
I think your suggestion of `default=null` for `Optional` is good.

> And one question, why we want to invent new format? Why not use JSON Schema spec?

see https://github.com/samuelcolvin/pydantic/issues/129#issuecomment-364754565, also:
1. It would be masses of work (perhaps more than the rest of pydantic combined) to fully support JSON schema, which is only draft anyway.
1. pydantic isn't tied to JSON and I think we would run into lots of conflicts, eg. (AFAIK) json schema doesn't cover lots of types with pydantic supports, eg. decimal, uuid, dict keys have to be strings in json
Understand. Can you please share how cases `Union[int, FoobarModel]` and `Union[FooModel, BarModel]` will be looks in proposed format?
Ok, I've redesigned it, (maybe we're now back at JSON schema? :smile:):

simple:
```py
'type': {
  'type': 'float',
}
```

`Union[float, int]`:
```py
'type': {
  'type': 'any_of',
  'types': [
    {
      'type': 'float',
    },
    {
      'type': 'int',
    },
  ],
}
```

`List[int]`:
```py
'type': {
  'type': 'list',
  'item_type': {
    'type': 'int',
  },
}
```

`List[Union[int, str]]`:
```py
'type': {
  'type': 'list',
  'item_type': {
    'mode': 'any_of',
    'types': [
      {
      'type': 'float',
      },
      {
      'type': 'int',
      },
    ],
  }
}
```

`Dict[str, int]`:
```py
'type': {
  'type': 'dict',
  'item_type': {
    'type': 'int',
  },
  'key_type': {
    'type': 'str',
  },
  'shape': 'dict',
}
```

`FooModel`:
```py
'type': {
  'type': 'object',
  'properties': {
    ...
  }
}
```

`Enum(int)`:
```py
'type': {
  'type': 'int',
  'choices': [
      [1, 'Male'],
      [2, 'Female'],
      [3, 'Other'],
  ]
}
```

Then `Union[int, FoobarModel]` is just:
```py
'type': {
  'mode': 'any_of',
  'types': [
    {
      'type': 'int',
    },
      'type': 'object',
      'properties': {
        ...
      }
    }
  ],
}
```

and `Union[FooModel, BarModel]` is:
```py
'type': {
  'type': 'any_of',
  'types': [
    {
      'type': 'object',
      'properties': {
        ...
      }
    },
      'type': 'object',
      'properties': {
        ...
      }
    }
  ],
}
```

In other words types are recursive; `any_of`, `list` and `dict` are just types of type.

For a whole field, we then get:
```py
'age': {
  'type': {
    'type': 'float',
  },
  'name': 'Age',
  'required': True,
}
```

What do you think?

--------------------------------

Update: correct object.
Update 2: `mode` > `type`
LGTM, let's try to go with your proposal.
as per discussion on gitter:
* the top level property should change `type` > `schema`
* there should be a shortened display of just `'schema': {'type': 'int'}` which is just `'schema': 'int'`, this applies both at the top level for simple types and where simple types are nested within more complex types.

Overal the design is encapsulated by:

```py
'foobar': {
  'schema': {
    'type': 'list',
    'item_type': {
      'mode': 'any_of',
      'types': [
        'float', 'int',
      ],
    }
  },
  'name': 'Foobar',
  'required': True,
}
```


Resolving patch:
diff --git a/pydantic/fields.py b/pydantic/fields.py
--- a/pydantic/fields.py
+++ b/pydantic/fields.py
@@ -136,31 +136,77 @@ def prepare(self):
         self._populate_validators()
 
     def schema(self, by_alias=True):
-        s = self.type_.schema(by_alias) if hasattr(self.type_, 'schema') else {}
-        s.update(
-            type=s.get('type') or display_as_type(self.type_),
-            title=self._schema.title or s.get('title') or self.alias.title(),
+        s = dict(
+            title=self._schema.title or self.alias.title(),
             required=self.required,
         )
 
         if not self.required and self.default is not None:
             s['default'] = self.default
-        if issubclass(self.type_, Enum):
-            choice_names = self._schema.choice_names or {}
-            s['choices'] = [
-                (v.value, choice_names.get(v.value) or k.title())
-                for k, v in self.type_.__members__.items()
-            ]
         s.update(self._schema.extra)
+
+        ts = self.type_schema(by_alias)
+        s.update(ts if isinstance(ts, dict) else {'type': ts})
         return s
 
+    def type_schema(self, by_alias):
+        if self.shape is Shape.LIST:
+            return {
+                'type': 'list',
+                'item_type': self._singleton_schema(by_alias),
+            }
+        if self.shape is Shape.SET:
+            return {
+                'type': 'set',
+                'item_type': self._singleton_schema(by_alias),
+            }
+        elif self.shape is Shape.MAPPING:
+            return {
+                'type': 'mapping',
+                'item_type': self._singleton_schema(by_alias),
+                'key_type': self.key_field.type_schema(by_alias)
+            }
+        elif self.shape is Shape.TUPLE:
+            return {
+                'type': 'tuple',
+                'item_types': [sf.type_schema(by_alias) for sf in self.sub_fields],
+            }
+        else:
+            assert self.shape is Shape.SINGLETON, self.shape
+            return self._singleton_schema(by_alias)
+
+    def _singleton_schema(self, by_alias):
+        if self.sub_fields:
+            if len(self.sub_fields) == 1:
+                return self.sub_fields[0].type_schema(by_alias)
+            else:
+                return {
+                    'type': 'any_of',
+                    'types': [sf.type_schema(by_alias) for sf in self.sub_fields]
+                }
+        elif issubclass(self.type_, Enum):
+            choice_names = self._schema.choice_names or {}
+            return {
+                'type': display_as_type(self.type_),
+                'choices': [
+                    (v.value, choice_names.get(v.value) or k.title())
+                    for k, v in self.type_.__members__.items()
+                ]
+            }
+
+        type_schema_method = getattr(self.type_, 'type_schema', None)
+        if callable(type_schema_method):
+            return type_schema_method(by_alias)
+        else:
+            return display_as_type(self.type_)
+
     def _populate_sub_fields(self):
         # typing interface is horrible, we have to do some ugly checks
         if isinstance(self.type_, type) and issubclass(self.type_, JsonWrapper):
             self.type_ = self.type_.inner_type
             self.parse_json = True
 
-        origin = _get_type_origin(self.type_)
+        origin = getattr(self.type_, '__origin__', None)
         if origin is None:
             # field is not "typing" object eg. Union, Dict, List etc.
             return
@@ -169,6 +215,7 @@ def _populate_sub_fields(self):
             for type_ in self.type_.__args__:
                 if type_ is NoneType:
                     self.allow_none = True
+                    self.required = False
                 else:
                     types_.append(type_)
             self.sub_fields = [self._create_sub_type(t, f'{self.name}_{display_as_type(t)}') for t in types_]
@@ -193,7 +240,7 @@ def _populate_sub_fields(self):
             self.type_ = self.type_.__args__[1]
             self.shape = Shape.MAPPING
 
-        if _get_type_origin(self.type_):
+        if getattr(self.type_, '__origin__', None):
             # type_ has been refined eg. as the type of a List and sub_fields needs to be populated
             self.sub_fields = [self._create_sub_type(self.type_, '_' + self.name)]
 
@@ -413,10 +460,3 @@ def _get_validator_signature(validator):
         raise errors_.ConfigError(f'Invalid signature for validator {validator}: {signature}, should be: '
                                   f'(value) or (value, *, values, config, field) or for class validators '
                                   f'(cls, value) or (cls, value, *, values, config, field)') from e
-
-
-def _get_type_origin(obj):
-    """
-    Like obj.__class__ or type(obj) but for typing objects
-    """
-    return getattr(obj, '__origin__', None)
diff --git a/pydantic/main.py b/pydantic/main.py
--- a/pydantic/main.py
+++ b/pydantic/main.py
@@ -276,24 +276,27 @@ def copy(self, *, include: Set[str]=None, exclude: Set[str]=None, update: Dict[s
     def fields(self):
         return self.__fields__
 
+    @classmethod
+    def type_schema(cls, by_alias):
+        return {
+            'type': 'object',
+            'properties': (
+                {f.alias: f.schema(by_alias) for f in cls.__fields__.values()}
+                if by_alias else
+                {k: f.schema(by_alias) for k, f in cls.__fields__.items()}
+            )
+        }
+
     @classmethod
     def schema(cls, by_alias=True) -> Dict[str, Any]:
         cached = cls._schema_cache.get(by_alias)
         if cached is not None:
             return cached
-
-        s = {
-            'type': 'object',
-            'title': cls.__config__.title or cls.__name__,
-        }
+        s = {'title': cls.__config__.title or cls.__name__}
         if cls.__doc__:
             s['description'] = clean_docstring(cls.__doc__)
 
-        if by_alias:
-            s['properties'] = {f.alias: f.schema(by_alias) for f in cls.__fields__.values()}
-        else:
-            s['properties'] = {k: f.schema(by_alias) for k, f in cls.__fields__.items()}
-
+        s.update(cls.type_schema(by_alias))
         cls._schema_cache[by_alias] = s
         return s
 
diff --git a/pydantic/version.py b/pydantic/version.py
--- a/pydantic/version.py
+++ b/pydantic/version.py
@@ -2,4 +2,4 @@
 
 __all__ = ['VERSION']
 
-VERSION = StrictVersion('0.11.2')
+VERSION = StrictVersion('0.12a1')


Test patch:
diff --git a/tests/test_complex.py b/tests/test_complex.py
--- a/tests/test_complex.py
+++ b/tests/test_complex.py
@@ -1,7 +1,7 @@
 import re
 from decimal import Decimal
 from enum import Enum
-from typing import Any, Dict, List, Set, Tuple, Union
+from typing import Any, Dict, List, Optional, Set, Tuple, Union
 
 import pytest
 
@@ -674,3 +674,12 @@ class Model(BaseModel):
             'type': 'value_error.missing'
         }
     ]
+
+
+def test_optional_required():
+    class Model(BaseModel):
+        bar: Optional[int]
+
+    assert Model(bar=123).dict() == {'bar': 123}
+    assert Model().dict() == {'bar': None}
+    assert Model(bar=None).dict() == {'bar': None}
diff --git a/tests/test_schema.py b/tests/test_schema.py
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -1,6 +1,7 @@
 import json
 from decimal import Decimal
 from enum import Enum, IntEnum
+from typing import Dict, List, Optional, Set, Tuple, Union
 
 import pytest
 
@@ -17,8 +18,6 @@ class ApplePie(BaseModel):
 
     s = {
         'type': 'object',
-        'title': 'ApplePie',
-        'description': 'This is a test.',
         'properties': {
             'a': {
                 'type': 'float',
@@ -32,6 +31,8 @@ class ApplePie(BaseModel):
                 'default': 10,
             },
         },
+        'title': 'ApplePie',
+        'description': 'This is a test.',
     }
     assert True not in ApplePie._schema_cache
     assert False not in ApplePie._schema_cache
@@ -93,7 +94,7 @@ class Bar(BaseModel):
             },
             'b': {
                 'type': 'object',
-                'title': 'Foo',
+                'title': 'B',
                 'properties': {
                     'b': {
                         'type': 'float',
@@ -101,7 +102,6 @@ class Bar(BaseModel):
                         'required': True,
                     },
                 },
-                'description': 'hello',
                 'required': False,
             },
         },
@@ -205,21 +205,179 @@ class Model(BaseModel):
 
     assert Model.schema_json(indent=2) == (
         '{\n'
-        '  "type": "object",\n'
         '  "title": "Model",\n'
+        '  "type": "object",\n'
         '  "properties": {\n'
         '    "a": {\n'
-        '      "type": "bytes",\n'
         '      "title": "A",\n'
         '      "required": false,\n'
-        '      "default": "foobar"\n'
+        '      "default": "foobar",\n'
+        '      "type": "bytes"\n'
         '    },\n'
         '    "b": {\n'
-        '      "type": "Decimal",\n'
         '      "title": "B",\n'
         '      "required": false,\n'
-        '      "default": 12.34\n'
+        '      "default": 12.34,\n'
+        '      "type": "Decimal"\n'
         '    }\n'
         '  }\n'
         '}'
     )
+
+
+def test_list_sub_model():
+    class Foo(BaseModel):
+        a: float
+
+    class Bar(BaseModel):
+        b: List[Foo]
+
+    assert Bar.schema() == {
+        'title': 'Bar',
+        'type': 'object',
+        'properties': {
+            'b': {
+                'type': 'list',
+                'item_type': {
+                    'type': 'object',
+                    'properties': {
+                        'a': {
+                            'type': 'float',
+                            'title': 'A',
+                            'required': True,
+                        },
+                    },
+                },
+                'title': 'B',
+                'required': True,
+            },
+        },
+    }
+
+
+def test_optional():
+    class Model(BaseModel):
+        a: Optional[str]
+
+    assert Model.schema() == {
+        'title': 'Model',
+        'type': 'object',
+        'properties': {
+            'a': {
+                'type': 'str',
+                'title': 'A',
+                'required': False,
+            },
+        },
+    }
+
+
+def test_set():
+    class Model(BaseModel):
+        a: Set[int]
+
+    assert Model.schema() == {
+        'title': 'Model',
+        'type': 'object',
+        'properties': {
+            'a': {
+                'title': 'A',
+                'required': True,
+                'type': 'set',
+                'item_type': 'int'
+            }
+        }
+    }
+
+
+def test_tuple():
+    class Model(BaseModel):
+        a: Tuple[str, int, Union[str, int, float], float]
+
+    assert Model.schema() == {
+        'title': 'Model',
+        'type': 'object',
+        'properties': {
+            'a': {
+                'title': 'A',
+                'required': True,
+                'type': 'tuple',
+                'item_types': [
+                    'str',
+                    'int',
+                    {
+                        'type': 'any_of',
+                        'types': [
+                            'str',
+                            'int',
+                            'float',
+                        ],
+                    },
+                    'float',
+                ],
+            },
+        },
+    }
+
+
+def test_list_union_dict():
+    class Foo(BaseModel):
+        a: float
+
+    class Model(BaseModel):
+        """party time"""
+        a: Union[int, str]
+        b: List[int]
+        c: Dict[int, Foo]
+        d: Union[None, Foo]
+
+    assert Model.schema() == {
+        'title': 'Model',
+        'description': 'party time',
+        'type': 'object',
+        'properties': {
+            'a': {
+                'title': 'A',
+                'required': True,
+                'type': 'any_of',
+                'types': [
+                    'int',
+                    'str',
+                ],
+            },
+            'b': {
+                'title': 'B',
+                'required': True,
+                'type': 'list',
+                'item_type': 'int',
+            },
+            'c': {
+                'title': 'C',
+                'required': True,
+                'type': 'mapping',
+                'item_type': {
+                    'type': 'object',
+                    'properties': {
+                        'a': {
+                            'title': 'A',
+                            'required': True,
+                            'type': 'float',
+                        },
+                    },
+                },
+                'key_type': 'int',
+            },
+            'd': {
+                'title': 'D',
+                'required': False,
+                'type': 'object',
+                'properties': {
+                    'a': {
+                        'title': 'A',
+                        'required': True,
+                        'type': 'float',
+                    },
+                },
+            },
+        },
+    }