Repository: marshmallow-code/marshmallow

Issue:
No attribute '_add_to_schema' when dumping Schema and Nested Field is None
Hi Marshmallow Code!

I have recently discovered and began implementing your code for my own projects and I must say you have changed how I work with databases forever! 

I have been running up against a wall with this issue however, and decided to reach out to the pros! It could be that I am missing something simple.

For context, I am storing hosts in a database. 'Hosts' have a one-to-many relationship with 'Ports'. 

I have developed a HostSchema and PortSchema using ModelSchema from marshmallow-sqlalchemy. This works BEAUTIFULLY, except when a host does not have any ports associated with it. 

Attached below is example code which demonstrates the expected behavior as well as the error. Please excuse me if this isn't an actual bug and just user error, although some insight would be very much appreciated.

### Truncated Pip Freeze
- Flask==1.0.2
- flask-marshmallow==0.9.0
- Flask-SQLAlchemy==2.3.2
- marshmallow==2.15.4
- marshmallow-sqlalchemy==0.14.1
- SQLAlchemy==1.2.11

### Minimal Example
```py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_marshmallow import Marshmallow

from marshmallow import fields, post_dump, pre_dump
import json

app = Flask(__name__)
db = SQLAlchemy(app)
ma = Marshmallow(app)

class Host(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    ip = db.Column(db.String(16), index=True, unique=True, nullable=False)

    ports = db.relationship('Port', backref='host', lazy='dynamic')


class Port(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    host_id = db.Column(db.Integer, db.ForeignKey('host.id'), index=True)
    port = db.Column(db.Integer)
    transport = db.Column(db.String(8))
    service = db.Column(db.String(16))


class PortSchema(ma.ModelSchema):
    class Meta:
        model = Port
        exclude = ('id' , 'host')
        sqla_session = db.Session

class HostSchema(ma.ModelSchema):
    ports = fields.Nested(PortSchema, many=True)
    class Meta:
        model = Host
        sqla_session = db.Session
```
### Behavior when 'ports' is defined in 'host' object.
This works as expected.

```py
host = Host(ip='127.0.0.1')
port = Port(port=80, transport='tcp', service='http')
host.ports.append(port)

result = HostSchema().dump(host)

print json.dumps(result.data, indent=2)


# {
#   "ip": "127.0.0.1",
#   "ports": [
#     {
#       "transport": "tcp",
#       "service": "http",
#       "port": 80
#     }
#   ],
#   "id": null
# }
```

### Unexpected behavior when 'ports' is undefined in 'host' object.
From what I've gathered from the documentation the default behavior is to skip any fields which are missing during serialization. I have also tried using `missing` and `default`, but if I understand correctly these only apply for deserialization. 

```py
host = Host(ip='127.0.0.1')
result = HostSchema().dump(host)
```

### Expected Behavior
```py
{
  "ip": "127.0.0.1",
  "ports": {},
  "id": null
}
```
### Actual result:
```py
Traceback (most recent call last):
  File "scratch/missing-field-nested-bug.py", line 72, in <module>
    result = HostSchema().dump(host)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/schema.py", line 509, in dump
    **kwargs
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/marshalling.py", line 138, in serialize
    index=(index if index_errors else None)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/marshalling.py", line 62, in call_and_store
    value = getter_func(data)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/marshalling.py", line 132, in <lambda>
    getter = lambda d: field_obj.serialize(attr_name, d, accessor=accessor)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/fields.py", line 252, in serialize
    return self._serialize(value, attr, obj)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/fields.py", line 447, in _serialize
    schema._update_fields(obj=nested_obj, many=self.many)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/schema.py", line 767, in _update_fields
    self.__set_field_attrs(ret)
  File "/Users/kai/.virtualenvs/SPEZZI/lib/python2.7/site-packages/marshmallow/schema.py", line 788, in __set_field_attrs
    field_obj._add_to_schema(field_name, self)
AttributeError: 'NoneType' object has no attribute '_add_to_schema'
```


Thank you all for any help or insights you can offer! Please let me know if further clarification is required.

Discussion:

Patch:
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index aff06634..d30adf1c 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -1,6 +1,14 @@
 Changelog
 ---------
 
+2.15.5
++++++++++++++++++++
+
+Bug fixes:
+
+- Handle empty SQAlchemy lazy lists gracefully when dumping (:issue:`948`).
+  Thanks :user:`vke-code` for the catch and :user:`YuriHeupa` for the patch.
+
 2.15.4 (2018-08-04)
 +++++++++++++++++++
 
diff --git a/marshmallow/schema.py b/marshmallow/schema.py
index 90e84bff..1d2a71db 100644
--- a/marshmallow/schema.py
+++ b/marshmallow/schema.py
@@ -819,7 +819,7 @@ class BaseSchema(base.SchemaABC):
                 else:
                     obj_prototype = next(iter(obj))
             except (StopIteration, IndexError):  # Nothing to serialize
-                return self.declared_fields
+                return dict((k, v) for k, v in self.declared_fields.items() if k in field_names)
             obj = obj_prototype
         ret = self.dict_class()
         for key in field_names:


Test patch:
diff --git a/tests/test_schema.py b/tests/test_schema.py
index 5a6fd959..7f0e8cb8 100755
--- a/tests/test_schema.py
+++ b/tests/test_schema.py
@@ -694,6 +694,36 @@ def test_error_raised_if_additional_option_is_not_list():
                 additional = 'email'
 
 
+def test_nested_custom_set_in_exclude_reusing_schema():
+
+    class CustomSet(object):
+        # This custom set is to allow the obj check in BaseSchema.__filter_fields
+        # to pass, since it'll be a valid instance, and this class overrides
+        # getitem method to allow the hasattr check to pass too, which will try
+        # to access the first obj index and will simulate a IndexError throwing.
+        # e.g. SqlAlchemy.Query is a valid use case for this "obj".
+
+        def __getitem__(self, item):
+            return [][item]
+
+    class ChildSchema(Schema):
+        foo = fields.Field(required=True)
+        bar = fields.Field()
+
+        class Meta:
+            only = ('bar', )
+
+    class ParentSchema(Schema):
+        child = fields.Nested(ChildSchema, many=True, exclude=('foo',))
+
+    sch = ParentSchema(strict=True)
+    obj = dict(child=CustomSet())
+    sch.dumps(obj)
+    data = dict(child=[{'bar': 1}])
+    result = sch.load(data, partial=True)
+    assert not result.errors
+
+
 def test_nested_only():
     class ChildSchema(Schema):
         foo = fields.Field()