Repository: django-json-api/django-rest-framework-json-api

Issue:
Including data fails on lists
If the inclusion parameter is used in retrieving a list of models, every relation is only considered once (utils.py line 417 removes the relation once)

``` python
        try:
            included_resources.remove(field_name)
            new_included_resources = [key.replace('%s.' % field_name, '', 1) for key in included_resources]
            relation_instance_or_manager = getattr(resource_instance, field_name)
            serializer_data = resource.get(field_name)
        except ValueError:
            # Skip fields not in requested included resources
            continue
```

Example:

Assuming table `Person`:

| pk | fruit (fk to `Fruit`) | Person |
| --- | --- | --- |
| 1 | 1 | John |
| 2 | 1 | Jack |
| 3 | 2 | Jill |

and table `Fruit`:

| pk | name |
| --- | --- |
| 1 | apple |
| 2 | banana |

Now let's do `GET /api/v1/person?include=fruit` which is programmed to return all persons

Gives roughly following result (I wrote this from memory ):

``` json
    {
        "data": [{
                "type": "persons",
                "id": 1,
                "attributes": { "name": "John" },
                "relationships": {
                    "fruit": {
                        "data": { "type": "fruit", "id": "1" }
                    }
                }
            },
            {
                "type": "persons",
                "id": 2,
                "attributes": { "name": "Jack" },
                "relationships": {
                    "fruit": {
                        "data": { "type": "fruit", "id": "1" }
                    }
                }
            },
            {
                "type": "persons",
                "id": 3,
                "attributes": { "name": "Jill" },
                "relationships": {
                    "fruit": {
                        "data": { "type": "fruit", "id": "2" }
                    }
                }
            }
        # so far so good, now here it gets interesting
        ],
        "included": [{
            "type": "fruit",
            "id": 1,
            "attributes": {
                "name": "apple"
            }
        }]
        # banana is missing, even though the person 3 references it
    },
```

Including data fails on lists
If the inclusion parameter is used in retrieving a list of models, every relation is only considered once (utils.py line 417 removes the relation once)

``` python
        try:
            included_resources.remove(field_name)
            new_included_resources = [key.replace('%s.' % field_name, '', 1) for key in included_resources]
            relation_instance_or_manager = getattr(resource_instance, field_name)
            serializer_data = resource.get(field_name)
        except ValueError:
            # Skip fields not in requested included resources
            continue
```

Example:

Assuming table `Person`:

| pk | fruit (fk to `Fruit`) | Person |
| --- | --- | --- |
| 1 | 1 | John |
| 2 | 1 | Jack |
| 3 | 2 | Jill |

and table `Fruit`:

| pk | name |
| --- | --- |
| 1 | apple |
| 2 | banana |

Now let's do `GET /api/v1/person?include=fruit` which is programmed to return all persons

Gives roughly following result (I wrote this from memory ):

``` json
    {
        "data": [{
                "type": "persons",
                "id": 1,
                "attributes": { "name": "John" },
                "relationships": {
                    "fruit": {
                        "data": { "type": "fruit", "id": "1" }
                    }
                }
            },
            {
                "type": "persons",
                "id": 2,
                "attributes": { "name": "Jack" },
                "relationships": {
                    "fruit": {
                        "data": { "type": "fruit", "id": "1" }
                    }
                }
            },
            {
                "type": "persons",
                "id": 3,
                "attributes": { "name": "Jill" },
                "relationships": {
                    "fruit": {
                        "data": { "type": "fruit", "id": "2" }
                    }
                }
            }
        # so far so good, now here it gets interesting
        ],
        "included": [{
            "type": "fruit",
            "id": 1,
            "attributes": {
                "name": "apple"
            }
        }]
        # banana is missing, even though the person 3 references it
    },
```



Discussion:
I think the problem is that the renderer passes the `included_resources` by reference (lines 110 and 119 in renderers.py). Another way to fix this for my test case would be to pass by value:

``` python
included = utils.extract_included(fields, resource, resource_instance, list(included_resources))
```

Edit: the recursive calls to extract_included need that too. 

Could you install the latest from the develop branch and test it? The symptoms you describe sound suspiciously similar to a bug I fixed yesterday in the same area of the code. 

@schtibe would it be possible for you to write a failing test for your use case? Here is the file it should go in and you can copy the existing `multiple_entries` test: https://github.com/django-json-api/django-rest-framework-json-api/blob/develop/example/tests/integration/test_includes.py

I wrote a test and a fix for the `include` bug I was seeing and I didn't run across the issue you're reporting. If we can get a test around it we can fix it.

@jerel sure, check out PR #140

@schtibe thank you!

I think the problem is that the renderer passes the `included_resources` by reference (lines 110 and 119 in renderers.py). Another way to fix this for my test case would be to pass by value:

``` python
included = utils.extract_included(fields, resource, resource_instance, list(included_resources))
```

Edit: the recursive calls to extract_included need that too. 

Could you install the latest from the develop branch and test it? The symptoms you describe sound suspiciously similar to a bug I fixed yesterday in the same area of the code. 

@schtibe would it be possible for you to write a failing test for your use case? Here is the file it should go in and you can copy the existing `multiple_entries` test: https://github.com/django-json-api/django-rest-framework-json-api/blob/develop/example/tests/integration/test_includes.py

I wrote a test and a fix for the `include` bug I was seeing and I didn't run across the issue you're reporting. If we can get a test around it we can fix it.

@jerel sure, check out PR #140

@schtibe thank you!


Resolving patch:
diff --git a/rest_framework_json_api/utils.py b/rest_framework_json_api/utils.py
--- a/rest_framework_json_api/utils.py
+++ b/rest_framework_json_api/utils.py
@@ -2,6 +2,7 @@
 Utils.
 """
 import copy
+
 import inflection
 from django.conf import settings
 from django.utils import six, encoding
@@ -410,6 +411,7 @@ def extract_included(fields, resource, resource_instance, included_resources):
     current_serializer = fields.serializer
     context = current_serializer.context
     included_serializers = get_included_serializers(current_serializer)
+    included_resources = copy.copy(included_resources)
 
     for field_name, field in six.iteritems(fields):
         # Skip URL field


Test patch:
diff --git a/example/tests/integration/test_includes.py b/example/tests/integration/test_includes.py
--- a/example/tests/integration/test_includes.py
+++ b/example/tests/integration/test_includes.py
@@ -1,20 +1,29 @@
-import pytest, json
+import pytest
 from django.core.urlresolvers import reverse
+
 from example.tests.utils import load_json
 
 pytestmark = pytest.mark.django_db
 
 
 def test_included_data_on_list(multiple_entries, client):
-    multiple_entries[1].comments = []
-    response = client.get(reverse("entry-list") + '?include=comments')
+    response = client.get(reverse("entry-list") + '?include=comments&page_size=5')
     included = load_json(response.content).get('included')
 
-    assert [x.get('type') for x in included] == ['comments']
+    assert len(load_json(response.content)['data']) == len(multiple_entries), 'Incorrect entry count'
+    assert [x.get('type') for x in included] == ['comments', 'comments'], 'List included types are incorrect'
+
+    comment_count = len([resource for resource in included if resource["type"] == "comments"])
+    expected_comment_count = sum([entry.comment_set.count() for entry in multiple_entries])
+    assert comment_count == expected_comment_count, 'List comment count is incorrect'
+
 
 def test_included_data_on_detail(single_entry, client):
     response = client.get(reverse("entry-detail", kwargs={'pk': single_entry.pk}) + '?include=comments')
     included = load_json(response.content).get('included')
 
-    assert [x.get('type') for x in included] == ['comments']
+    assert [x.get('type') for x in included] == ['comments'], 'Detail included types are incorrect'
 
+    comment_count = len([resource for resource in included if resource["type"] == "comments"])
+    expected_comment_count = single_entry.comment_set.count()
+    assert comment_count == expected_comment_count, 'Detail comment count is incorrect'