Repository: django-json-api/django-rest-framework-json-api

Issue:
Release 7.0.0 breaks sparse fields request
## Description of the Bug Report

As of release 7.0.0 a sparse fields request throws a KeyError for `url` at https://github.com/django-json-api/django-rest-framework-json-api/blob/366769e55e821eba11928125c903ad5043f9ff43/rest_framework_json_api/renderers.py#L485-L486 This was working up through release 6.1.0.

I expect that this is related to this change identified in the CHANGELOG:
https://github.com/django-json-api/django-rest-framework-json-api/blob/366769e55e821eba11928125c903ad5043f9ff43/CHANGELOG.md?plain=1#L27

How to reproduce:
```
git clone https://github.com/columbia-it/django-jsonapi-training
cd django-jsonapi-training
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip freeze | grep json
# see the version is djangorestframework-jsonapi==7.0.0
./manage.py migrate
./manage.py loaddata myapp/fixtures/*yaml
openssl genrsa -out oidc.key 4096
OIDC_RSA_PRIVATE_KEY_FILE=oidc.key ./manage.py runserver
```
Use Postman with OAuth2 as [described here](https://columbia-it-django-jsonapi-training.readthedocs.io/en/latest/using_dot/#get-an-oauth-20-token):
```
Token Name: *pick a name*
Grant Type: Authorization Code (With PKCE)
Callback URL: https://www.getpostman.com/oauth2/callback
Auth URL: http://localhost:8000/authorize/
Access Token URL: http://localhost:8000/token/
Client ID: demo_djt_web_client
Client Secret: demo_djt_web_secret
Scope: auth-columbia demo-djt-sla-bronze read openid profile email https://api.columbia.edu/scope/group
Client Authentication: Send as Basic Auth header
```

Click **Get New Access Token** and login as user `admin` pass `admin123` and **Authorize** then proceed and **Use Token**

Send GET http://localhost:8000/v1/courses/?fields[courses]=course_name

observe this error:
```
  File "/Users/ac45/src/django-jsonapi-training/venv/lib/python3.12/site-packages/rest_framework_json_api/renderers.py", line 486, in build_json_resource_obj
    fields[api_settings.URL_FIELD_NAME], relations.RelatedField
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'url'
```

Then send GET http://localhost:8000//v1/courses/?fields[courses]=course_name,url

and get back a correct JSONAPI response.

Now ^C the runserver, downgrade to v6.1.0, and try again.

```
pip install djangorestframework-jsonapi==6.1.0
OIDC_RSA_PRIVATE_KEY_FILE=oidc.key ./manage.py runserver

Send GET http://localhost:8000/v1/courses/?fields[courses]=course_name

and get back a correct JSONAPI response.

## Checklist

- [x] Certain that this is a bug (if unsure or you have a question use [discussions](https://github.com/django-json-api/django-rest-framework-json-api/discussions) instead)
- [x] Code snippet or unit test added to reproduce bug



Discussion:
Thanks for the heads up. It seems in PR #1221 the url gets filtered out in the renderer but kept in the readable fields on the serialzer. Also actually id should be kept for overwriting id on a serialzer basis and not being filtered out when using sparse fields.

Strange that the tests did not catch it. Do you see what the difference of your view and our sparse fieldset test could be?

I will certainly need to analyze it more closely.
I’ll try to code a test case in the next few days.

On Fri, May 31, 2024 at 9:29 AM Oliver Sauder ***@***.***>
wrote:

> Thanks for the heads up. It seems in PR #1221
> <https://github.com/django-json-api/django-rest-framework-json-api/pull/1221>
> the url gets filtered out in the renderer but kept in the readable fields
> on the serialzer. Also actually id should be kept for overwriting id on a
> serialzer basis and not being filtered out when using sparse fields.
>
> Strange that the tests did not catch it. Do you see what the difference of
> your view and our sparse fieldset test could be?
>
> I will certainly need to analyze it more closely.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/django-json-api/django-rest-framework-json-api/issues/1228#issuecomment-2142168345>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABBHS557INPIYDGGNA4SEBLZFB3J5AVCNFSM6AAAAABIRWAH5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBSGE3DQMZUGU>
> .
> You are receiving this because you authored the thread.Message ID:
> <django-json-api/django-rest-framework-json-api/issues/1228/2142168345@
> github.com>
>

I think the issue is the HyperlinkedModelSerializer which is not tested in our test suite :sniff:
Yes agree. I can code a test.

On Fri, May 31, 2024 at 10:20 AM Oliver Sauder ***@***.***>
wrote:

> I think the issue is the HyperlinkedModelSerializer which is not tested in
> our test suite :sniff:
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/django-json-api/django-rest-framework-json-api/issues/1228#issuecomment-2142330666>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABBHS53UZ36DCR2I4LDLRY3ZFCBLXAVCNFSM6AAAAABIRWAH5OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBSGMZTANRWGY>
> .
> You are receiving this because you authored the thread.Message ID:
> <django-json-api/django-rest-framework-json-api/issues/1228/2142330666@
> github.com>
>

sure sounds good. In case I will get to it earlier I will let you know.
@sliverc I've been staring at these tests for a while and can't come up with a proper breaking test. Any suggestions?
In the `tests/tests_views.py` we could add another serializer/view at the bottom to the test routing whereas the serializer is derived from `HyperlinkedModelSerializer`, which I would hope should expose the issue?

I might also see whether I get some more time to look into it later on today.



> In the `tests/tests_views.py` we could add another serializer/view at the bottom to the test routing whereas the serializer is derived from `HyperlinkedModelSerializer`, which I would hope should expose the issue?
> 
> I might also see whether I get some more time to look into it later on today.

Yeah I tried a few permutations of that but I think it also has to have a ResourceRelatedField and a ForeignKey model *and* using sparse fieldsets query.... I can try some more.
I have some time at hand, let me give it a shot and I will let you know.

Resolving patch:
diff --git a/rest_framework_json_api/renderers.py b/rest_framework_json_api/renderers.py
--- a/rest_framework_json_api/renderers.py
+++ b/rest_framework_json_api/renderers.py
@@ -446,7 +446,10 @@ def _filter_sparse_fields(cls, serializer, fields, resource_name):
                 return {
                     field_name: field
                     for field_name, field, in fields.items()
-                    if field_name in sparse_fields
+                    if field.field_name in sparse_fields
+                    # URL field is not considered a field in JSON:API spec
+                    # but a link so need to keep it
+                    or field.field_name == api_settings.URL_FIELD_NAME
                 }
 
         return fields
diff --git a/rest_framework_json_api/serializers.py b/rest_framework_json_api/serializers.py
--- a/rest_framework_json_api/serializers.py
+++ b/rest_framework_json_api/serializers.py
@@ -94,10 +94,15 @@ def _readable_fields(self):
                         field
                         for field in readable_fields
                         if field.field_name in sparse_fields
+                        # URL field is not considered a field in JSON:API spec
+                        # but a link so need to keep it
                         or field.field_name == api_settings.URL_FIELD_NAME
+                        # ID is a required field which might have been overwritten
+                        # so need to keep it
+                        or field.field_name == "id"
                     )
             except AttributeError:
-                # no type on serializer, must be used only as only nested
+                # no type on serializer, may only be used nested
                 pass
 
         return readable_fields


Test patch:
diff --git a/tests/serializers.py b/tests/serializers.py
--- a/tests/serializers.py
+++ b/tests/serializers.py
@@ -1,3 +1,5 @@
+from rest_framework.settings import api_settings
+
 from rest_framework_json_api import serializers
 from tests.models import (
     BasicModel,
@@ -32,6 +34,16 @@ class Meta:
         )
 
 
+class ForeignKeySourcetHyperlinkedSerializer(serializers.HyperlinkedModelSerializer):
+    class Meta:
+        model = ForeignKeySource
+        fields = (
+            "name",
+            "target",
+            api_settings.URL_FIELD_NAME,
+        )
+
+
 class ManyToManyTargetSerializer(serializers.ModelSerializer):
     class Meta:
         fields = ("name",)
diff --git a/tests/test_views.py b/tests/test_views.py
--- a/tests/test_views.py
+++ b/tests/test_views.py
@@ -16,7 +16,9 @@
 from tests.serializers import BasicModelSerializer, ForeignKeyTargetSerializer
 from tests.views import (
     BasicModelViewSet,
+    ForeignKeySourcetHyperlinkedViewSet,
     ForeignKeySourceViewSet,
+    ForeignKeyTargetViewSet,
     ManyToManySourceViewSet,
     NestedRelatedSourceViewSet,
 )
@@ -87,7 +89,7 @@ def test_list(self, client, model):
 
     @pytest.mark.urls(__name__)
     def test_list_with_include_foreign_key(self, client, foreign_key_source):
-        url = reverse("foreign-key-source-list")
+        url = reverse("foreignkeysource-list")
         response = client.get(url, data={"include": "target"})
         assert response.status_code == status.HTTP_200_OK
         result = response.json()
@@ -156,7 +158,7 @@ def test_list_with_include_nested_related_field(
 
     @pytest.mark.urls(__name__)
     def test_list_with_invalid_include(self, client, foreign_key_source):
-        url = reverse("foreign-key-source-list")
+        url = reverse("foreignkeysource-list")
         response = client.get(url, data={"include": "invalid"})
         assert response.status_code == status.HTTP_400_BAD_REQUEST
         result = response.json()
@@ -195,7 +197,7 @@ def test_retrieve(self, client, model):
 
     @pytest.mark.urls(__name__)
     def test_retrieve_with_include_foreign_key(self, client, foreign_key_source):
-        url = reverse("foreign-key-source-detail", kwargs={"pk": foreign_key_source.pk})
+        url = reverse("foreignkeysource-detail", kwargs={"pk": foreign_key_source.pk})
         response = client.get(url, data={"include": "target"})
         assert response.status_code == status.HTTP_200_OK
         result = response.json()
@@ -208,6 +210,20 @@ def test_retrieve_with_include_foreign_key(self, client, foreign_key_source):
             }
         ] == result["included"]
 
+    @pytest.mark.urls(__name__)
+    def test_retrieve_hyperlinked_with_sparse_fields(self, client, foreign_key_source):
+        url = reverse(
+            "foreignkeysourcehyperlinked-detail", kwargs={"pk": foreign_key_source.pk}
+        )
+        response = client.get(url, data={"fields[ForeignKeySource]": "name"})
+        assert response.status_code == status.HTTP_200_OK
+        data = response.json()["data"]
+        assert data["attributes"] == {"name": foreign_key_source.name}
+        assert "relationships" not in data
+        assert data["links"] == {
+            "self": f"http://testserver/foreign_key_sources/{foreign_key_source.pk}/"
+        }
+
     @pytest.mark.urls(__name__)
     def test_patch(self, client, model):
         data = {
@@ -239,7 +255,7 @@ def test_delete(self, client, model):
 
     @pytest.mark.urls(__name__)
     def test_create_with_sparse_fields(self, client, foreign_key_target):
-        url = reverse("foreign-key-source-list")
+        url = reverse("foreignkeysource-list")
         data = {
             "data": {
                 "id": None,
@@ -379,6 +395,28 @@ def test_patch_with_custom_id(self, client):
             }
         }
 
+    @pytest.mark.urls(__name__)
+    def test_patch_with_custom_id_with_sparse_fields(self, client):
+        data = {
+            "data": {
+                "id": 2_193_102,
+                "type": "custom",
+                "attributes": {"body": "hello"},
+            }
+        }
+
+        url = reverse("custom-id")
+
+        response = client.patch(f"{url}?fields[custom]=body", data=data)
+        assert response.status_code == status.HTTP_200_OK
+        assert response.json() == {
+            "data": {
+                "type": "custom",
+                "id": "2176ce",  # get_id() -> hex
+                "attributes": {"body": "hello"},
+            }
+        }
+
 
 # Routing setup
 
@@ -415,13 +453,16 @@ class CustomModelSerializer(serializers.Serializer):
     id = serializers.IntegerField()
 
 
-class CustomIdModelSerializer(serializers.Serializer):
+class CustomIdSerializer(serializers.Serializer):
     id = serializers.SerializerMethodField()
     body = serializers.CharField()
 
     def get_id(self, obj):
         return hex(obj.id)[2:]
 
+    class Meta:
+        resource_name = "custom"
+
 
 class CustomAPIView(APIView):
     parser_classes = [JSONParser]
@@ -443,14 +484,23 @@ class CustomIdAPIView(APIView):
     resource_name = "custom"
 
     def patch(self, request, *args, **kwargs):
-        serializer = CustomIdModelSerializer(CustomModel(request.data))
+        serializer = CustomIdSerializer(
+            CustomModel(request.data), context={"request": self.request}
+        )
         return Response(status=status.HTTP_200_OK, data=serializer.data)
 
 
+# TODO remove basename and use default (lowercase of model)
+# this makes using HyperlinkedIdentityField easier and reduces
+# configuration in general
 router = SimpleRouter()
 router.register(r"basic_models", BasicModelViewSet, basename="basic-model")
+router.register(r"foreign_key_sources", ForeignKeySourceViewSet)
+router.register(r"foreign_key_targets", ForeignKeyTargetViewSet)
 router.register(
-    r"foreign_key_sources", ForeignKeySourceViewSet, basename="foreign-key-source"
+    r"foreign_key_sources_hyperlinked",
+    ForeignKeySourcetHyperlinkedViewSet,
+    "foreignkeysourcehyperlinked",
 )
 router.register(
     r"many_to_many_sources", ManyToManySourceViewSet, basename="many-to-many-source"
diff --git a/tests/views.py b/tests/views.py
--- a/tests/views.py
+++ b/tests/views.py
@@ -2,12 +2,15 @@
 from tests.models import (
     BasicModel,
     ForeignKeySource,
+    ForeignKeyTarget,
     ManyToManySource,
     NestedRelatedSource,
 )
 from tests.serializers import (
     BasicModelSerializer,
     ForeignKeySourceSerializer,
+    ForeignKeySourcetHyperlinkedSerializer,
+    ForeignKeyTargetSerializer,
     ManyToManySourceSerializer,
     NestedRelatedSourceSerializer,
 )
@@ -25,6 +28,18 @@ class ForeignKeySourceViewSet(ModelViewSet):
     ordering = ["name"]
 
 
+class ForeignKeySourcetHyperlinkedViewSet(ModelViewSet):
+    serializer_class = ForeignKeySourcetHyperlinkedSerializer
+    queryset = ForeignKeySource.objects.all()
+    ordering = ["name"]
+
+
+class ForeignKeyTargetViewSet(ModelViewSet):
+    serializer_class = ForeignKeyTargetSerializer
+    queryset = ForeignKeyTarget.objects.all()
+    ordering = ["name"]
+
+
 class ManyToManySourceViewSet(ModelViewSet):
     serializer_class = ManyToManySourceSerializer
     queryset = ManyToManySource.objects.all()