Repository: django-json-api/django-rest-framework-json-api

Issue:
Release 7.0.0 breaks sparse fields request
## Description of the Bug Report

As of release 7.0.0 a sparse fields request throws a KeyError for `url` at https://github.com/django-json-api/django-rest-framework-json-api/blob/366769e55e821eba11928125c903ad5043f9ff43/rest_framework_json_api/renderers.py#L485-L486 This was working up through release 6.1.0.

I expect that this is related to this change identified in the CHANGELOG:
https://github.com/django-json-api/django-rest-framework-json-api/blob/366769e55e821eba11928125c903ad5043f9ff43/CHANGELOG.md?plain=1#L27

How to reproduce:
```
git clone https://github.com/columbia-it/django-jsonapi-training
cd django-jsonapi-training
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
pip freeze | grep json
# see the version is djangorestframework-jsonapi==7.0.0
./manage.py migrate
./manage.py loaddata myapp/fixtures/*yaml
openssl genrsa -out oidc.key 4096
OIDC_RSA_PRIVATE_KEY_FILE=oidc.key ./manage.py runserver
```
Use Postman with OAuth2 as [described here](https://columbia-it-django-jsonapi-training.readthedocs.io/en/latest/using_dot/#get-an-oauth-20-token):
```
Token Name: *pick a name*
Grant Type: Authorization Code (With PKCE)
Callback URL: https://www.getpostman.com/oauth2/callback
Auth URL: http://localhost:8000/authorize/
Access Token URL: http://localhost:8000/token/
Client ID: demo_djt_web_client
Client Secret: demo_djt_web_secret
Scope: auth-columbia demo-djt-sla-bronze read openid profile email https://api.columbia.edu/scope/group
Client Authentication: Send as Basic Auth header
```

Click **Get New Access Token** and login as user `admin` pass `admin123` and **Authorize** then proceed and **Use Token**

Send GET http://localhost:8000/v1/courses/?fields[courses]=course_name

observe this error:
```
  File "/Users/ac45/src/django-jsonapi-training/venv/lib/python3.12/site-packages/rest_framework_json_api/renderers.py", line 486, in build_json_resource_obj
    fields[api_settings.URL_FIELD_NAME], relations.RelatedField
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'url'
```

Then send GET http://localhost:8000//v1/courses/?fields[courses]=course_name,url

and get back a correct JSONAPI response.

Now ^C the runserver, downgrade to v6.1.0, and try again.

```
pip install djangorestframework-jsonapi==6.1.0
OIDC_RSA_PRIVATE_KEY_FILE=oidc.key ./manage.py runserver

Send GET http://localhost:8000/v1/courses/?fields[courses]=course_name

and get back a correct JSONAPI response.

## Checklist

- [x] Certain that this is a bug (if unsure or you have a question use [discussions](https://github.com/django-json-api/django-rest-framework-json-api/discussions) instead)
- [x] Code snippet or unit test added to reproduce bug


Discussion:
sliverc: @n2ygk I have finally figured it out. The URL field needs to be part of the fields in the serializer meta class for the error to happen. In the code where you found the error all fields are included and as it is a HyperlinkedModelSerializer URL is also part of it.

Additionally, as mentioned before custom id was also affected by the same error which should be fixed with this PR as well.

Could you try it out with the tutorial repo whether this PR works as expected?

Thanks.

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a61d5fa..47a6dec 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -14,6 +14,10 @@ any parts of the framework not mentioned in the documentation should generally b
 
 * Added `429 Too Many Requests` as a possible error response in the OpenAPI schema.
 
+### Fixed
+
+* Ensured that URL and id field are kept when using sparse fields (regression since 7.0.0)
+
 ## [7.0.0] - 2024-05-02
 
 ### Added
diff --git a/rest_framework_json_api/renderers.py b/rest_framework_json_api/renderers.py
index 8c632f6..5980b95 100644
--- a/rest_framework_json_api/renderers.py
+++ b/rest_framework_json_api/renderers.py
@@ -446,7 +446,10 @@ class JSONRenderer(renderers.JSONRenderer):
                 return {
                     field_name: field
                     for field_name, field, in fields.items()
-                    if field_name in sparse_fields
+                    if field.field_name in sparse_fields
+                    # URL field is not considered a field in JSON:API spec
+                    # but a link so need to keep it
+                    or field.field_name == api_settings.URL_FIELD_NAME
                 }
 
         return fields
diff --git a/rest_framework_json_api/serializers.py b/rest_framework_json_api/serializers.py
index 66650ca..3ba9de8 100644
--- a/rest_framework_json_api/serializers.py
+++ b/rest_framework_json_api/serializers.py
@@ -94,10 +94,15 @@ class SparseFieldsetsMixin:
                         field
                         for field in readable_fields
                         if field.field_name in sparse_fields
+                        # URL field is not considered a field in JSON:API spec
+                        # but a link so need to keep it
                         or field.field_name == api_settings.URL_FIELD_NAME
+                        # ID is a required field which might have been overwritten
+                        # so need to keep it
+                        or field.field_name == "id"
                     )
             except AttributeError:
-                # no type on serializer, must be used only as only nested
+                # no type on serializer, may only be used nested
                 pass
 
         return readable_fields


Test patch:
diff --git a/tests/serializers.py b/tests/serializers.py
index ddf28f9..c312b83 100644
--- a/tests/serializers.py
+++ b/tests/serializers.py
@@ -1,3 +1,5 @@
+from rest_framework.settings import api_settings
+
 from rest_framework_json_api import serializers
 from tests.models import (
     BasicModel,
@@ -32,6 +34,16 @@ class ForeignKeySourceSerializer(serializers.ModelSerializer):
         )
 
 
+class ForeignKeySourcetHyperlinkedSerializer(serializers.HyperlinkedModelSerializer):
+    class Meta:
+        model = ForeignKeySource
+        fields = (
+            "name",
+            "target",
+            api_settings.URL_FIELD_NAME,
+        )
+
+
 class ManyToManyTargetSerializer(serializers.ModelSerializer):
     class Meta:
         fields = ("name",)
diff --git a/tests/test_views.py b/tests/test_views.py
index 468c2cb..45f8aac 100644
--- a/tests/test_views.py
+++ b/tests/test_views.py
@@ -16,7 +16,9 @@ from tests.models import BasicModel, ForeignKeySource
 from tests.serializers import BasicModelSerializer, ForeignKeyTargetSerializer
 from tests.views import (
     BasicModelViewSet,
+    ForeignKeySourcetHyperlinkedViewSet,
     ForeignKeySourceViewSet,
+    ForeignKeyTargetViewSet,
     ManyToManySourceViewSet,
     NestedRelatedSourceViewSet,
 )
@@ -87,7 +89,7 @@ class TestModelViewSet:
 
     @pytest.mark.urls(__name__)
     def test_list_with_include_foreign_key(self, client, foreign_key_source):
-        url = reverse("foreign-key-source-list")
+        url = reverse("foreignkeysource-list")
         response = client.get(url, data={"include": "target"})
         assert response.status_code == status.HTTP_200_OK
         result = response.json()
@@ -156,7 +158,7 @@ class TestModelViewSet:
 
     @pytest.mark.urls(__name__)
     def test_list_with_invalid_include(self, client, foreign_key_source):
-        url = reverse("foreign-key-source-list")
+        url = reverse("foreignkeysource-list")
         response = client.get(url, data={"include": "invalid"})
         assert response.status_code == status.HTTP_400_BAD_REQUEST
         result = response.json()
@@ -195,7 +197,7 @@ class TestModelViewSet:
 
     @pytest.mark.urls(__name__)
     def test_retrieve_with_include_foreign_key(self, client, foreign_key_source):
-        url = reverse("foreign-key-source-detail", kwargs={"pk": foreign_key_source.pk})
+        url = reverse("foreignkeysource-detail", kwargs={"pk": foreign_key_source.pk})
         response = client.get(url, data={"include": "target"})
         assert response.status_code == status.HTTP_200_OK
         result = response.json()
@@ -208,6 +210,20 @@ class TestModelViewSet:
             }
         ] == result["included"]
 
+    @pytest.mark.urls(__name__)
+    def test_retrieve_hyperlinked_with_sparse_fields(self, client, foreign_key_source):
+        url = reverse(
+            "foreignkeysourcehyperlinked-detail", kwargs={"pk": foreign_key_source.pk}
+        )
+        response = client.get(url, data={"fields[ForeignKeySource]": "name"})
+        assert response.status_code == status.HTTP_200_OK
+        data = response.json()["data"]
+        assert data["attributes"] == {"name": foreign_key_source.name}
+        assert "relationships" not in data
+        assert data["links"] == {
+            "self": f"http://testserver/foreign_key_sources/{foreign_key_source.pk}/"
+        }
+
     @pytest.mark.urls(__name__)
     def test_patch(self, client, model):
         data = {
@@ -239,7 +255,7 @@ class TestModelViewSet:
 
     @pytest.mark.urls(__name__)
     def test_create_with_sparse_fields(self, client, foreign_key_target):
-        url = reverse("foreign-key-source-list")
+        url = reverse("foreignkeysource-list")
         data = {
             "data": {
                 "id": None,
@@ -379,6 +395,28 @@ class TestAPIView:
             }
         }
 
+    @pytest.mark.urls(__name__)
+    def test_patch_with_custom_id_with_sparse_fields(self, client):
+        data = {
+            "data": {
+                "id": 2_193_102,
+                "type": "custom",
+                "attributes": {"body": "hello"},
+            }
+        }
+
+        url = reverse("custom-id")
+
+        response = client.patch(f"{url}?fields[custom]=body", data=data)
+        assert response.status_code == status.HTTP_200_OK
+        assert response.json() == {
+            "data": {
+                "type": "custom",
+                "id": "2176ce",  # get_id() -> hex
+                "attributes": {"body": "hello"},
+            }
+        }
+
 
 # Routing setup
 
@@ -415,13 +453,16 @@ class CustomModelSerializer(serializers.Serializer):
     id = serializers.IntegerField()
 
 
-class CustomIdModelSerializer(serializers.Serializer):
+class CustomIdSerializer(serializers.Serializer):
     id = serializers.SerializerMethodField()
     body = serializers.CharField()
 
     def get_id(self, obj):
         return hex(obj.id)[2:]
 
+    class Meta:
+        resource_name = "custom"
+
 
 class CustomAPIView(APIView):
     parser_classes = [JSONParser]
@@ -443,14 +484,23 @@ class CustomIdAPIView(APIView):
     resource_name = "custom"
 
     def patch(self, request, *args, **kwargs):
-        serializer = CustomIdModelSerializer(CustomModel(request.data))
+        serializer = CustomIdSerializer(
+            CustomModel(request.data), context={"request": self.request}
+        )
         return Response(status=status.HTTP_200_OK, data=serializer.data)
 
 
+# TODO remove basename and use default (lowercase of model)
+# this makes using HyperlinkedIdentityField easier and reduces
+# configuration in general
 router = SimpleRouter()
 router.register(r"basic_models", BasicModelViewSet, basename="basic-model")
+router.register(r"foreign_key_sources", ForeignKeySourceViewSet)
+router.register(r"foreign_key_targets", ForeignKeyTargetViewSet)
 router.register(
-    r"foreign_key_sources", ForeignKeySourceViewSet, basename="foreign-key-source"
+    r"foreign_key_sources_hyperlinked",
+    ForeignKeySourcetHyperlinkedViewSet,
+    "foreignkeysourcehyperlinked",
 )
 router.register(
     r"many_to_many_sources", ManyToManySourceViewSet, basename="many-to-many-source"
diff --git a/tests/views.py b/tests/views.py
index 72a7ea5..dba769a 100644
--- a/tests/views.py
+++ b/tests/views.py
@@ -2,12 +2,15 @@ from rest_framework_json_api.views import ModelViewSet
 from tests.models import (
     BasicModel,
     ForeignKeySource,
+    ForeignKeyTarget,
     ManyToManySource,
     NestedRelatedSource,
 )
 from tests.serializers import (
     BasicModelSerializer,
     ForeignKeySourceSerializer,
+    ForeignKeySourcetHyperlinkedSerializer,
+    ForeignKeyTargetSerializer,
     ManyToManySourceSerializer,
     NestedRelatedSourceSerializer,
 )
@@ -25,6 +28,18 @@ class ForeignKeySourceViewSet(ModelViewSet):
     ordering = ["name"]
 
 
+class ForeignKeySourcetHyperlinkedViewSet(ModelViewSet):
+    serializer_class = ForeignKeySourcetHyperlinkedSerializer
+    queryset = ForeignKeySource.objects.all()
+    ordering = ["name"]
+
+
+class ForeignKeyTargetViewSet(ModelViewSet):
+    serializer_class = ForeignKeyTargetSerializer
+    queryset = ForeignKeyTarget.objects.all()
+    ordering = ["name"]
+
+
 class ManyToManySourceViewSet(ModelViewSet):
     serializer_class = ManyToManySourceSerializer
     queryset = ManyToManySource.objects.all()