Repository: django-json-api/django-rest-framework-json-api

Issue:
JSONAPIOrderingFilter
Fixes #432 

## Description of the Change
Partially addresses #432 by adding `JSONAPIOrderingFilter` backend. Broken up into smaller PR's per @sliverc.

## Checklist

- [x] PR only contains one change (considered splitting up PR)
- [x] unit-test added
- [x] documentation updated
- [x] changelog entry added to `CHANGELOG.md`
- [x] author name in `AUTHORS`



Discussion:
# [Codecov](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459?src=pr&el=h1) Report
> Merging [#459](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459?src=pr&el=desc) into [master](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/commit/95e6d8db1e072a93122a0df7e5c5f26a8a0a1cdf?src=pr&el=desc) will **increase** coverage by `0.08%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459/graphs/tree.svg?token=kJ4T4hcVh8&width=650&height=150&src=pr)](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #459      +/-   ##
==========================================
+ Coverage   93.24%   93.33%   +0.08%     
==========================================
  Files          54       56       +2     
  Lines        3123     3164      +41     
==========================================
+ Hits         2912     2953      +41     
  Misses        211      211
```


| [Impacted Files](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [example/settings/dev.py](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459/diff?src=pr&el=tree#diff-ZXhhbXBsZS9zZXR0aW5ncy9kZXYucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [rest\_framework\_json\_api/backends.py](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459/diff?src=pr&el=tree#diff-cmVzdF9mcmFtZXdvcmtfanNvbl9hcGkvYmFja2VuZHMucHk=) | `100% <100%> (ø)` | |
| [example/tests/test\_backends.py](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459/diff?src=pr&el=tree#diff-ZXhhbXBsZS90ZXN0cy90ZXN0X2JhY2tlbmRzLnB5) | `100% <100%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459?src=pr&el=footer). Last update [95e6d8d...779aed9](https://codecov.io/gh/django-json-api/django-rest-framework-json-api/pull/459?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I am bit confused that you merged this before I had the chance to look at the changes you did again.

I think it is important that we only merge once changes are approved.

Reasons for this:
1. Requested changes need to be reviewed again whether there are done as expected.
2. sometimes because of complexity of code errors might only be found at a second glance

In this case the latter is actually the case. Just when looking at the code I have the feeling that when we pass on a camel cased variable e.g. blogEntry in case `blog_entry` sort field exists no error would be throw but the parameter be removed and no sorting will happen because parent class with `remove_invalid_fields` is called.

It would be great if you could verify this (best in a test) and open up another PR.

Also another thing I noticed is I think we should stick to the same naming of modules as Django REST Framework does meaning the module should be named `filters.py` instead of `backends.py`. This seems to be more consistent to me.


Sorry, I thought you indicated the changes were sufficient. Won’t do it
again.

On Wed, Aug 22, 2018 at 3:08 AM Oliver Sauder <notifications@github.com>
wrote:

> I am bit confused that you merged this before I had the chance to look at
> the changes you did again.
>
> I think it is important that we only merge once changes are approved.
>
> Reasons for this:
>
>    1. Requested changes need to be reviewed again whether there are done
>    as expected.
>    2. sometimes because of complexity of code errors might only be found
>    at a second glance
>
> In this case the latter is actually the case. Just when looking at the
> code I have the feeling that when we pass on a camel cased variable e.g.
> blogEntry in case blog_entry sort field exists no error would be throw
> but the parameter be removed and no sorting will happen because parent
> class with remove_invalid_fields is called.
>
> It would be great if you could verify this (best in a test) and open up
> another PR.
>
> Also another thing I noticed is I think we should stick to the same naming
> of modules as Django REST Framework does meaning the module should be named
> filters.py instead of backends.py. This seems to be more consistent to me.
>
> —
> You are receiving this because you modified the open/close state.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/django-json-api/django-rest-framework-json-api/pull/459#issuecomment-414933886>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEJ5d-BKtmxQVtYqm90t7nzpvk2xJ6sBks5uTQNugaJpZM4WEx9f>
> .
>

@sliverc wrote:
> In this case the latter is actually the case. Just when looking at the code I have the feeling that when we pass on a camel cased variable e.g. blogEntry in case blog_entry sort field exists no error would be throw but the parameter be removed and no sorting will happen because parent class with remove_invalid_fields is called.

I override `remove_invalid_fields` and use `format_value` here https://github.com/django-json-api/django-rest-framework-json-api/blob/e6290af5c34bf1760cb8c79fb8648b58e9336685/rest_framework_json_api/backends.py#L29

However, I think I see a fundamental problem with the information-losing reversibility of camel case, etc. Namely, if the model uses camel case rather than the "preferred" underscore for field names, the `format_value(...'underscore')` will do the wrong thing. I guess perhaps we just need to make the documentation a bit stronger about that requirement?

I'll take a look at more explicit testing and submit a PR shortly.

Resolving patch:
diff --git a/example/settings/dev.py b/example/settings/dev.py
--- a/example/settings/dev.py
+++ b/example/settings/dev.py
@@ -89,9 +89,8 @@
     ),
     'DEFAULT_METADATA_CLASS': 'rest_framework_json_api.metadata.JSONAPIMetadata',
     'DEFAULT_FILTER_BACKENDS': (
-        'rest_framework_json_api.backends.JSONAPIOrderingFilter',
+        'rest_framework_json_api.filters.JSONAPIOrderingFilter',
     ),
-    'ORDERING_PARAM': 'sort',
     'TEST_REQUEST_RENDERER_CLASSES': (
         'rest_framework_json_api.renderers.JSONRenderer',
     ),
diff --git a/example/views.py b/example/views.py
--- a/example/views.py
+++ b/example/views.py
@@ -90,6 +90,7 @@ class NoPagination(PageNumberPagination):
 
 class NonPaginatedEntryViewSet(EntryViewSet):
     pagination_class = NoPagination
+    ordering_fields = ('headline', 'body_text', 'blog__name', 'blog__id')
 
 
 class AuthorViewSet(ModelViewSet):
diff --git a/rest_framework_json_api/backends.py b/rest_framework_json_api/filters.py
similarity index 51%
rename from rest_framework_json_api/backends.py
rename to rest_framework_json_api/filters.py
--- a/rest_framework_json_api/backends.py
+++ b/rest_framework_json_api/filters.py
@@ -10,27 +10,35 @@ class JSONAPIOrderingFilter(OrderingFilter):
     if any sort field is invalid. If you prefer *not* to report 400 errors for
     invalid sort fields, just use OrderingFilter with `ordering_param='sort'`
 
-    TODO: Add sorting based upon relationships (sort=relname.fieldname)
+    Also applies DJA format_value() to convert (e.g. camelcase) to underscore.
+    (See JSON_API_FORMAT_FIELD_NAMES in docs/usage.md)
     """
     ordering_param = 'sort'
 
     def remove_invalid_fields(self, queryset, fields, view, request):
-        """
-        overrides remove_invalid_fields to raise a 400 exception instead of
-        silently removing them. set `ignore_bad_sort_fields = True` to not
-        do this validation.
-        """
         valid_fields = [
             item[0] for item in self.get_valid_fields(queryset, view,
                                                       {'request': request})
         ]
         bad_terms = [
             term for term in fields
-            if format_value(term.lstrip('-'), "underscore") not in valid_fields
+            if format_value(term.replace(".", "__").lstrip('-'), "underscore") not in valid_fields
         ]
         if bad_terms:
             raise ValidationError('invalid sort parameter{}: {}'.format(
                 ('s' if len(bad_terms) > 1 else ''), ','.join(bad_terms)))
+        # this looks like it duplicates code above, but we want the ValidationError to report
+        # the actual parameter supplied while we want the fields passed to the super() to
+        # be correctly rewritten.
+        # The leading `-` has to be stripped to prevent format_value from turning it into `_`.
+        underscore_fields = []
+        for item in fields:
+            item_rewritten = item.replace(".", "__")
+            if item_rewritten.startswith('-'):
+                underscore_fields.append(
+                    '-' + format_value(item_rewritten.lstrip('-'), "underscore"))
+            else:
+                underscore_fields.append(format_value(item_rewritten, "underscore"))
 
         return super(JSONAPIOrderingFilter, self).remove_invalid_fields(
-            queryset, fields, view, request)
+            queryset, underscore_fields, view, request)


Test patch:
diff --git a/example/tests/test_backends.py b/example/tests/test_backends.py
deleted file mode 100644
--- a/example/tests/test_backends.py
+++ /dev/null
@@ -1,54 +0,0 @@
-from rest_framework.reverse import reverse
-from rest_framework.test import APITestCase
-
-from ..models import Blog, Entry
-
-
-class DJATestParameters(APITestCase):
-    """
-    tests of JSON:API backends
-    """
-    fixtures = ('blogentry',)
-
-    def setUp(self):
-        self.entries = Entry.objects.all()
-        self.blogs = Blog.objects.all()
-        self.url = reverse('nopage-entry-list')
-
-    def test_sort(self):
-        """
-        test sort
-        """
-        response = self.client.get(self.url, data={'sort': 'headline'})
-        self.assertEqual(response.status_code, 200,
-                         msg=response.content.decode("utf-8"))
-        dja_response = response.json()
-        headlines = [c['attributes']['headline'] for c in dja_response['data']]
-        sorted_headlines = [c['attributes']['headline'] for c in dja_response['data']]
-        sorted_headlines.sort()
-        self.assertEqual(headlines, sorted_headlines)
-
-    def test_sort_reverse(self):
-        """
-        confirm switching the sort order actually works
-        """
-        response = self.client.get(self.url, data={'sort': '-headline'})
-        self.assertEqual(response.status_code, 200,
-                         msg=response.content.decode("utf-8"))
-        dja_response = response.json()
-        headlines = [c['attributes']['headline'] for c in dja_response['data']]
-        sorted_headlines = [c['attributes']['headline'] for c in dja_response['data']]
-        sorted_headlines.sort()
-        self.assertNotEqual(headlines, sorted_headlines)
-
-    def test_sort_invalid(self):
-        """
-        test sort of invalid field
-        """
-        response = self.client.get(self.url,
-                                   data={'sort': 'nonesuch,headline,-not_a_field'})
-        self.assertEqual(response.status_code, 400,
-                         msg=response.content.decode("utf-8"))
-        dja_response = response.json()
-        self.assertEqual(dja_response['errors'][0]['detail'],
-                         "invalid sort parameters: nonesuch,-not_a_field")
diff --git a/example/tests/test_filters.py b/example/tests/test_filters.py
new file mode 100644
--- /dev/null
+++ b/example/tests/test_filters.py
@@ -0,0 +1,105 @@
+from rest_framework.reverse import reverse
+from rest_framework.test import APITestCase
+
+from ..models import Blog, Entry
+
+
+class DJATestParameters(APITestCase):
+    """
+    tests of JSON:API backends
+    """
+    fixtures = ('blogentry',)
+
+    def setUp(self):
+        self.entries = Entry.objects.all()
+        self.blogs = Blog.objects.all()
+        self.url = reverse('nopage-entry-list')
+
+    def test_sort(self):
+        """
+        test sort
+        """
+        response = self.client.get(self.url, data={'sort': 'headline'})
+        self.assertEqual(response.status_code, 200,
+                         msg=response.content.decode("utf-8"))
+        dja_response = response.json()
+        headlines = [c['attributes']['headline'] for c in dja_response['data']]
+        sorted_headlines = sorted(headlines)
+        self.assertEqual(headlines, sorted_headlines)
+
+    def test_sort_reverse(self):
+        """
+        confirm switching the sort order actually works
+        """
+        response = self.client.get(self.url, data={'sort': '-headline'})
+        self.assertEqual(response.status_code, 200,
+                         msg=response.content.decode("utf-8"))
+        dja_response = response.json()
+        headlines = [c['attributes']['headline'] for c in dja_response['data']]
+        sorted_headlines = sorted(headlines)
+        self.assertNotEqual(headlines, sorted_headlines)
+
+    def test_sort_double_negative(self):
+        """
+        what if they provide multiple `-`'s? It's OK.
+        """
+        response = self.client.get(self.url, data={'sort': '--headline'})
+        self.assertEqual(response.status_code, 200,
+                         msg=response.content.decode("utf-8"))
+        dja_response = response.json()
+        headlines = [c['attributes']['headline'] for c in dja_response['data']]
+        sorted_headlines = sorted(headlines)
+        self.assertNotEqual(headlines, sorted_headlines)
+
+    def test_sort_invalid(self):
+        """
+        test sort of invalid field
+        """
+        response = self.client.get(self.url,
+                                   data={'sort': 'nonesuch,headline,-not_a_field'})
+        self.assertEqual(response.status_code, 400,
+                         msg=response.content.decode("utf-8"))
+        dja_response = response.json()
+        self.assertEqual(dja_response['errors'][0]['detail'],
+                         "invalid sort parameters: nonesuch,-not_a_field")
+
+    def test_sort_camelcase(self):
+        """
+        test sort of camelcase field name
+        """
+        response = self.client.get(self.url, data={'sort': 'bodyText'})
+        self.assertEqual(response.status_code, 200,
+                         msg=response.content.decode("utf-8"))
+        dja_response = response.json()
+        blog_ids = [(c['attributes']['bodyText'] or '') for c in dja_response['data']]
+        sorted_blog_ids = sorted(blog_ids)
+        self.assertEqual(blog_ids, sorted_blog_ids)
+
+    def test_sort_underscore(self):
+        """
+        test sort of underscore field name
+        Do we allow this notation in a search even if camelcase is in effect?
+        "Be conservative in what you send, be liberal in what you accept"
+                   --  https://en.wikipedia.org/wiki/Robustness_principle
+        """
+        response = self.client.get(self.url, data={'sort': 'body_text'})
+        self.assertEqual(response.status_code, 200,
+                         msg=response.content.decode("utf-8"))
+        dja_response = response.json()
+        blog_ids = [(c['attributes']['bodyText'] or '') for c in dja_response['data']]
+        sorted_blog_ids = sorted(blog_ids)
+        self.assertEqual(blog_ids, sorted_blog_ids)
+
+    def test_sort_related(self):
+        """
+        test sort via related field using jsonapi path `.` and django orm `__` notation.
+        ORM relations must be predefined in the View's .ordering_fields attr
+        """
+        for datum in ('blog__id', 'blog.id'):
+            response = self.client.get(self.url, data={'sort': datum})
+            self.assertEqual(response.status_code, 200,
+                             msg=response.content.decode("utf-8"))
+            dja_response = response.json()
+            blog_ids = [c['relationships']['blog']['data']['id'] for c in dja_response['data']]
+            sorted_blog_ids = sorted(blog_ids)
+            self.assertEqual(blog_ids, sorted_blog_ids)