Repository: Kinto/kinto

Issue:
Filter objects where a particular field is null 
`/records?field.name=null` ?
Crash on dev ota server when requesting /records?min_target.version=53
```
$ http GET https://kinto-ota.dev.mozaws.net/v1/buckets/build-hub/collections/archives/records\?min_target.version\=53`

HTTP/1.1 503 Service Unavailable
Access-Control-Expose-Headers: Total-Records, Next-Page, Expires, Last-Modified, ETag, Content-Length, Pragma, Alert, Cache-Control, Retry-After, Backoff
Connection: keep-alive
Content-Length: 151
Content-Type: application/json
Date: Fri, 05 May 2017 15:13:28 GMT
Retry-After: 3
Server: nginx
X-Content-Type-Options: nosniff

{
    "code": 503, 
    "errno": 201, 
    "error": "Service Unavailable", 
    "message": "Service temporary unavailable due to overloading or maintenance, please retry later."
}
```



Discussion:
Related #343 #344
Isn't that syntax ambiguous? It's the same to search for any object whose value is the string `"null"`. I'd rather be more rigid about what our query syntax is for this case, maybe introducing a new syntax for JSON-y values like `is_field.name=null` or `json_field.name=null`.
``https://kinto-ota.dev.mozaws.net/v1/buckets/build-hub/collections/releases/records?build.id=null``

Returns an empty list, so I guess it is not sufficient.
Should `field.name=null` match records where the field is missing, or only where it is explicitly present and null? (I would vote for the second option)

```
raise exceptions.BackendError(original=e) from e\nkinto.core.storage.exceptions.BackendError: DataError: (psycopg2.DataError) invalid input syntax for type numeric: "53.0b4"
```
Work around: put quotes around `53`:  `/records?min_target.version="53"`
Is that a feature we support or a side effect?
> Is that a feature we support or a side effect?

What?

We want to use numeric comparison if both field and filter values are numeric, textual otherwise.
I was talking about the fact of adding quotes in the querystring.
It is not properly documented, so I guess we can classify as side effect :))

Do you think that something we should support?
Refs #1216 and #1215. I think we're trying too hard to be clever with an ad-hoc string query syntax and IMO we should leave the current syntax in place ("simple" syntax, where anything that can be converted to a number is a number and everything else is a string) and add a robust syntax that can encode any value as well as its type. I'd propose JSON (so `/records?min_target.version=53` for number, `/records?min_target.version="53"` for string, `/records?min_target.version=null` for null, etc.).

For comparison order, one reason I don't want to be too clever here is that I'm afraid of inventing an inconsistent order. E.g. 2.0 < 10 but 10 < "2.0b". Having a robust syntax doesn't completely avoid this problem because data in different records can be of different types, and we will have to order them somehow (if we get e.g. `_sort=different_typed_field`). How do we handle that? I don't think throwing an exception in this case is necessarily a bad thing, but if we have to invent an ordering, I think "all ints are less than all strings" [i.e. the [Python2.7 comparison rules](https://docs.python.org/2/library/stdtypes.html)] is fine.


@glasserc I tried your idea of putting a JSON decoder there. But apparently it doesn't fix any of the issue we have. Because even if we decode the number as a number then if we do a Postgresql request with a numeric value on a row that contains string we still have the issue.
There were two parts to this bug and the JSON decoder fixes the first one. Previously, this request would work "sometimes" depending on whether the data in the record looked numeric or not. Now, it never works, which is an improvement.

The second part of the bug is about how we compare numbers to strings. Like I wrote originally, we already have that problem. You can put record1 = {"flavor": "1"} and record2 = {"flavor": 1} and then _sort=flavor. What order comes out? Does Postgres let you do that? I think the answer to those questions should guide our fix.
It seems that Postgres's sorting order is all strings, followed by all numbers:

```
{
    "data": [
        {
            "flavor": "1",
            "id": "1b210b24-b0e6-44c4-b427-e912a3c94697",
            "last_modified": 1496695080659,
            "null": null,
            "people": []
        },
        {
            "flavor": "2",
            "id": "417fa284-42d0-41ad-a39e-16f40542120c",
            "last_modified": 1496695076284,
            "null": null,
            "people": []
        },
        {
            "flavor": "strawberry",
            "id": "7f163dc4-4aad-4680-bb1d-0c8a4dbcf479",
            "last_modified": 1496694495121,
            "null": null,
            "people": []
        },
        {
            "flavor": 1,
            "id": "d41694ee-94a5-4f11-93c3-3b3cfee38e5a",
            "last_modified": 1496695085202,
            "null": null,
            "people": []
        },
        {
            "flavor": 2,
            "id": "83f1952a-5af8-4d98-95f3-93602adc3427",
            "last_modified": 1496695037864,
            "null": null,
            "people": []
        }
    ]
}
```

So I would say we should compare numbers as different from non-numerics, but compare them earlier.
Do we have a case where we mix string and integer in the database? Cannot we use the schema to make sure a fields is always of type string even if that string contains only numbers?

Resolving patch:
diff --git a/kinto/core/storage/memory.py b/kinto/core/storage/memory.py
--- a/kinto/core/storage/memory.py
+++ b/kinto/core/storage/memory.py
@@ -1,6 +1,8 @@
 import re
 import operator
 from collections import defaultdict
+from collections import abc
+import numbers
 
 from kinto.core import utils
 from kinto.core.decorators import synchronized
@@ -12,6 +14,13 @@
 import ujson
 
 
+class Missing():
+    pass
+
+
+MISSING = Missing()
+
+
 def tree():
     return defaultdict(tree)
 
@@ -332,26 +341,54 @@ def apply_filters(records, filters):
                 if isinstance(right, int):
                     right = str(right)
 
-            left = find_nested_value(record, f.field)
+            left = find_nested_value(record, f.field, MISSING)
 
             if f.operator in (COMPARISON.IN, COMPARISON.EXCLUDE):
                 right, left = left, right
+            elif f.operator not in (COMPARISON.LIKE, COMPARISON.HAS):
+                left = schwartzian_transform(left)
+                right = schwartzian_transform(right)
+
+            if f.operator == COMPARISON.HAS:
+                matches = left != MISSING if f.value else left == MISSING
             else:
-                if left is None:
-                    right_is_number = (
-                        isinstance(right, (int, float)) and
-                        right not in (True, False))
-                    if right_is_number:
-                        # Python3 cannot compare None to a number.
-                        matches = False
-                        continue
-                    else:
-                        left = ''  # To mimic what we do for postgresql.
-            matches = matches and operators[f.operator](left, right)
+                matches = matches and operators[f.operator](left, right)
         if matches:
             yield record
 
 
+def schwartzian_transform(value):
+    """Decorate a value with a tag that enforces the Postgres sort order.
+
+    The sort order, per https://www.postgresql.org/docs/9.6/static/datatype-json.html, is:
+
+    Object > Array > Boolean > Number > String > Null
+
+    Note that there are more interesting rules for comparing objects
+    and arrays but we probably don't need to be that compatible.
+
+    MISSING represents what would be a SQL NULL, which is "bigger"
+    than everything else.
+    """
+    if value is None:
+        return (0, value)
+    if isinstance(value, str):
+        return (1, value)
+    if isinstance(value, bool):
+        # This has to be before Number, because bools are a subclass
+        # of int :(
+        return (3, value)
+    if isinstance(value, numbers.Number):
+        return (2, value)
+    if isinstance(value, abc.Sequence):
+        return (4, value)
+    if isinstance(value, abc.Mapping):
+        return (5, value)
+    if value is MISSING:
+        return (6, value)
+    raise ValueError("Unknown value: {}".format(value))   # pragma: no cover
+
+
 def apply_sorting(records, sorting):
     """Sort the specified records, using cumulative python sorting.
     """
@@ -360,14 +397,12 @@ def apply_sorting(records, sorting):
     if not result:
         return result
 
-    first_record = result[0]
-
-    def column(first, record, name):
-        return find_nested_value(record, name, default=float('inf'))
+    def column(record, name):
+        return schwartzian_transform(find_nested_value(record, name, default=MISSING))
 
     for sort in reversed(sorting):
         result = sorted(result,
-                        key=lambda r: column(first_record, r, sort.field),
+                        key=lambda r: column(r, sort.field),
                         reverse=(sort.direction < 0))
 
     return result
diff --git a/kinto/core/storage/postgresql/__init__.py b/kinto/core/storage/postgresql/__init__.py
--- a/kinto/core/storage/postgresql/__init__.py
+++ b/kinto/core/storage/postgresql/__init__.py
@@ -7,7 +7,7 @@
     StorageBase, exceptions,
     DEFAULT_ID_FIELD, DEFAULT_MODIFIED_FIELD, DEFAULT_DELETED_FIELD)
 from kinto.core.storage.postgresql.client import create_from_config
-from kinto.core.utils import COMPARISON, json, is_numeric
+from kinto.core.utils import COMPARISON, json
 
 
 logger = logging.getLogger(__name__)
@@ -686,6 +686,7 @@ def _format_conditions(self, filters, id_field, modified_field,
         holders = {}
         for i, filtr in enumerate(filters):
             value = filtr.value
+            is_like_query = filtr.operator == COMPARISON.LIKE
 
             if filtr.field == id_field:
                 sql_field = 'id'
@@ -701,43 +702,75 @@ def _format_conditions(self, filters, id_field, modified_field,
                     # Safely escape field name
                     field_holder = '{}_field_{}_{}'.format(prefix, i, j)
                     holders[field_holder] = subfield
-                    # Use ->> to convert the last level to text.
-                    column_name += "->>" if j == len(subfields) - 1 else "->"
+                    # Use ->> to convert the last level to text if
+                    # needed for LIKE query. (Other queries do JSONB comparison.)
+                    column_name += "->>" if j == len(subfields) - 1 and is_like_query else "->"
                     column_name += ":{}".format(field_holder)
 
-                # If field is missing, we default to ''.
-                sql_field = "coalesce({}, '')".format(column_name)
-                # Cast when comparing to number (eg. '4' < '12')
-                try:
-                    if value and (is_numeric(value) or all([is_numeric(v) for v in value])):
-                        sql_field = "({})::numeric".format(column_name)
-                except TypeError:  # not iterable
-                    pass
-
-            if filtr.operator not in (COMPARISON.IN, COMPARISON.EXCLUDE):
-                # For the IN operator, let psycopg escape the values list.
-                # Otherwise JSON-ify the native value (e.g. True -> 'true')
-                if not isinstance(value, str):
-                    value = json.dumps(value).strip('"')
-            else:
-                # If not all numeric, fallback to string comparison.
-                if not all([is_numeric(v) for v in value]):
-                    value = [str(v) for v in value]
+                # If the field is missing, column_name will produce
+                # NULL. NULL has strange properties with comparisons
+                # in SQL -- NULL = anything => NULL, NULL <> anything => NULL.
+                # We generally want missing fields to be treated as a
+                # special value that compares as different from
+                # everything, including JSON null. Do this on a
+                # per-operator basis.
+                null_false_operators = (
+                    # NULLs aren't EQ to anything (definitionally).
+                    COMPARISON.EQ,
+                    # So they can't match anything in an INCLUDE.
+                    COMPARISON.IN,
+                    # Nor can they be LIKE anything.
+                    COMPARISON.LIKE,
+                )
+                null_true_operators = (
+                    # NULLs are automatically not equal to everything.
+                    COMPARISON.NOT,
+                    # Thus they can never be excluded.
+                    COMPARISON.EXCLUDE,
+                    # Match Postgres's default sort behavior
+                    # (NULLS LAST) by allowing NULLs to
+                    # automatically be greater than everything.
+                    COMPARISON.GT, COMPARISON.MIN,
+                )
+
+                if filtr.operator in null_false_operators:
+                    sql_field = "{} IS NOT NULL AND {}".format(column_name, column_name)
+                elif filtr.operator in null_true_operators:
+                    sql_field = "{} IS NULL OR {}".format(column_name, column_name)
+                else:
+                    # No need to check for LT and MAX because NULL < foo
+                    # is NULL, which is falsy in SQL.
+                    sql_field = column_name
+
+            string_field = filtr.field in (id_field, modified_field) or is_like_query
+            if not string_field:
+                # JSONB-ify the value.
+                if filtr.operator not in (COMPARISON.IN, COMPARISON.EXCLUDE):
+                    value = json.dumps(value)
+                else:
+                    value = [json.dumps(v) for v in value]
+
+            if filtr.operator in (COMPARISON.IN, COMPARISON.EXCLUDE):
                 value = tuple(value)
                 # WHERE field IN ();  -- Fails with syntax error.
                 if len(value) == 0:
                     value = (None,)
 
-            if filtr.operator == COMPARISON.LIKE:
+            if is_like_query:
+                # Operand should be a string.
                 value = '%{}%'.format(value)
 
-            # Safely escape value
-            value_holder = '{}_value_{}'.format(prefix, i)
-            holders[value_holder] = value
+            if filtr.operator == COMPARISON.HAS:
+                operator = 'IS NOT NULL' if filtr.value else 'IS NULL'
+                cond = "{} {}".format(sql_field, operator)
+            else:
+                # Safely escape value
+                value_holder = '{}_value_{}'.format(prefix, i)
+                holders[value_holder] = value
 
-            sql_operator = operators.setdefault(filtr.operator,
-                                                filtr.operator.value)
-            cond = "{} {} :{}".format(sql_field, sql_operator, value_holder)
+                sql_operator = operators.setdefault(filtr.operator,
+                                                    filtr.operator.value)
+                cond = "{} {} :{}".format(sql_field, sql_operator, value_holder)
             conditions.append(cond)
 
         safe_sql = ' AND '.join(conditions)
diff --git a/kinto/core/utils.py b/kinto/core/utils.py
--- a/kinto/core/utils.py
+++ b/kinto/core/utils.py
@@ -57,14 +57,6 @@ def classname(obj):
     return obj.__class__.__name__.lower()
 
 
-def is_numeric(value):
-    """Check if the provided value is a numeric value.
-
-    :rtype: bool
-    """
-    return isinstance(value, (int, float)) and not isinstance(value, bool)
-
-
 def merge_dicts(a, b):
     """Merge b into a recursively, without overwriting values.
 
@@ -228,6 +220,7 @@ class COMPARISON(Enum):
     IN = 'in'
     EXCLUDE = 'exclude'
     LIKE = 'like'
+    HAS = 'has'
 
 
 def reapply_cors(request, response):


Test patch:
diff --git a/kinto/core/storage/testing.py b/kinto/core/storage/testing.py
--- a/kinto/core/storage/testing.py
+++ b/kinto/core/storage/testing.py
@@ -350,6 +350,50 @@ def test_get_all_handle_sorting_on_subobject(self):
         self.assertLess(records[0]['person']['age'],
                         records[-1]['person']['age'])
 
+    def test_get_all_sorting_is_consistent_with_filtering(self):
+        self.create_record({'flavor': 'strawberry'})
+        self.create_record({'flavor': 'blueberry', 'author': None})
+        self.create_record({'flavor': 'raspberry', 'author': 1})
+        self.create_record({'flavor': 'orange', 'author': True})
+        self.create_record({'flavor': 'watermelon', 'author': 'Ethan'})
+        sorting = [Sort('author', 1)]
+        records, _ = self.storage.get_all(sorting=sorting, **self.storage_kw)
+        # Some interesting values to compare against
+        values = ['A', 'Z', '', 0, 4]
+
+        for value in values:
+            # Together, these filters should provide the entire list
+            filter_less = Filter('author', value, utils.COMPARISON.LT)
+            filter_min = Filter('author', value, utils.COMPARISON.MIN)
+            smaller_records, _ = self.storage.get_all(filters=[filter_less],
+                                                      sorting=sorting,
+                                                      **self.storage_kw)
+            greater_records, _ = self.storage.get_all(filters=[filter_min],
+                                                      sorting=sorting,
+                                                      **self.storage_kw)
+            other_records = smaller_records + greater_records
+            self.assertEqual(records, other_records,
+                             "Filtering is not consistent with sorting when filtering "
+                             "using value {}: {} (LT) + {} (MIN) != {}".format(
+                                 value, smaller_records, greater_records, records))
+
+        # Same test but with MAX and GT
+        for value in values:
+            # Together, these filters should provide the entire list
+            filter_less = Filter('author', value, utils.COMPARISON.MAX)
+            filter_min = Filter('author', value, utils.COMPARISON.GT)
+            smaller_records, _ = self.storage.get_all(filters=[filter_less],
+                                                      sorting=sorting,
+                                                      **self.storage_kw)
+            greater_records, _ = self.storage.get_all(filters=[filter_min],
+                                                      sorting=sorting,
+                                                      **self.storage_kw)
+            other_records = smaller_records + greater_records
+            self.assertEqual(records, other_records,
+                             "Filtering is not consistent with sorting when filtering "
+                             "using value {}: {} (MAX) + {} (GT) != {}".format(
+                                 value, smaller_records, greater_records, records))
+
     def test_get_all_can_filter_with_list_of_values(self):
         for l in ['a', 'b', 'c']:
             self.create_record({'code': l})
@@ -428,18 +472,54 @@ def test_get_all_can_filter_with_strings(self):
         self.assertEqual(records[1]['name'], "Marie")
         self.assertEqual(len(records), 2)
 
-    def test_get_all_can_filter_with_none_values(self):
+    def test_get_all_can_filter_minimum_value_with_strings(self):
+        for v in ["49.0", "6.0", "53.0b4"]:
+            self.create_record({"product": {"version": v}})
+        sorting = [Sort("product.version", 1)]
+        filters = [Filter("product.version", "50.0", utils.COMPARISON.MIN)]
+        records, _ = self.storage.get_all(sorting=sorting, filters=filters,
+                                          **self.storage_kw)
+        self.assertEqual(records[0]["product"]["version"], "53.0b4")
+        self.assertEqual(records[1]["product"]["version"], "6.0")
+        self.assertEqual(len(records), 2)
+
+    def test_get_all_does_not_implicitly_cast(self):
+        for v in ["49.0", "6.0", "53.0b4"]:
+            self.create_record({"product": {"version": v}})
+        sorting = [Sort("product.version", 1)]
+        filters = [Filter("product.version", 50.0, utils.COMPARISON.MIN)]
+        records, _ = self.storage.get_all(sorting=sorting, filters=filters,
+                                          **self.storage_kw)
+        self.assertEqual(len(records), 0)  # 50 (number) > strings
+
+    def test_get_all_can_deal_with_none_values(self):
         self.create_record({"name": "Alexis"})
         self.create_record({"title": "haha"})
         self.create_record({"name": "Mathieu"})
         filters = [Filter("name", "Fanny", utils.COMPARISON.GT)]
         records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
-        self.assertEqual(len(records), 1)  # None is not greater than "Fanny"
-        self.assertEqual(records[0]["name"], "Mathieu")
+        # NULLs compare as greater than everything
+        self.assertEqual(len(records), 2)
+        # But we aren't clear on what the order will be
+        mathieu_record = records[0] if 'name' in records[0] else records[1]
+        haha_record = records[1] if 'name' in records[0] else records[0]
+        self.assertEqual(mathieu_record["name"], "Mathieu")
+        self.assertEqual(haha_record["title"], "haha")
 
         filters = [Filter("name", "Fanny", utils.COMPARISON.LT)]
         records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
-        self.assertEqual(len(records), 2)  # None is less than "Fanny"
+        self.assertEqual(len(records), 1)
+        self.assertEqual(records[0]["name"], "Alexis")
+
+    def test_get_all_can_filter_with_none_values(self):
+        self.create_record({"name": "Alexis", "salary": None})
+        self.create_record({"name": "Mathieu", "salary": "null"})
+        self.create_record({"name": "Niko", "salary": ""})
+        self.create_record({"name": "Ethan"})   # missing salary
+        filters = [Filter("salary", None, utils.COMPARISON.EQ)]
+        records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
+        self.assertEqual(len(records), 1)
+        self.assertEqual(records[0]["name"], "Alexis")
 
     def test_get_all_can_filter_with_list_of_values_on_id(self):
         record1 = self.create_record({'code': 'a'})
@@ -489,6 +569,60 @@ def test_get_all_can_filter_a_list_of_integer_values_on_subobjects(self):
                                           **self.storage_kw)
         self.assertEqual(len(records), 1)
 
+    def test_get_all_can_filter_matching_a_list(self):
+        self.create_record({"flavor": "strawberry", "orders": []})
+        self.create_record({"flavor": "blueberry", "orders": [1]})
+        self.create_record({"flavor": "pineapple", "orders": [1, 2]})
+        self.create_record({"flavor": "watermelon", "orders": ""})
+        self.create_record({"flavor": "raspberry", "orders": {}})
+        filters = [Filter("orders", [], utils.COMPARISON.EQ)]
+        records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
+        self.assertEqual(len(records), 1)
+        self.assertEqual(records[0]["flavor"], "strawberry")
+
+        filters = [Filter("orders", [1], utils.COMPARISON.EQ)]
+        records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
+        self.assertEqual(len(records), 1)
+        self.assertEqual(records[0]["flavor"], "blueberry")
+
+    def test_get_all_can_filter_matching_an_object(self):
+        self.create_record({"flavor": "strawberry", "attributes": {}})
+        self.create_record({
+            "flavor": "blueberry",
+            "attributes": {"ibu": 25, "seen_on": "2017-06-01"},
+        })
+        self.create_record({
+            "flavor": "watermelon",
+            "attributes": {"ibu": 25, "seen_on": "2017-06-01", "price": 9.99},
+        })
+        self.create_record({"flavor": "raspberry", "attributes": []})
+        filters = [Filter("attributes", {}, utils.COMPARISON.EQ)]
+        records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
+        self.assertEqual(len(records), 1)
+        self.assertEqual(records[0]["flavor"], "strawberry")
+
+        filters = [Filter("attributes", {"ibu": 25, "seen_on": "2017-06-01"}, utils.COMPARISON.EQ)]
+        records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
+        self.assertEqual(len(records), 1)
+        self.assertEqual(records[0]["flavor"], "blueberry")
+
+    def test_get_all_supports_has(self):
+        self.create_record({"flavor": "strawberry"})
+        self.create_record({"flavor": "blueberry", "author": None})
+        self.create_record({"flavor": "raspberry", "author": ""})
+        self.create_record({"flavor": "watermelon", "author": "hello"})
+        self.create_record({"flavor": "pineapple", "author": "null"})
+        filters = [Filter("author", True, utils.COMPARISON.HAS)]
+        records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
+        self.assertEqual(len(records), 4)
+        self.assertEqual(sorted([r['flavor'] for r in records]),
+                         ["blueberry", "pineapple", "raspberry", "watermelon"])
+
+        filters = [Filter("author", False, utils.COMPARISON.HAS)]
+        records, _ = self.storage.get_all(filters=filters, **self.storage_kw)
+        self.assertEqual(len(records), 1)
+        self.assertEqual(records[0]["flavor"], "strawberry")
+
     def test_get_all_can_filter_by_subobjects_values(self):
         for l in ['a', 'b', 'c']:
             self.create_record({'code': {'sub': l}})
diff --git a/tests/core/resource/test_filter.py b/tests/core/resource/test_filter.py
--- a/tests/core/resource/test_filter.py
+++ b/tests/core/resource/test_filter.py
@@ -12,12 +12,12 @@ def setUp(self):
         records = [
             {'title': 'MoFo', 'status': 0, 'favorite': True},
             {'title': 'MoFo', 'status': 1, 'favorite': False},
-            {'title': 'MoFo', 'status': 2, 'favorite': False},
-            {'title': 'MoFo', 'status': 0, 'favorite': False},
+            {'title': 'MoFo', 'status': 2, 'favorite': False, 'sometimes': 'available'},
+            {'title': 'MoFo', 'status': 0, 'favorite': False, 'sometimes': None},
             {'title': 'MoFo', 'status': 1, 'favorite': True},
             {'title': 'MoFo', 'status': 2, 'favorite': False},
             {'title': 'Foo', 'status': 3, 'favorite': False},
-            {'title': 'Bar', 'status': 3, 'favorite': False},
+            {'title': 'Bar', 'status': 3, 'favorite': False, 'sometimes': 'present'},
         ]
         for r in records:
             self.model.create_record(r)
@@ -175,6 +175,19 @@ def test_exclude_values(self):
         values = [item['status'] for item in result['data']]
         self.assertEqual(sorted(values), [1, 1, 2, 2, 3, 3])
 
+    def test_has_values(self):
+        self.validated['querystring'] = {'has_sometimes': True}
+        result = self.resource.collection_get()
+        values = [item['sometimes'] for item in result['data']]
+        assert None in values
+        self.assertEqual(sorted([v for v in values if v]), ['available', 'present'])
+
+    def test_has_values_false(self):
+        self.validated['querystring'] = {'has_sometimes': False}
+        result = self.resource.collection_get()
+        values = ['sometimes' in item for item in result['data']]
+        self.assertEqual(sorted(values), [False, False, False, False, False])
+
     def test_include_returns_400_if_value_has_wrong_type(self):
         self.validated['querystring'] = {'in_id': [0, 1]}
         with self.assertRaises(httpexceptions.HTTPBadRequest) as cm:
@@ -224,3 +237,62 @@ def test_subobjects_filters_works_with_directives(self):
         result = self.resource.collection_get()
         values = [item['party']['voters'] for item in result['data']]
         self.assertEqual(sorted(values), [1, 2, 3])
+
+
+class JSONFilteringTest(BaseTest):
+    def setUp(self):
+        super().setUp()
+        self.validated = self.resource.request.validated
+        self.patch_known_field.start()
+        records = [
+            {
+                "id": "strawberry",
+                "flavor": "strawberry",
+                "orders": [],
+                "attributes": {"ibu": 25, "seen_on": "2017-06-01"},
+                "author": None,
+            },
+            {"id": "blueberry-1", "flavor": "blueberry", "orders": [1]},
+            {"id": "blueberry-2", "flavor": "blueberry", "orders": ""},
+            {"id": "raspberry-1", "flavor": "raspberry", "attributes": {}},
+            {"id": "raspberry-2", "flavor": "raspberry", "attributes": []},
+            {
+                "id": "raspberry-3",
+                "flavor": "raspberry",
+                "attributes": {"ibu": 25, "seen_on": "2017-06-01", "price": 9.99},
+            },
+            {"id": "watermelon-1", "flavor": "watermelon", "author": "null"},
+            {"id": "watermelon-2", "flavor": "watermelon", "author": 0},
+        ]
+        for r in records:
+            self.model.create_record(r)
+
+    def test_filter_by_empty_array(self):
+        self.validated['querystring'] = {'orders': []}
+        result = self.resource.collection_get()
+        assert len(result['data']) == 1
+        assert result['data'][0]['id'] == 'strawberry'
+
+    def test_filter_by_nonempty_array(self):
+        self.validated['querystring'] = {'orders': [1]}
+        result = self.resource.collection_get()
+        assert len(result['data']) == 1
+        assert result['data'][0]['id'] == 'blueberry-1'
+
+    def test_filter_by_empty_object(self):
+        self.validated['querystring'] = {'attributes': {}}
+        result = self.resource.collection_get()
+        assert len(result['data']) == 1
+        assert result['data'][0]['id'] == 'raspberry-1'
+
+    def test_filter_by_nonempty_object(self):
+        self.validated['querystring'] = {'attributes': {'ibu': 25, 'seen_on': '2017-06-01'}}
+        result = self.resource.collection_get()
+        assert len(result['data']) == 1
+        assert result['data'][0]['id'] == 'strawberry'
+
+    def test_filter_by_null(self):
+        self.validated['querystring'] = {'author': None}
+        result = self.resource.collection_get()
+        assert len(result['data']) == 1
+        assert result['data'][0]['id'] == 'strawberry'
diff --git a/tests/test_views_collections.py b/tests/test_views_collections.py
--- a/tests/test_views_collections.py
+++ b/tests/test_views_collections.py
@@ -91,7 +91,7 @@ def test_collections_can_be_filtered_by_arbitrary_attribute(self):
         self.app.put_json('/buckets/beers/collections/moderator',
                           collection,
                           headers=self.headers)
-        resp = self.app.get('/buckets/beers/collections?min_size=2',
+        resp = self.app.get('/buckets/beers/collections?has_size=true&min_size=2',
                             headers=self.headers)
         data = resp.json['data']
         self.assertEqual(len(data), 1)
diff --git a/tests/test_views_groups.py b/tests/test_views_groups.py
--- a/tests/test_views_groups.py
+++ b/tests/test_views_groups.py
@@ -84,7 +84,7 @@ def test_groups_can_be_filtered_by_arbitrary_attribute(self):
         self.app.put_json('/buckets/beers/groups/moderator',
                           group,
                           headers=self.headers)
-        resp = self.app.get('/buckets/beers/groups?min_size=2',
+        resp = self.app.get('/buckets/beers/groups?has_size=true&min_size=2',
                             headers=self.headers)
         data = resp.json['data']
         self.assertEqual(len(data), 1)
diff --git a/tests/test_views_records.py b/tests/test_views_records.py
--- a/tests/test_views_records.py
+++ b/tests/test_views_records.py
@@ -403,3 +403,49 @@ def test_patch_raises_400_on_wrong_path(self):
                             json,
                             headers=self.patch_headers,
                             status=400)
+
+
+class RecordsViewFilterTest(BaseWebTest, unittest.TestCase):
+    collection_url = '/buckets/beers/collections/barley/records'
+
+    RECORDS = [
+        {
+            "id": "strawberry",
+            "flavor": "strawberry",
+            "attributes": {"ibu": 25, "seen_on": "2017-06-01"},
+            "author": None,
+        },
+        {"id": "raspberry-1", "flavor": "raspberry", "attributes": {}},
+        {"id": "raspberry-2", "flavor": "raspberry", "attributes": []},
+        {
+            "id": "raspberry-3",
+            "flavor": "raspberry",
+            "attributes": {"ibu": 25, "seen_on": "2017-06-01", "price": 9.99},
+        },
+    ]
+
+    def setUp(self):
+        super().setUp()
+        self.app.put_json('/buckets/beers', MINIMALIST_BUCKET,
+                          headers=self.headers)
+        self.app.put_json('/buckets/beers/collections/barley',
+                          MINIMALIST_COLLECTION,
+                          headers=self.headers)
+
+        for record in self.RECORDS:
+            self.app.post_json(self.collection_url,
+                               {"data": record},
+                               headers=self.headers)
+
+    def test_records_can_be_filtered_using_json(self):
+        response = self.app.get(self.collection_url + '?flavor="strawberry"',
+                                headers=self.headers)
+        assert len(response.json['data']) == 1
+        assert response.json['data'][0]['id'] == 'strawberry'
+
+    def test_records_can_be_filtered_with_object(self):
+        query = self.collection_url + '?attributes={"ibu": 25, "seen_on": "2017-06-01"}'
+        response = self.app.get(query,
+                                headers=self.headers)
+        assert len(response.json['data']) == 1
+        assert response.json['data'][0]['id'] == 'strawberry'