Repository: encode/django-rest-framework

Issue:
Support grouped choices.
**Update from @tomchristie**

**Closing off some related tickets and rolling them into this one...**
- [ ] Include core support.
- [ ] Render in browsable API. - #1636
- [ ] Display as metadata in response to `OPTIONS` requests. - #3101

---

One of our models has a Choicefield with following choices:

```
TYPE_CHOICES = (
        ('Default', (
            (1, 'Option 1'),
            (2, 'Option 2'),
            (3, 'Option 3'))),
        (4, 'Option 4'))
```

Now in the serializer if i have something _self.field.from_native(3)_, we get a ValidationError. Upon inspection of the DRF code we found that logic for valid_value method in the Choicefield is incorrect if you have nested sets with integer keys.

```
def valid_value(self, value):
        """
        Check to see if the provided value is a valid choice.
        """
        for k, v in self.choices:
            if isinstance(v, (list, tuple)):
                # This is an optgroup, so look inside the group for options
                for k2, v2 in v:
                    if value == smart_text(k2):
                        return True
            else:
                if value == smart_text(k) or value == k:
                    return True
        return False
```

Line _if value == smart_text(k2):_ should really be: _if value == smart_text(k2) or value == k2:_



Discussion:
Yup, that seems valid to me.

DjangoCon sprint guidance - check out Django's docs on grouped choices. Demonstrate to yourself that the browsable API in REST framework does not currently support them. Make a pull request with the fix, including a screenshot of before and after. Ensure that `OPTIONS` requests continue to function identically before and after for endpoints that include choice fields.

I'll take a look at solving this at the DjangoCon Sprints today/tomorrow.

Ace.

Tom,

Thanks for the guidance on this. However I don't believe this issue affects the browsable API because integer values are always set as strings in the <option> tag. As a result of this, they are POSTed as strings, and appear as such in the request QueryDict. For this reason, the condition `value == smart_text(k2)` actually evaluates as `True` because smart_text() effectively typecasts `k2` as a string, matching the string sent by the browser and causing validation to succeed in this case.

However, using curl the issue is easily repeatable, e.g.: `curl -H "Content-Type: application/json" -d '{"this_is_an_integer_field": 1 }' http://127.0.0.1:8000/api/v1/widgets/` causes validation to fail unexpectedly as reported by @ameyc.

For this reason on the pull request I'll just include a targeted unit test that demonstrates the main failure case and succeeds with the change. Looking at the test suite structure I'm planning to add it to test_validation.py. Good for you?

Chris

Sounds okay.

This appears to be broken again in the latest release (3.1.3). The test `test_nested_choices` was commented in https://github.com/tomchristie/django-rest-framework/commit/de301f3b6647e1c79a506405a88071ef977418d1#diff-92133d9c634f57bfbbad89754f6713af  and then removed in https://github.com/tomchristie/django-rest-framework/commit/c8764de7881f419c9269913ec3654fc1d904ab2e#diff-92133d9c634f57bfbbad89754f6713af - it is worth noting that [`CHOICES_NESTED`](https://github.com/tomchristie/django-rest-framework/blob/3.1.3/tests/test_validation.py#L148-L155) used by the test is still there.

Has support for grouped choices been dropped?

I will take a look at fixing this


Resolving patch:
diff --git a/rest_framework/fields.py b/rest_framework/fields.py
--- a/rest_framework/fields.py
+++ b/rest_framework/fields.py
@@ -108,6 +108,53 @@ def set_value(dictionary, keys, value):
     dictionary[keys[-1]] = value
 
 
+def to_choices_dict(choices):
+    """
+    Convert choices into key/value dicts.
+
+    pairwise_choices([1]) -> {1: 1}
+    pairwise_choices([(1, '1st'), (2, '2nd')]) -> {1: '1st', 2: '2nd'}
+    pairwise_choices([('Group', ((1, '1st'), 2))]) -> {'Group': {1: '1st', 2: '2nd'}}
+    """
+    # Allow single, paired or grouped choices style:
+    # choices = [1, 2, 3]
+    # choices = [(1, 'First'), (2, 'Second'), (3, 'Third')]
+    # choices = [('Category', ((1, 'First'), (2, 'Second'))), (3, 'Third')]
+    ret = OrderedDict()
+    for choice in choices:
+        if (not isinstance(choice, (list, tuple))):
+            # single choice
+            ret[choice] = choice
+        else:
+            key, value = choice
+            if isinstance(value, (list, tuple)):
+                # grouped choices (category, sub choices)
+                ret[key] = to_choices_dict(value)
+            else:
+                # paired choice (key, display value)
+                ret[key] = value
+    return ret
+
+
+def flatten_choices_dict(choices):
+    """
+    Convert a group choices dict into a flat dict of choices.
+
+    flatten_choices({1: '1st', 2: '2nd'}) -> {1: '1st', 2: '2nd'}
+    flatten_choices({'Group': {1: '1st', 2: '2nd'}}) -> {1: '1st', 2: '2nd'}
+    """
+    ret = OrderedDict()
+    for key, value in choices.items():
+        if isinstance(value, dict):
+            # grouped choices (category, sub choices)
+            for sub_key, sub_value in value.items():
+                ret[sub_key] = sub_value
+        else:
+            # choice (key, display value)
+            ret[key] = value
+    return ret
+
+
 class CreateOnlyDefault(object):
     """
     This class may be used to provide default values that are only used
@@ -1111,17 +1158,8 @@ class ChoiceField(Field):
     }
 
     def __init__(self, choices, **kwargs):
-        # Allow either single or paired choices style:
-        # choices = [1, 2, 3]
-        # choices = [(1, 'First'), (2, 'Second'), (3, 'Third')]
-        pairs = [
-            isinstance(item, (list, tuple)) and len(item) == 2
-            for item in choices
-        ]
-        if all(pairs):
-            self.choices = OrderedDict([(key, display_value) for key, display_value in choices])
-        else:
-            self.choices = OrderedDict([(item, item) for item in choices])
+        self.grouped_choices = to_choices_dict(choices)
+        self.choices = flatten_choices_dict(self.grouped_choices)
 
         # Map the string representation of choices to the underlying value.
         # Allows us to deal with eg. integer choices while supporting either
@@ -1148,6 +1186,38 @@ def to_representation(self, value):
             return value
         return self.choice_strings_to_values.get(six.text_type(value), value)
 
+    def iter_options(self):
+        """
+        Helper method for use with templates rendering select widgets.
+        """
+        class StartOptionGroup(object):
+            start_option_group = True
+            end_option_group = False
+
+            def __init__(self, label):
+                self.label = label
+
+        class EndOptionGroup(object):
+            start_option_group = False
+            end_option_group = True
+
+        class Option(object):
+            start_option_group = False
+            end_option_group = False
+
+            def __init__(self, value, display_text):
+                self.value = value
+                self.display_text = display_text
+
+        for key, value in self.grouped_choices.items():
+            if isinstance(value, dict):
+                yield StartOptionGroup(label=key)
+                for sub_key, sub_value in value.items():
+                    yield Option(value=sub_key, display_text=sub_value)
+                yield EndOptionGroup()
+            else:
+                yield Option(value=key, display_text=value)
+
 
 class MultipleChoiceField(ChoiceField):
     default_error_messages = {
diff --git a/rest_framework/utils/field_mapping.py b/rest_framework/utils/field_mapping.py
--- a/rest_framework/utils/field_mapping.py
+++ b/rest_framework/utils/field_mapping.py
@@ -107,10 +107,10 @@ def get_field_kwargs(field_name, model_field):
                               isinstance(model_field, models.TextField)):
         kwargs['allow_blank'] = True
 
-    if model_field.flatchoices:
+    if model_field.choices:
         # If this model field contains choices, then return early.
         # Further keyword arguments are not valid.
-        kwargs['choices'] = model_field.flatchoices
+        kwargs['choices'] = model_field.choices
         return kwargs
 
     # Ensure that max_length is passed explicitly as a keyword arg,


Test patch:
diff --git a/tests/test_fields.py b/tests/test_fields.py
--- a/tests/test_fields.py
+++ b/tests/test_fields.py
@@ -1107,6 +1107,34 @@ def test_allow_null(self):
         output = field.run_validation(None)
         assert output is None
 
+    def test_iter_options(self):
+        """
+        iter_options() should return a list of options and option groups.
+        """
+        field = serializers.ChoiceField(
+            choices=[
+                ('Numbers', ['integer', 'float']),
+                ('Strings', ['text', 'email', 'url']),
+                'boolean'
+            ]
+        )
+        items = list(field.iter_options())
+
+        assert items[0].start_option_group
+        assert items[0].label == 'Numbers'
+        assert items[1].value == 'integer'
+        assert items[2].value == 'float'
+        assert items[3].end_option_group
+
+        assert items[4].start_option_group
+        assert items[4].label == 'Strings'
+        assert items[5].value == 'text'
+        assert items[6].value == 'email'
+        assert items[7].value == 'url'
+        assert items[8].end_option_group
+
+        assert items[9].value == 'boolean'
+
 
 class TestChoiceFieldWithType(FieldValues):
     """
@@ -1153,6 +1181,66 @@ class TestChoiceFieldWithListChoices(FieldValues):
     field = serializers.ChoiceField(choices=('poor', 'medium', 'good'))
 
 
+class TestChoiceFieldWithGroupedChoices(FieldValues):
+    """
+    Valid and invalid values for a `Choice` field that uses a grouped list for the
+    choices, rather than a list of pairs of (`value`, `description`).
+    """
+    valid_inputs = {
+        'poor': 'poor',
+        'medium': 'medium',
+        'good': 'good',
+    }
+    invalid_inputs = {
+        'awful': ['"awful" is not a valid choice.']
+    }
+    outputs = {
+        'good': 'good'
+    }
+    field = serializers.ChoiceField(
+        choices=[
+            (
+                'Category',
+                (
+                    ('poor', 'Poor quality'),
+                    ('medium', 'Medium quality'),
+                ),
+            ),
+            ('good', 'Good quality'),
+        ]
+    )
+
+
+class TestChoiceFieldWithMixedChoices(FieldValues):
+    """
+    Valid and invalid values for a `Choice` field that uses a single paired or
+    grouped.
+    """
+    valid_inputs = {
+        'poor': 'poor',
+        'medium': 'medium',
+        'good': 'good',
+    }
+    invalid_inputs = {
+        'awful': ['"awful" is not a valid choice.']
+    }
+    outputs = {
+        'good': 'good'
+    }
+    field = serializers.ChoiceField(
+        choices=[
+            (
+                'Category',
+                (
+                    ('poor', 'Poor quality'),
+                ),
+            ),
+            'medium',
+            ('good', 'Good quality'),
+        ]
+    )
+
+
 class TestMultipleChoiceField(FieldValues):
     """
     Valid and invalid values for `MultipleChoiceField`.
diff --git a/tests/test_model_serializer.py b/tests/test_model_serializer.py
--- a/tests/test_model_serializer.py
+++ b/tests/test_model_serializer.py
@@ -181,7 +181,7 @@ class Meta:
                 null_field = IntegerField(allow_null=True, required=False)
                 default_field = IntegerField(required=False)
                 descriptive_field = IntegerField(help_text='Some help text', label='A label')
-                choices_field = ChoiceField(choices=[('red', 'Red'), ('blue', 'Blue'), ('green', 'Green')])
+                choices_field = ChoiceField(choices=(('red', 'Red'), ('blue', 'Blue'), ('green', 'Green')))
         """)
         if six.PY2:
             # This particular case is too awkward to resolve fully across
diff --git a/tests/test_validation.py b/tests/test_validation.py
--- a/tests/test_validation.py
+++ b/tests/test_validation.py
@@ -141,6 +141,8 @@ def test_max_value_validation_fail(self):
         self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
 
 
+# regression tests for issue: 1533
+
 class TestChoiceFieldChoicesValidate(TestCase):
     CHOICES = [
         (0, 'Small'),
@@ -148,6 +150,8 @@ class TestChoiceFieldChoicesValidate(TestCase):
         (2, 'Large'),
     ]
 
+    SINGLE_CHOICES = [0, 1, 2]
+
     CHOICES_NESTED = [
         ('Category', (
             (1, 'First'),
@@ -157,6 +161,15 @@ class TestChoiceFieldChoicesValidate(TestCase):
         (4, 'Fourth'),
     ]
 
+    MIXED_CHOICES = [
+        ('Category', (
+            (1, 'First'),
+            (2, 'Second'),
+        )),
+        3,
+        (4, 'Fourth'),
+    ]
+
     def test_choices(self):
         """
         Make sure a value for choices works as expected.
@@ -168,6 +181,39 @@ def test_choices(self):
         except serializers.ValidationError:
             self.fail("Value %s does not validate" % str(value))
 
+    def test_single_choices(self):
+        """
+        Make sure a single value for choices works as expected.
+        """
+        f = serializers.ChoiceField(choices=self.SINGLE_CHOICES)
+        value = self.SINGLE_CHOICES[0]
+        try:
+            f.to_internal_value(value)
+        except serializers.ValidationError:
+            self.fail("Value %s does not validate" % str(value))
+
+    def test_nested_choices(self):
+        """
+        Make sure a nested value for choices works as expected.
+        """
+        f = serializers.ChoiceField(choices=self.CHOICES_NESTED)
+        value = self.CHOICES_NESTED[0][1][0][0]
+        try:
+            f.to_internal_value(value)
+        except serializers.ValidationError:
+            self.fail("Value %s does not validate" % str(value))
+
+    def test_mixed_choices(self):
+        """
+        Make sure mixed values for choices works as expected.
+        """
+        f = serializers.ChoiceField(choices=self.MIXED_CHOICES)
+        value = self.MIXED_CHOICES[1]
+        try:
+            f.to_internal_value(value)
+        except serializers.ValidationError:
+            self.fail("Value %s does not validate" % str(value))
+
 
 class RegexSerializer(serializers.Serializer):
     pin = serializers.CharField(