Repository: encode/django-rest-framework

Issue:
using serializer as field fails at import time (e.g. with unit tests) with django 1.7
I'm using DRF 2.4.2 and due to some custom methods / representations, I'm writing some unit tests for my serializers using `py.test` and `pytest-django`, which all worked very well.

I wanted to change a related field (just showed the ID) to the serialized representation so I used an existing serializer as a field. (as explained in the [docs](http://www.django-rest-framework.org/api-guide/relations#nested-relationships)

Running tests using py.test (with pytest-django) will blow up with the giant (sanitized) stack trace shown at the bottom of this post.

Consider the documentation example:

```
class TrackSerializer(serializers.ModelSerializer):
    class Meta:
        model = Track
        fields = ('order', 'title')

class AlbumSerializer(serializers.ModelSerializer):
    tracks = TrackSerializer(many=True)

    class Meta:
        model = Album
        fields = ('album_name', 'artist', 'tracks')
```

and this key part of the stack trace:

```
env/lib/python2.7/site-packages/rest_framework/fields.py:146: in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
env/lib/python2.7/site-packages/rest_framework/fields.py:116: in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)
```

The short version based on my investigation:
- the instantiation of the `TrackSerializer` will happen when this is imported
- when the `TrackSerializer` is instantiated, it creates all the fields from the model and sets up help text for them (I assume for the html form).
- the help text is using django's force_text method which (check the rest of the trace) will use the "underscore" translation service
- this doesn't happen using `runserver`, which I assume is because something to do with the import timing of loading a WSGI app.

Where I'm at:
- I'm not sure if this works using django test runner
- I can work around it in my code, so not blocking
- I'm not sure what the action is, especially if it works on django test runner
  - I was hoping that I could avoid initialization of the serializer by setting the value equal to the class instead of an instance, but no luck, that may be a conceptually simple/ok solution
- this may all get thrown out in a pending major release, since I know that serializers were a big focus of the upcoming development calendar

```
_______________________________________________ ERROR collecting tests/test_serializers.py _______________________________________________
my/app.py: in <module>
    class MySerializer(serializers.ModelSerializer):
hass/endpoint/serializers.py:35: in EndpointSerializer
    related = RelatedSerialzer()
env/lib/python2.7/site-packages/rest_framework/serializers.py:200: in __init__
    self.fields = self.get_fields()
env/lib/python2.7/site-packages/rest_framework/serializers.py:236: in get_fields
    default_fields = self.get_default_fields()
env/lib/python2.7/site-packages/rest_framework/serializers.py:691: in get_default_fields
    serializer_pk_field = self.get_pk_field(pk_field)
env/lib/python2.7/site-packages/rest_framework/serializers.py:818: in get_pk_field
    return self.get_field(model_field)
env/lib/python2.7/site-packages/rest_framework/serializers.py:925: in get_field
    return serializer_field_class(**kwargs)
env/lib/python2.7/site-packages/rest_framework/fields.py:468: in __init__
    super(CharField, self).__init__(*args, **kwargs)
env/lib/python2.7/site-packages/rest_framework/fields.py:272: in __init__
    super(WritableField, self).__init__(source=source, label=label, help_text=help_text)
env/lib/python2.7/site-packages/rest_framework/fields.py:146: in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
env/lib/python2.7/site-packages/rest_framework/fields.py:116: in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)
env/lib/python2.7/site-packages/django/utils/encoding.py:85: in force_text
    s = six.text_type(s)
env/lib/python2.7/site-packages/django/utils/functional.py:144: in __text_cast
    return func(*self.__args, **self.__kw)
env/lib/python2.7/site-packages/django/utils/translation/__init__.py:83: in ugettext
    return _trans.ugettext(message)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:325: in ugettext
    return do_translate(message, 'ugettext')
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:306: in do_translate
    _default = translation(settings.LANGUAGE_CODE)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:209: in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
env/lib/python2.7/site-packages/django/utils/translation/trans_real.py:189: in _fetch
    "The translation infrastructure cannot be initialized before the "
E   AppRegistryNotReady: The translation infrastructure cannot be initialized before the apps registry is ready. Check that you don't make non-lazy gettext calls at import time.
```



Discussion:
I have this same issue but it's happening when I'm starting my wsgi server instead.
Tests are running fine, the app is also running fine when ran with `runserver`.

@Geekfish Is your case also tracing back to the translation infrastructure, or are you seeing something that's related to `AppRegistryNotReady` some other ways?

edit: didn't realize I clarified that I was using `runserver` in the OP, editing in another clarification about the unit test framework I was using

@tomchristie 
The stack trace is very similar:

```
Traceback (most recent call last):
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
    response = self.get_response(request)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 199, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 239, in handle_uncaught_exception
    if resolver.urlconf_module is None:
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 361, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/var/www/tt/api/builds/my_project/staging/urls.py", line 6, in <module>
    from my_project.data.legacy_views import (
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/legacy_views.py", line 12, in <module>
    from my_project.data.serializers import SomeObjectSerializer
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/serializers.py", line 10, in <module>
    class SomeOtherObject(BaseSerializer):
  File "/var/www/tt/api/builds/my_project/staging/my_project/data/serializers.py", line 12, in SomeOtherObject
    identifier = fields.CharField(help_text='Unique string identifier.')
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 470, in __init__
    super(CharField, self).__init__(*args, **kwargs)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 275, in __init__
    super(WritableField, self).__init__(source=source, label=label, help_text=help_text)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 142, in __init__
    self.help_text = strip_multiple_choice_msg(smart_text(help_text))
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/rest_framework/fields.py", line 112, in strip_multiple_choice_msg
    multiple_choice_msg = force_text(multiple_choice_msg)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/encoding.py", line 85, in force_text
    s = six.text_type(s)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/functional.py", line 144, in __text_cast
    return func(*self.__args, **self.__kw)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/__init__.py", line 83, in ugettext
    return _trans.ugettext(message)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 325, in ugettext
    return do_translate(message, 'ugettext')
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 306, in do_translate
    _default = translation(settings.LANGUAGE_CODE)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 209, in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
  File "/var/www/tt/api/virtualenvs/my_project/staging/local/lib/python2.7/site-packages/django/utils/translation/trans_real.py", line 189, in _fetch
    "The translation infrastructure cannot be initialized before the "
django.core.exceptions.AppRegistryNotReady: The translation infrastructure cannot be initialized before the apps registry is ready. Check that you don't make non-lazy gettext calls at import time.
```

Actually I should clarify, the error doesn't happen on startup, it happens when receiving a request.

Are there any ideas on how to proceed with this one? I think I understand why this is happening, but I'm not sure of what's the best way to fix.
Would checking for the django version before deciding whether to apply the fix in `strip_multiple_choice_msg` work?

Thanks for taking on this issue, I appreciate it :)

@Geekfish Does the linked commit fix this for you? I'm curious because (although I think it's a good change nonetheless) it doesn't fix the underlying issue for me - I think this goes deeper.

When I comment out the call to `strip_multiple_choice_msg`, my traceback just changes to this:

```
coachhub/tests/converse/test_viewmodels.py:3: in <module>
    from coachhub.converse.viewmodels import ViewThread
coachhub/converse/viewmodels.py:3: in <module>
    from coachhub.api.serializers import MessageSerializer
coachhub/api/serializers.py:108: in <module>
    class CoachProfileSerializer(Serializer):
coachhub/api/serializers.py:109: in CoachProfileSerializer
    user = UserSerializer(required=True)
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:200: in __init__
    self.fields = self.get_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:236: in get_fields
    default_fields = self.get_default_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:1095: in get_default_fields
    fields = super(HyperlinkedModelSerializer, self).get_default_fields()
../../../.venvs/coachhub/lib/python2.7/site-packages/rest_framework/serializers.py:749: in get_default_fields
    reverse_rels = opts.get_all_related_objects()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:498: in get_all_related_objects
    include_proxy_eq=include_proxy_eq)]
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:510: in get_all_related_objects_with_model
    self._fill_related_objects_cache()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/db/models/options.py:533: in _fill_related_objects_cache
    for klass in self.apps.get_models(include_auto_created=True):
../../../.venvs/coachhub/lib/python2.7/site-packages/django/utils/lru_cache.py:101: in wrapper
    result = user_function(*args, **kwds)
../../../.venvs/coachhub/lib/python2.7/site-packages/django/apps/registry.py:168: in get_models
    self.check_models_ready()
../../../.venvs/coachhub/lib/python2.7/site-packages/django/apps/registry.py:131: in check_models_ready
    raise AppRegistryNotReady("Models aren't loaded yet.")
E   AppRegistryNotReady: Models aren't loaded yet.
```

Rather than hitting the problem when it tries to load translations, it just hits a bit later when it tries to load all related objects for a model. (This is with a `ModelSerializer`).

Basically the issue is that a `ModelSerializer` does a lot of model introspection when it's instantiated, introspection that really shouldn't be done at import time when not all models are loaded yet, and that simply isn't allowed at import time in Django 1.7. I think fixing this may require non-trivial rethinking of how serializers-as-fields work. Maybe something to be considered in the 3.0 serializer rewrite?

(Note: you can work around this by just being careful about where you import your serializers module; you end up doing a lot of imports-within-functions instead of imports at module level.)

:-|

@carljm I've since run into this in a second project, and importing inside functions is my current work-around. 

The cost comes with more friction in adding particular types of tests, but at least I haven't personally seen this new project blow up on requests despite using a serializer as a field (using 2.4.3).

In general it doesn't cause a problem on actual requests, as long as you don't import your serializers from a `models.py` or anything imported from a `models.py` (in other words, a place that Django actively imports for you at setup time.) Usually that's not the case - your serializers are imported from views or somewhere else that only gets imported on-demand.


Resolving patch:
diff --git a/rest_framework/serializers.py b/rest_framework/serializers.py
--- a/rest_framework/serializers.py
+++ b/rest_framework/serializers.py
@@ -22,6 +22,7 @@
 from django.forms import widgets
 from django.utils import six
 from django.utils.datastructures import SortedDict
+from django.utils.functional import cached_property
 from django.core.exceptions import ObjectDoesNotExist
 from rest_framework.settings import api_settings
 
@@ -197,7 +198,6 @@ def __init__(self, instance=None, data=None, files=None,
         self.init_data = data
         self.init_files = files
         self.object = instance
-        self.fields = self.get_fields()
 
         self._data = None
         self._files = None
@@ -212,6 +212,10 @@ def __init__(self, instance=None, data=None, files=None,
     #####
     # Methods to determine which fields to use when (de)serializing objects.
 
+    @cached_property
+    def fields(self):
+        return self.get_fields()
+
     def get_default_fields(self):
         """
         Return the complete set of default fields for the object, as a dict.


Test patch:
diff --git a/tests/test_relations.py b/tests/test_relations.py
--- a/tests/test_relations.py
+++ b/tests/test_relations.py
@@ -102,7 +102,7 @@ class ClassWithQuerysetMethod(object):
         self.assertEqual(value, ['BlogPost object'])
 
     # Regression for #1129
-    def test_exception_for_incorect_fk(self):
+    def test_exception_for_incorrect_fk(self):
         """
         Check that the exception message are correct if the source field
         doesn't exist.
@@ -123,8 +123,9 @@ class Meta:
             (serializers.ModelSerializer,),
             attrs
         )
+        serializer = TestSerializer(data={'name': 'foo'})
         with self.assertRaises(AttributeError):
-            TestSerializer(data={'name': 'foo'})
+            serializer.fields
 
 
 @unittest.skipIf(get_version() < '1.6.0', 'Upstream behaviour changed in v1.6')
diff --git a/tests/test_serializer.py b/tests/test_serializer.py
--- a/tests/test_serializer.py
+++ b/tests/test_serializer.py
@@ -327,7 +327,9 @@ def test_invalid_read_only_fields(self):
         """
         Regression test for #652.
         """
-        self.assertRaises(AssertionError, PersonSerializerInvalidReadOnly, [])
+        serializer = PersonSerializerInvalidReadOnly()
+        with self.assertRaises(AssertionError):
+            serializer.fields
 
     def test_serializer_data_is_cleared_on_save(self):
         """