Repository: encode/django-rest-framework

Issue:
Fix timezone handling for DateTimeField
This should fix #3732, as well as the current/active timezone handling [mentioned](https://github.com/encode/django-rest-framework/issues/3732#issuecomment-326865278) by @RichardForshaw.
Serializer DateTimeField has unexpected timezone information
I have `TIME_ZONE = 'Asia/Kolkata'` and `USE_TZ = True` in my settings.

When I create a new object with the browsable api, the serializer displays the newly created object with datetimes that have a trailing `+5:30`, indicating the timezone. The database is storing the times in UTC.

The unexpected behavior is that when the object is serialized again later, the datetimes are all in UTC format with a trailing `Z`. According to the Django docs on current and default time zone, I would expect the serializer to convert the datetimes to the current time zone, which defaults to the default time zone, which is set by `TIME_ZONE = 'Asia/Kolkata'`.

Am I missing something?



Discussion:

[I asked on stack overflow](http://stackoverflow.com/questions/34275588/djangorestframework-modelserializer-datetimefield-only-converting-to-current-tim) and the conclusion is that the current timezone is only used for user-provided datetimes, as the datetime is serialized as-is after validation (ie after create or update). From the Django documentation, I find it hard to believe that this is intended behavior, but this is an issue with Django itself rather than the Rest Framework, so I will close this issue.

Hi @tomchristie ,

I think this is a DRF bug.
IMHO, the natural use of DRF is to use it  as an output like a django templating.
In a django template the timezone is correctly showed, why the drf serializer do not respects the django timezone setting?
Hi,

looking into the code in `fields.py` at `DatetimeField` class I discover that Django timezone setting is well considered only for the internal value (`to_internal_value()`).

I have seen that using model serializer like this:
```
class MyModelSerializer(ModelSerializer):

    class Meta:
        model = MyModel
        depth = 1
        fields = ('some_field', 'my_date_time_field')
```
for the field `my_date_time_field` (that i think it's a DateTimeField :) ) the timezone  for the representation is None for default (`to_representation()`).

In otherwords, if I'm not wrong, the Django timezone is considered only when the value is wrote into the storage backend.

IMHO I think the return value of `to_representation()` should be like this:
`return self.enforce_timezone(value).strftime(output_format)`
according to the Django `USE_TZ` setting.

I write a pull request for this.

Ciao
Valentino
@vpistis It'd be easier to review this if you could give an example of the API behavior you currently see, and what you'd expect to see (rather than discussing the implementation aspects first)
Set your `TIME_ZONE = 'Asia/Kolkata'` and create a model serializer with a single datetime field, `appointment`.

During create/update, send:
```
{
    "appointment": "2016-12-19T10:00:00"
}
```
and get back:
```
{
    "appointment": "2016-12-19T10:00:00+5:30"
}
```
But if you retrieve or list that object again, you get:
```
{
    "appointment": "2016-12-19T04:30:00Z"
}
```

During creation, if the Z isn't specified, DRF assumes the client is using the timezone specified by `TIME_ZONE` and returns a time formatted to that timezone (by adding the `+5:30` at the end, which would actually be invalid if it were a client-provided time). It would probably make more sense if future accesses also returned a time formatted to that timezone, so the response during retrieval/list were the same as during create/update.

There is also the question of whether to return times in the configured timezone when the trailing Z is provided during creation/update, such that sending:
```
{
    "appointment": "2016-12-19T04:30:00Z"
}
```
returns:
```
{
    "appointment": "2016-12-19T10:00:00+5:30"
}
```
I would be for this, as it keeps the response consistent with the response for lists/retrievals.


Another option entirely would be to always return UTC times, even during creation/updates, but I find that less useful. Either way, having consistent timezones would be preferable to the 50/50ish situation we have right now.
> Set your TIME_ZONE = 'Asia/Kolkata' and create a model serializer with a single datetime field, appointment.
> 
> During create/update, send:
> 
> {
>     "appointment": "2016-12-19T10:00:00"
> }
> and get back:
> 
> {
>     "appointment": "2016-12-19T10:00:00+5:30"
> }
> But if you retrieve or list that object again, you get:
> 
> {
>     "appointment": "2016-12-19T04:30:00Z"
> }

thanx @jonathan-golorry this is the exact behavior that I actually see.
For me the behavior should be like this (using @jonathan-golorry example :) ):

During create/update with default DATETIME_FORMAT, send:
```json
{
    "appointment": "2016-12-19T10:00:00"
}
```
and get back:
```json
{
    "appointment": "2016-12-19T10:00:00+5:30"
}
```
If you retrieve or list that object again , you get:
```json
{
    "appointment": "2016-12-19T10:00:00+5:30"
}
```

IMHO maybe should be a DRF setting to manage this behavior, for example, a setting to force the DateTimeField to be represented with the default timezone.
thanx a lot @tomchristie 
The inconsistency between differing actions is the clincher for reopening here. I'd not realized that was the case. I'd expect us to use UTC throughout by default, although we *could* make that optional.
For a production web app that use DRF 3.4.6, we have resolved with this workaround:
https://github.com/vpistis/django-rest-framework/commit/be62db9080b19998d4de3a1f651a291d691718f6

If anyone wants to submit a pull request that either includes:

* just includes failing test cases, *or*...
* test case + fix

that'd be most welcome.
I have wrote some code to test, but I'm not sure how  use drf test cases. I don't know how can manage django settings to change Timezone and others settings at runtime.
Please, link me some specific example or guide.

thanx 
If you're looking to modify the global test settings, they're located [here](https://github.com/tomchristie/django-rest-framework/blob/master/tests/conftest.py).

If you're trying to override the settings during testing, you can use the `override_settings` [decorator](https://docs.djangoproject.com/en/1.10/topics/testing/tools/#django.test.override_settings).
Would also appreciate this feature. I do feel that the the `USE_TZ` setting should be respected and the values should be converted, similar to standard Django behavior.


First step here would be to write a failing test case that we can add to the current test suite.
Next step would be to start a fix for that case :)
Hi,
for me this is a test case for this feature

``` python
class TestDateTimeFieldTimeZone(TestCase):
    """
    Valid and invalid values for `DateTimeField`.
    """
    from django.utils import timezone

    valid_inputs = {
        '2001-01-01 13:00': datetime.datetime(2001, 1, 1, 13, 00, tzinfo=timezone.get_default_timezone()),
        '2001-01-01T13:00': datetime.datetime(2001, 1, 1, 13, 00, tzinfo=timezone.get_default_timezone()),
        '2001-01-01T13:00Z': datetime.datetime(2001, 1, 1, 13, 00, tzinfo=timezone.get_default_timezone()),
        datetime.datetime(2001, 1, 1, 13, 00): datetime.datetime(2001, 1, 1, 13, 00, tzinfo=timezone.get_default_timezone()),
        datetime.datetime(2001, 1, 1, 13, 00, tzinfo=timezone.UTC()): datetime.datetime(2001, 1, 1, 13, 00,
                                                                                        tzinfo=timezone.get_default_timezone()),
        # Django 1.4 does not support timezone string parsing.
        '2001-01-01T13:00Z': datetime.datetime(2001, 1, 1, 13, 00, tzinfo=timezone.get_default_timezone())
    }
    invalid_inputs = {}
    outputs = {
        # This is not simple, for now I suppose TIME_ZONE = "Europe/Rome"
        datetime.datetime(2001, 1, 1, 13, 00, tzinfo=timezone.get_default_timezone()): '2001-01-01T13:00:00+01:00',
        datetime.datetime(2001, 1, 1, 13, 00, ): '2001-01-01T13:00:00+01:00',
    }

    field = serializers.DateTimeField()
```

In my fork I use some trick to get times in the correct timezone [my 3.6.2_tz_fix](https://github.com/vpistis/django-rest-framework/tree/3.6.2_tz_fix).

I hope this help :)
I see this is closed but I am running drf 3.6.3 and in my postgres database I have this timestamp "2017-07-12 14:26:00-06" but when I get the data using postman I get this "timestamp": "2017-07-12T20:26:00Z".  I looks like it is adding on the -06 hours.

My django settings use tzlocal to set the timezone `TIME_ZONE = str(get_localzone())`. So the timezone is set on startup.

I am using a basic modelSerializer

```
class SnapshotSerializer(serializers.ModelSerializer):
    class Meta:
        model = Snapshot
        resource_name = 'snapshot' 
        read_only_fields = ('id',)
        fields = ('id', 'timestamp', 'snapshot')
```

Am I missing something?
not closed, it's still open :)
the red "closed button" that you see is for the reference issue.
...and you are right, the "bug" is still there ;(
The milestone is changed form 3.6.3 to 3.6.4.
Oh ok. Thank you!

On Jul 12, 2017 5:10 PM, "Valentino Pistis" <notifications@github.com>
wrote:

> not closed, it's still open :)
> the red "closed button" that you see is for the reference issue.
> ...and you are right, the "bug" is still there ;(
> The milestone is changed form 3.6.3 to 3.6.4.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/encode/django-rest-framework/issues/3732#issuecomment-314923582>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AIMBTcx-6PPbi_SOqeLCjeWV1Rb59-Ohks5sNVJ0gaJpZM4G0aRE>
> .
>

Hi,

I'm not sure if this is totally related to the original question, but should DRF be honouring any timezone overrides which are set, as per https://docs.djangoproject.com/en/1.11/ref/utils/#django.utils.timezone.activate ?

I have a system where my users are associated with a timezone, and the API receives naiive datetimes. I expect to be able to convert those datetimes into the current user's timezone, but I notice that `../rest_framework/fields.py` is applying the default timezone (i.e. the one from the django setting file:

```
    def enforce_timezone(self, value):
        field_timezone = getattr(self, 'timezone', self.default_timezone())

        if (field_timezone is not None) and not timezone.is_aware(value):
            try:
                return timezone.make_aware(value, field_timezone)

[...]

    def default_timezone(self):
        return timezone.get_default_timezone() if settings.USE_TZ else None
```
Should this really be using `timezone.get_current_timezone()` as a preference in case the application has set an override, such as in my case?
Hi @RichardForshaw that seems like a distinct issue, but same ball park certainly.

If we can get a decent set of test cases covering the expected behaviour, I think we’d certainly look at a PR here.

My first thought is beyond that is to make sure you’re send the time zone info to the API, rather than relying on the server’s configured time zone. Beyond that I’m also inclined to think you need to be prepared to localise a time on the client. 

But yes, there’s an inconsistency here to be addressed. (Did I mention PRs Welcome? 🙂)
carltongibson/django-filter#750 should be relevant here. I originally based django-filter's timezone handling on DRF, so the changes in 750 could easily be applied here. 
Sorry for my newbie-ness but what exactly is the issue here? The timestamp in my psql database is correct and Django is set to use the correct timezone.  Is there a DRF settings to just make it not convert timestamps?
Hi @michaelaelise — if you look at the example of the data (near the top):

1. They're sending a  datetime without timezone info. (This is a bad move in my book.) 
2. The server is applying its local timezone and it comes back as that (`+5:30` in this case)
3. But later, when you fetch it, it comes back as UTC (`Z`,  for "Zulu" I suppose). 

There's no real issue **on the assumption your client will handle the timezone conversion for you**. (Except maybe No1, because who's to say your client has the same timezone setting as your server...?) 

**But** it's a little inconsistency, surely 2 and 3 should have the same format? (Even though a client will, correctly, accept either as equivalent values.)
I'm inclined to close this. 

* There's no logic error here. 
    * These are the same time:  `"2016-12-19T10:00:00+5:30"` and `"2016-12-19T04:30:00Z"` — to a certain extent, who cares how they come back? 
* Thus, it's not something I can justify allocating time to really. 
* The ticket is 2 years old and no-one has offered a PR. 

I'm happy to look at a PR but I'm not sure I want to really consider this an _Open Issue_. 

Oh, I did not realize that in my original post in this issue thread that the "Z" meant that. 
So DRF is converting the aware datetime to UTC to be handled by the UI/caller?
Thank you for that clarification.
One last thing.   
What if we would like "2016-12-19T10:00:00+5:30" to be returned because we are polling devices in different timezones.
Could this be a setting "RETURN_DATETIME_WITH_TIMEZONE"?

We are using django/drf on edge devices.  So all datetimes being inserted do not care if it is naive or not because the edge device timezone is configured and postgres field will always be accurate for that devices datetime.

The cloud server in the current scenario would then need to know the timezone of each device, it probably will, that just shifts the work from django/drf to the cloud app to handle.
Assuming USE_TZ, DRF is already returning date times with the time zone info. So it’s already doing what you need there. 

The only issue here is whether the same DT is formatted as in one time zone or another. (But they’re still the same time.)



@carltongibson 
> There's no logic error here.
> These are the same time: "2016-12-19T10:00:00+5:30" and "2016-12-19T04:30:00Z" — to a certain extent, who cares how they come back?

IMHO this is the problem: the returned string!
I use Django Time zone settings and all templates return the correct time `"2016-12-19T10:00:00+5:30"` like we expected, but DRF not. DRF return `"2016-12-19T04:30:00Z"`.
Into the client, that consumes my REST apis, ther's no logic, no times conversions or datetime string interpretation.
In other words, I expect that the datetime from a DRF response is identical to the Django Template "response": the server prepare all data for the client and the client only shows it.

Anyway, thanx a lot for your patience, support and your great work to this fantastic project!
@vpistis my point here is just that the represented date is correct, just the representation Is not expected. As soon as you parse that to a native Date, however your language handles that, there is no difference. 

I would expect users to be parsing the date string to a Date, however their client language provides for that, rather than consuming the raw string. 

I accept if you’re consuming the raw string your expectations wont be met here. (But don’t do that: imagine if we sent UNIX timestamps; there’s no way you’d consume those raw. Convert to a proper Date object, whatever that is in your client language.)

I’m really happy to take a PR on this. (I haven’t closed it yet!)

But it’s been nearly two years since reported and nine months since the first comment (yours, a year later). Nobody has even given us a failing test case. It can’t be that important to anyone. As such it’s hard to allocate it time. 

(As such I’m inclined to close it on the basis that we’ll take a PR if one ever turns up)


Resolving patch:
diff --git a/rest_framework/fields.py b/rest_framework/fields.py
--- a/rest_framework/fields.py
+++ b/rest_framework/fields.py
@@ -1125,7 +1125,9 @@ def enforce_timezone(self, value):
         """
         field_timezone = getattr(self, 'timezone', self.default_timezone())
 
-        if (field_timezone is not None) and not timezone.is_aware(value):
+        if field_timezone is not None:
+            if timezone.is_aware(value):
+                return value.astimezone(field_timezone)
             try:
                 return timezone.make_aware(value, field_timezone)
             except InvalidTimeError:
@@ -1135,7 +1137,7 @@ def enforce_timezone(self, value):
         return value
 
     def default_timezone(self):
-        return timezone.get_default_timezone() if settings.USE_TZ else None
+        return timezone.get_current_timezone() if settings.USE_TZ else None
 
     def to_internal_value(self, value):
         input_formats = getattr(self, 'input_formats', api_settings.DATETIME_INPUT_FORMATS)
@@ -1174,6 +1176,7 @@ def to_representation(self, value):
             return value
 
         if output_format.lower() == ISO_8601:
+            value = self.enforce_timezone(value)
             value = value.isoformat()
             if value.endswith('+00:00'):
                 value = value[:-6] + 'Z'


Test patch:
diff --git a/tests/test_fields.py b/tests/test_fields.py
--- a/tests/test_fields.py
+++ b/tests/test_fields.py
@@ -10,12 +10,17 @@
 from django.http import QueryDict
 from django.test import TestCase, override_settings
 from django.utils import six
-from django.utils.timezone import utc
+from django.utils.timezone import activate, deactivate, utc
 
 import rest_framework
 from rest_framework import compat, serializers
 from rest_framework.fields import is_simple_callable
 
+try:
+    import pytz
+except ImportError:
+    pytz = None
+
 try:
     import typings
 except ImportError:
@@ -1168,7 +1173,7 @@ class TestDateTimeField(FieldValues):
         datetime.date(2001, 1, 1): ['Expected a datetime but got a date.'],
     }
     outputs = {
-        datetime.datetime(2001, 1, 1, 13, 00): '2001-01-01T13:00:00',
+        datetime.datetime(2001, 1, 1, 13, 00): '2001-01-01T13:00:00Z',
         datetime.datetime(2001, 1, 1, 13, 00, tzinfo=utc): '2001-01-01T13:00:00Z',
         '2001-01-01T00:00:00': '2001-01-01T00:00:00',
         six.text_type('2016-01-10T00:00:00'): '2016-01-10T00:00:00',
@@ -1230,10 +1235,59 @@ class TestNaiveDateTimeField(FieldValues):
         '2001-01-01 13:00': datetime.datetime(2001, 1, 1, 13, 00),
     }
     invalid_inputs = {}
-    outputs = {}
+    outputs = {
+        datetime.datetime(2001, 1, 1, 13, 00): '2001-01-01T13:00:00',
+        datetime.datetime(2001, 1, 1, 13, 00, tzinfo=utc): '2001-01-01T13:00:00',
+    }
     field = serializers.DateTimeField(default_timezone=None)
 
 
+@pytest.mark.skipif(pytz is None, reason='pytz not installed')
+class TestTZWithDateTimeField(FieldValues):
+    """
+    Valid and invalid values for `DateTimeField` when not using UTC as the timezone.
+    """
+    @classmethod
+    def setup_class(cls):
+        # use class setup method, as class-level attribute will still be evaluated even if test is skipped
+        kolkata = pytz.timezone('Asia/Kolkata')
+
+        cls.valid_inputs = {
+            '2016-12-19T10:00:00': kolkata.localize(datetime.datetime(2016, 12, 19, 10)),
+            '2016-12-19T10:00:00+05:30': kolkata.localize(datetime.datetime(2016, 12, 19, 10)),
+            datetime.datetime(2016, 12, 19, 10): kolkata.localize(datetime.datetime(2016, 12, 19, 10)),
+        }
+        cls.invalid_inputs = {}
+        cls.outputs = {
+            datetime.datetime(2016, 12, 19, 10): '2016-12-19T10:00:00+05:30',
+            datetime.datetime(2016, 12, 19, 4, 30, tzinfo=utc): '2016-12-19T10:00:00+05:30',
+        }
+        cls.field = serializers.DateTimeField(default_timezone=kolkata)
+
+
+@pytest.mark.skipif(pytz is None, reason='pytz not installed')
+@override_settings(TIME_ZONE='UTC', USE_TZ=True)
+class TestDefaultTZDateTimeField(TestCase):
+    """
+    Test the current/default timezone handling in `DateTimeField`.
+    """
+
+    @classmethod
+    def setup_class(cls):
+        cls.field = serializers.DateTimeField()
+        cls.kolkata = pytz.timezone('Asia/Kolkata')
+
+    def test_default_timezone(self):
+        assert self.field.default_timezone() == utc
+
+    def test_current_timezone(self):
+        assert self.field.default_timezone() == utc
+        activate(self.kolkata)
+        assert self.field.default_timezone() == self.kolkata
+        deactivate()
+        assert self.field.default_timezone() == utc
+
+
 class TestNaiveDayLightSavingTimeTimeZoneDateTimeField(FieldValues):
     """
     Invalid values for `DateTimeField` with datetime in DST shift (non-existing or ambiguous) and timezone with DST.