Repository: agronholm/cbor2 (language: python)

Issue:
Failing unit-test
### Things to check first

- [X] I have searched the existing issues and didn't find my bug already reported there

- [X] I have checked that my bug is still present in the latest release


### cbor2 version

5.6.0

### Python version

3.11.5

### What happened?

I ran `python -m pytest -sv tests/` and one of the date tests failed:

```
tests/test_decoder.py:389: in test_date
    assert decoded == expected
E   assert datetime.date(2018, 12, 30) == datetime.date(2018, 12, 31)
```

### How can we reproduce the bug?

run `python -m pytest -sv tests/test_decoder.py`


Discussion / hints:
I can't reproduce this, nor does this happen in the CI. I need you to dig deeper and see what's going on.
With Python 3.11.5, I run `python setup.py build` to get a `/build/lib.linux-x86_64-cpython-311/_cbor2.cpython-311-x86_64-linux-gnu.so` which I then move to "." so that it is picked up by python.

Then, running the above command causes the error.

This is a Debian stable derived OS (PureOS 10 / Byzantium). I also tried on a clean clone on a Debian 12 (bookworm) system (with python 3.11.2) with the same results.

Edit: I didn't explicitly say yet, but it's only with the C version, not the pure-python version.
Ok, let me change the question then: can you produce a `Dockerfile` that lets me reproduce the problem over here?
So far, the answer is "no". I'm guessing it's "something locales? maybe?"

Here's what I have so far (this works in container, still fails on host system):

```
FROM debian

RUN apt update 
RUN apt install -y locales python3 python3-setuptools python3-pip python3-wheel git build-essential python3-venv && rm -rf /var/lib/apt/lists/* && localedef -i en_CA -c -f UTF-8 -A /usr/share/locale/locale.alias en_CA.UTF-8
ENV LANG en_CA.utf8
RUN cd / && git clone https://github.com/agronholm/cbor2.git
RUN env
RUN cd /cbor2 && python3 -m venv venv && ./venv/bin/pip install .[test] && python3 setup.py build && mv /cbor2/build/lib.linux-x86_64-cpython-311/_cbor2.cpython-311-x86_64-linux-gnu.so . && ./venv/bin/python3 -m pytest -sv tests/test_decoder.py
RUN locale
```

One thing I do notice when looking, though, is that the Python code for "semantic type 100" is doing `date.fromordinal(x + 719163)` whereas the corresponding C code is doing `PyDate_FromTimestamp()` and multiplying the encoded value by seconds-per-day (`24*60*60`).

So I'm not claiming either is "wrong", but they're definitely "different".

Any ideas on what might be different  from host -> docker that might explain this? (They're both now a Canadian utf8 locale ...)
Ok, looking at the docs, it would seem that `date.fromtimestamp()` might indeed be affected by the locale. There is, however, no direct C API equivalent for `date.fromordinal()` – perhaps I could call that as I would call any Python function though?
I guess you're on a time zone with a negative UTC offset (if in Canada), and that might affect the results?
yes, I'm UTC-6 or -7 normally -- but I did try my best to have the locale set to "canada, english" in the Docker container (however, I don't do much docker, so maybe that part isn't working as I intended)
p.s. since you're already depending on Hypothesis, it might be good to have more tests that let it explore weird cases (e.g. a "round-trip dates" test that uses `hypothesis.strategies.dates`)

This is (indirectly) how we found the other bug (with "binary strings over 65536 bytes")...
Unfortunately it seems like hypothesis also considerably slows down each test it's applied on. If I were to do the same with each type supported by cbor2, I fear it might make the test suite many times slower. I could be persuaded otherwise with some benchmarks though.
Yes, it tries 100 "examples" for each `@given` by default I believe. You can set it to do substantially fewer (e.g. during development, to speed up test-runs). Here's one way to do that: https://github.com/tahoe-lafs/magic-folder/blob/88f6c9266fb2218fac5608911f8130aea6ce972c/src/magic_folder/test/__init__.py#L59

It's definitely a bit of a tradeoff: you get "many tests", shrinking and theoretically better coverage -- but it takes more time. To be fair, it would also take more time if you had 100 test-cases for each kind, but ...

I think the most "bang" here can be for things that you _know_ can be weird, like UTF strings. On the flip side, having a couple very-specific cases for e.g. binary strings (where the actual content is unlikely to matter in this case) might be a better answer to the trade-off.

Another way to approach the slowdown problem would be to use `@pytest.mark` to mark those tests as "slow" and only run them in CI by default (so you get some property-testing / "fuzzing"-ish tests in CI).
I have a fix for this almost ready, but I'm currently on vacation so it'll have to wait until at least next Sunday.

Fix patch:
diff --git a/docs/versionhistory.rst b/docs/versionhistory.rst
index 2c67eaf..0c4c0b6 100644
--- a/docs/versionhistory.rst
+++ b/docs/versionhistory.rst
@@ -16,6 +16,7 @@ This library adheres to `Semantic Versioning <http://semver.org/>`_.
   string that contained only ASCII characters
 - Changed the return type annotations of ``cbor2.load()`` and ``cbor2.load()`` to return ``Any``
   instead of ``object`` so as not to force users to make type casts
+- Fixed decoding of epoch-based dates being affected by the local time zone in the C extension
 
 **5.6.1** (2024-02-01)
 
diff --git a/pyproject.toml b/pyproject.toml
index 308caf6..e366eba 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -97,6 +97,8 @@ minversion = 4.0.0
 commands = python -m pytest {posargs}
 package = editable
 extras = test
+setenv =
+  TZ=America/New_York
 
 [testenv:docs]
 extras = doc
diff --git a/source/decoder.c b/source/decoder.c
index a986b4c..2f34f90 100644
--- a/source/decoder.c
+++ b/source/decoder.c
@@ -1338,15 +1338,15 @@ static PyObject *
 CBORDecoder_decode_epoch_date(CBORDecoderObject *self)
 {
     // semantic type 100
-    PyObject *num, *tuple, *ret = NULL;
+    PyObject *num, *ordinal, *ret = NULL;
 
     num = decode(self, DECODE_NORMAL);
     if (num) {
         if (PyNumber_Check(num)) {
-            tuple = PyTuple_Pack(1, PyNumber_Multiply(num, PyLong_FromLong(24 * 60 * 60)));
-            if (tuple) {
-                ret = PyDate_FromTimestamp(tuple);
-                Py_DECREF(tuple);
+            ordinal = PyNumber_Add(num, _CBOR2_date_ordinal_offset);
+            if (ordinal) {
+                ret = PyObject_CallMethodObjArgs(PyDateTime_Date, _CBOR2_str_fromordinal, ordinal, NULL);
+                Py_DECREF(ordinal);
             }
         } else {
             PyErr_Format(
diff --git a/source/encoder.c b/source/encoder.c
index 7dbda35..f612e5d 100644
--- a/source/encoder.c
+++ b/source/encoder.c
@@ -1056,7 +1056,7 @@ CBOREncoder_encode_date(CBOREncoderObject *self, PyObject *value)
         tmp = PyObject_CallMethodObjArgs(
                 value, _CBOR2_str_toordinal, NULL);
         if (tmp && fp_write(self, "\xD8\x64", 2) == 0) {
-            ret = CBOREncoder_encode_int(self, PyNumber_Subtract(tmp, PyLong_FromLong(719163)));
+            ret = CBOREncoder_encode_int(self, PyNumber_Subtract(tmp, _CBOR2_date_ordinal_offset));
         }
     } else {
         tmp = PyObject_CallMethodObjArgs(
diff --git a/source/module.c b/source/module.c
index 1b8391c..7c96a91 100644
--- a/source/module.c
+++ b/source/module.c
@@ -621,6 +621,7 @@ _CBOR2_init_thread_locals(void)
 
 PyObject *_CBOR2_empty_bytes = NULL;
 PyObject *_CBOR2_empty_str = NULL;
+PyObject *_CBOR2_date_ordinal_offset = NULL;
 PyObject *_CBOR2_str_as_string = NULL;
 PyObject *_CBOR2_str_as_tuple = NULL;
 PyObject *_CBOR2_str_bit_length = NULL;
@@ -638,6 +639,7 @@ PyObject *_CBOR2_str_encode_date = NULL;
 PyObject *_CBOR2_str_Fraction = NULL;
 PyObject *_CBOR2_str_fromtimestamp = NULL;
 PyObject *_CBOR2_str_FrozenDict = NULL;
+PyObject *_CBOR2_str_fromordinal = NULL;
 PyObject *_CBOR2_str_getvalue = NULL;
 PyObject *_CBOR2_str_groups = NULL;
 PyObject *_CBOR2_str_ip_address = NULL;
@@ -971,6 +973,7 @@ PyInit__cbor2(void)
     INTERN_STRING(Fraction);
     INTERN_STRING(fromtimestamp);
     INTERN_STRING(FrozenDict);
+    INTERN_STRING(fromordinal);
     INTERN_STRING(getvalue);
     INTERN_STRING(groups);
     INTERN_STRING(ip_address);
@@ -1000,6 +1003,9 @@ PyInit__cbor2(void)
 
 #undef INTERN_STRING
 
+    if (!_CBOR2_date_ordinal_offset &&
+            !(_CBOR2_date_ordinal_offset = PyLong_FromLong(719163)))
+        goto error;
     if (!_CBOR2_str_utc_suffix &&
             !(_CBOR2_str_utc_suffix = PyUnicode_InternFromString("+00:00")))
         goto error;
diff --git a/source/module.h b/source/module.h
index edcb139..52a1820 100644
--- a/source/module.h
+++ b/source/module.h
@@ -49,6 +49,7 @@ extern PyObject *_CBOR2_str_encode_date;
 extern PyObject *_CBOR2_str_Fraction;
 extern PyObject *_CBOR2_str_fromtimestamp;
 extern PyObject *_CBOR2_str_FrozenDict;
+extern PyObject *_CBOR2_str_fromordinal;
 extern PyObject *_CBOR2_str_getvalue;
 extern PyObject *_CBOR2_str_groups;
 extern PyObject *_CBOR2_str_ip_address;


Test patch:
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 2a96a72..81bad50 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -37,6 +37,8 @@ jobs:
         CFLAGS: "-UNDEBUG"
     - name: Test with pytest
       run: coverage run -m pytest -v
+      env:
+        TZ: America/New_York
     - name: Generate coverage report
       run: coverage xml
     - name: Upload Coverage