Repository: celery/celery

Issue:
Celery beat UnicodeDecodeError (Python 3.4) issue
I am using Python 3.4 with Celery 3.1.19

Running without beat it works properly:

```
celery worker --app worker --config=celeryconfig --loglevel=info
```

But with celery beat:

```
celery worker --app worker -B --config=celeryconfig --loglevel=info
```

I got this exception:

```
Traceback (most recent call last):
  File "/env/lib/python3.4/site-packages/kombu/utils/__init__.py", line 320, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 111, in __getitem__
    value = self.cache[key]
KeyError: 'entries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/env/lib/python3.4/site-packages/billiard/process.py", line 292, in _bootstrap
    self.run()
  File "/env/lib/python3.4/site-packages/celery/beat.py", line 530, in run
    self.service.start(embedded_process=True)
  File "/env/lib/python3.4/site-packages/celery/beat.py", line 454, in start
    humanize_seconds(self.scheduler.max_interval))
  File "/env/lib/python3.4/site-packages/kombu/utils/__init__.py", line 322, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/env/lib/python3.4/site-packages/celery/beat.py", line 494, in scheduler
    return self.get_scheduler()
  File "/env/lib/python3.4/site-packages/celery/beat.py", line 489, in get_scheduler
    lazy=lazy)
  File "/env/lib/python3.4/site-packages/celery/utils/imports.py", line 53, in instantiate
    return symbol_by_name(name)(*args, **kwargs)
  File "/env/lib/python3.4/site-packages/celery/beat.py", line 358, in __init__
    Scheduler.__init__(self, *args, **kwargs)
  File "/env/lib/python3.4/site-packages/celery/beat.py", line 185, in __init__
    self.setup_schedule()
  File "/env/lib/python3.4/site-packages/celery/beat.py", line 377, in setup_schedule
    self._store['entries']
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/shelve.py", line 114, in __getitem__
    value = Unpickler(f).load()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 1: ordinal not in range(128)
```

Any ideas? Thanks in advance



Discussion:
It works for me here, maybe you could try removing the celerybeat-schedule file?

It worked! thanks! I removed the `celerybeat-schedule.db` file in the current directory.

Me too, thaks @ask ! Just so you know I've migrated from `Python 2.7.12` to `Python 3.5.3` and it started happening.
Worked for me also, what I did is rename the file to "celerybeat-schedule.db.backup"
in dicrectory Delete (rm) celerybeat-schedule.db and celerybeat-schedule like this 

```
cd project-directory
rm celerybeat-schedule.db celerybeat-schedule
```

Resolving patch:
diff --git a/celery/beat.py b/celery/beat.py
--- a/celery/beat.py
+++ b/celery/beat.py
@@ -568,11 +568,11 @@ def _create_schedule(self):
         for _ in (1, 2):
             try:
                 self._store['entries']
-            except KeyError:
+            except (KeyError, UnicodeDecodeError, TypeError):
                 # new schedule db
                 try:
                     self._store['entries'] = {}
-                except KeyError as exc:
+                except (KeyError, UnicodeDecodeError, TypeError) as exc:
                     self._store = self._destroy_open_corrupted_schedule(exc)
                     continue
             else:


Test patch:
diff --git a/t/unit/app/test_beat.py b/t/unit/app/test_beat.py
--- a/t/unit/app/test_beat.py
+++ b/t/unit/app/test_beat.py
@@ -2,7 +2,7 @@
 import sys
 from datetime import datetime, timedelta, timezone
 from pickle import dumps, loads
-from unittest.mock import Mock, call, patch
+from unittest.mock import MagicMock, Mock, call, patch
 
 import pytest
 
@@ -669,6 +669,38 @@ def test_remove_db(self, remove):
         with pytest.raises(OSError):
             s._remove_db()
 
+    def test_create_schedule_corrupted(self):
+        """
+        Test that any decoding errors that might happen when opening beat-schedule.db are caught
+        """
+        s = create_persistent_scheduler()[0](app=self.app,
+                                             schedule_filename='schedule')
+        s._store = MagicMock()
+        s._destroy_open_corrupted_schedule = Mock()
+        s._destroy_open_corrupted_schedule.return_value = MagicMock()
+
+        # self._store['entries'] will throw a KeyError
+        s._store.__getitem__.side_effect = KeyError()
+        # then, when _create_schedule tries to reset _store['entries'], throw another error
+        expected_error = UnicodeDecodeError("ascii", b"ordinal not in range(128)", 0, 0, "")
+        s._store.__setitem__.side_effect = expected_error
+
+        s._create_schedule()
+        s._destroy_open_corrupted_schedule.assert_called_with(expected_error)
+
+    def test_create_schedule_missing_entries(self):
+        """
+        Test that if _create_schedule can't find the key "entries" in _store it will recreate it
+        """
+        s = create_persistent_scheduler()[0](app=self.app, schedule_filename="schedule")
+        s._store = MagicMock()
+
+        # self._store['entries'] will throw a KeyError
+        s._store.__getitem__.side_effect = TypeError()
+
+        s._create_schedule()
+        s._store.__setitem__.assert_called_with("entries", {})
+
     def test_setup_schedule(self):
         s = create_persistent_scheduler()[0](app=self.app,
                                              schedule_filename='schedule')