Repository: djangopackages/djangopackages

Issue:
🚜  Migrate to a task queue (Add Celery)
I noticed that the package_updater was running three copies and seemed to be stuck. I noticed that there was a 5-second sleep between every package that is in our database. 

This works as a way to workaround rate limit issues, but we can move to a task queue and avoid blocking via sleep() completely. We should get either 5k (newer APIs might even go as large as 15k an hour) and be able to manage this both from the front and backend. 

I'd like to see the /post-data/ endpoint move the fetching logic to a queued task to cut down on IO too. 


Discussion:
Hi @jefftriplett! I want to work on this ticket :)
I'm thinking that we can use the same celery docker config as defined by Cookiecutter Django. What do you think?
We are currently running `package_updater` via cron schedule.

```bash
crontab -l

12 23 * * * /usr/local/bin/docker-compose -f /code/djangopackages/docker-compose.yml run --rm django-a python manage.py searchv2_build
36 17 * * * /usr/local/bin/docker-compose -f /code/djangopackages/docker-compose.yml run --rm django-a python manage.py package_updater
48 21 * * * /usr/local/bin/docker-compose -f /code/djangopackages/docker-compose.yml run --rm django-a python manage.py pypi_updater

```
@luzfcb thanks for the cron details. How do you feel about offloading some of the heavy loadings to Celery or some other task queue? I think it could clean a few things up and we wouldn't have as many blocking processes running either on the web side or tasks that pile up on the backend. 
> How do you feel about offloading some of the heavy loadings to Celery or some other task queue? I think it could clean a few things up and we wouldn't have as many blocking processes running either on the web side or tasks that pile up on the backend.

@jefftriplett No problem for me. I think it's the right thing to do. 


> I'm thinking that we can use the same celery docker config as defined by Cookiecutter Django. What do you think?

@nicysneiros this seems like a good starting point IMO. Let us know if you have any question and thanks 🙏 



Resolving patch:
diff --git a/settings/celery.py b/settings/celery.py
new file mode 100644
--- /dev/null
+++ b/settings/celery.py
@@ -0,0 +1,23 @@
+import os
+
+from celery import Celery
+
+
+# set the default Django settings module for the 'celery' program.
+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings.docker")
+
+app = Celery("config")
+
+# Using a string here means the worker doesn't have to serialize
+# the configuration object to child processes.
+# - namespace='CELERY' means all celery-related configuration keys
+#   should have a `CELERY_` prefix.
+app.config_from_object("django.conf:settings", namespace="CELERY")
+
+# Load task modules from all registered Django app configs.
+app.autodiscover_tasks()
+
+
+@app.task(bind=True)
+def debug_task(self):
+    print(f"Request: {self.request!r}")
diff --git a/settings/docker.py b/settings/docker.py
--- a/settings/docker.py
+++ b/settings/docker.py
@@ -193,3 +193,14 @@
 PACKAGE_HEALTHCHECK_URL = env.str("PACKAGE_HEALTHCHECK_URL", "")
 PYPI_HEALTHCHECK_URL = env.str("PYPI_HEALTHCHECK_URL", "")
 SEARCHV2_HEALTHCHECK_URL = env.str("SEARCHV2_HEALTHCHECK_URL", "")
+
+# Configure Redis
+REDIS_HOST = env("REDIS_HOST", default="redis")
+
+# Configure Celery
+CELERY_BROKER_URL = f"redis://{REDIS_HOST}:6379"
+CELERY_RESULT_BACKEND = f"redis://{REDIS_HOST}:6379"
+CELERY_ACCEPT_CONTENT = ["application/json"]
+CELERY_TASK_SERIALIZER = "json"
+CELERY_RESULT_SERIALIZER = "json"
+CELERY_TIMEZONE = "UTC"


Test patch: