Issue 1:
Checklist

I have checked the issues list

for similar or identical enhancement to an existing feature.

I have checked the pull requests list

for existing proposed enhancements.

I have checked the commit log

to find out if the if the same enhancement was already implemented in the

master branch.

I have included all related issues and possible duplicate issues in this issue

(If there are none, check this box anyway).

Related Issues and Possible Duplicates

Related Issues

None

Possible Duplicates

None

Brief Summary

This proposes Celery package to provide a generic delay(func, *args, **kwargs) that wraps any func or method thrown at it.

Design

Historically, to make a function "able" to be delayed via Celery you need to wrap it in a decorator at least. This can be ok for own code, but not for 3rd-party code. Also makes your code coupled with Celery or at least with Celery API for jobs.

"Modern" Python versions can natively pickle functions, so this is no more needed for simpler uses. If Celery provides a shared task that receives a function and its arguments, no custom code is needed for standard user code.

Example:

# mymodule.py
import requests
def ping_website(url):
requests.get(url)

# main.py
from celery.contrib import delay

from .mymodule import ping_website
from .myothermodule import greet

delay(ping_website, 'http://www.google.com.br')
delay(greet)

Note how the Celery dependent part occur only on the main module, not on each of function definitions. Even better, as we do not care about the response, such call could be wrote as:

delay(requests.get, 'http://www.google.com.br')

This is not possible with today's prescription of "create a Task to be able to .delay it", you need to import the 3rd-party funcs and code in a task that wraps it to be able to send to Celery.

Architectural Considerations

Code delay()ed can be considered more convenient. Yet could potentially carry more data than needed or more data than the former way of calling. People can optimize by writing in the former way if needed.

Methods carry the whole self when serialized and this should be informed to the user, as they can consider if is acceptable to pay the cost to do so. Also is needed to inform that such feature needs pickle and reproduce the notice about the danger of accepting pickled jobs.

Proposed Behavior

I propose some code along the following to be included on Celery release and the usage to be featured on Quickstart as the simpler way to sideload your code via Celery

Code to be included:

# celery/contrib.py
@shared_task(serializer='pickle')
def _call(func, *args, **kwargs):
return func(*args, **kwargs)

def delay(func, *args, **kwargs) -> celery.result.AsyncResult:
"""
Defer a call of func(*args, **kwargs) to a Celery worker
"""
return _call.delay(func, *args, **kwargs)

Example to be included in docs:

from celery.contrib import delay

# synchronous version:
my_slow_procedure(3, 4, spam='eggs')

# sideloaded version:
delay(my_slow_procedure, 3, 4, spam='eggs')

Proposed UI/UX

The simpler usage is to delay(fn, *args, **kwargs) arbitrary functions/methods.

Anything more advanced than this could be considered advanced enough to honor the today's way of do it.

Diagrams

N/A

Alternatives

None

Issue 2:
Hi,

I am trying to update celery from 2.5.5 to 3.0.7, I am using redis as broker and latest kombu.

What I notice is that when I switch the workers to 3.0.7 redis open files count goes up a lot.

I noticed that because it hits the max open file user limit setting (is running on ubuntu) and kills redis when all workers are up and running :(

Looking at strace -p redis_pid I see that redis is trying to open /var/log/redis/daemon.log (defined in redis.conf as the logfile value) but fails because of the too many open files limit.

Looking at the logfile I see no output when running 1 single worker @3.0.7

I did this 3 times, moving the same amount of workers (approx 5 workers with concurrency = 20) and when moving from 2.5.5 to 3.0.1 always produce this situation.

I know I can fix this by set higher limits on the redis server but first I would like to know if this is normal behaviour or a side effect of some change.

A [maybe] useful note about this problem:

I was one of the user affected by the empty poll bug 940, when running 3.0.6 with same settings and amount of workers as now I did not experienced this problem (and it run for few hours).