Repository: celery/celery

Issue:
Setting task_compression value doesn't actually do anything
## Checklist
* [ ] I have included the output of ``celery -A proj report`` in the issue.
      (if you are not able to do this, then at least specify the Celery
       version affected).
       Celery==4.2.0
* [x] I have verified that the issue exists against the `master` branch of Celery.

## Steps to reproduce
```python3
app = Celery()
app.conf.update(task_compression="gzip")
```

## Expected behavior
I would expect that if I set a global `task_compression` config, my tasks are indeed gzipped. I shouldn't have to manually define compression=gzip on each task I want.

## Actual behavior
I don't think this setting is respected. This could be the wrong place to look, but it seems like `amqp` is using the wrong compression setting when sending a message: https://github.com/celery/celery/blob/7d9300b3b94399eafb5e40a08a0cdc8b05f896aa/celery/app/amqp.py#L548. `default_compressor` is being set here: https://github.com/celery/celery/blob/7d9300b3b94399eafb5e40a08a0cdc8b05f896aa/celery/app/amqp.py#L483 - should this read `default_compressor = self.app.conf.task_compression` ? Or am I missing something?




Discussion:
sorry for the late. current status?
I can confirm that - working the same in my case.

Exactly as @asgoel points out - it is an obvious bug in the code and setting the `task_compression` does nothing, `result_compression` is used instead.

What is even more interesting `result_compression` compresses... just the tasks headers and results are still not compressed, because of the producer call in `AMQPBackend.store_result` looking like:

```
        with self.app.amqp.producer_pool.acquire(block=True) as producer:
            producer.publish(
                payload,
                exchange=self.exchange,
                routing_key=routing_key,
                correlation_id=correlation_id,
                serializer=self.serializer,
                retry=True, retry_policy=self.retry_policy,
                declare=self.on_reply_declare(task_id),
                delivery_mode=self.delivery_mode,
            )
```

So it is missing the compression kwarg.
It also seems that the `result_compression` does not work for Redis or any other backend too.
which version of celery are you using? 4.4.0?
@auvipy 4.4.0 right now. It compresses task messages, but results are not compressed. Checked that directly in redis too.
Same problem with Celery 4.4.7. No observable change in the messages looking in REDIS when my celeryconfig.py:task_compression="gzip"

When I submit the task with apply_async(compress="gzip") it does work
https://github.com/celery/celery/blob/5b86b35c81ea5a1fbfd439861f4fee6813148d16/celery/app/task.py#L303
I would think the "task_compression" would be referenced here in the code  but it is not.
   from_config = (
        ('serializer', 'task_serializer'),
        ('rate_limit', 'task_default_rate_limit'),
        ('priority', 'task_default_priority'),
        ('track_started', 'task_track_started'),
        ('acks_late', 'task_acks_late'),
        ('acks_on_failure_or_timeout', 'task_acks_on_failure_or_timeout'),
        ('reject_on_worker_lost', 'task_reject_on_worker_lost'),
        ('ignore_result', 'task_ignore_result'),
        ('store_errors_even_if_ignored', 'task_store_errors_even_if_ignored'),
    )
i solved this issue with custom serialization function. like this
```python
def pickle_bz2_encoder(data):
    return bz2.compress(pickle.dumps(data))

def pickle_bz2_decoder(data):
    return pickle.loads(bz2.decompress(data))

serialization.register('pickle-bzip2', pickle_bz2_encoder, pickle_bz2_decoder, 'application/x-pickle-bz2', 'binary')
```

and config
```python
result_serializer = 'pickle-bzip2'
accept_content = ['pickle', 'pickle-bzip2']
```
allowing using compression will be extremely helpful
Hello, 
is the `result_compression` option actually doing anything as well for the task results?

I have done a global search for it in `celery` (5.2.3) and this is the only place it is used:

```python
    def _create_task_sender(self):
        default_retry = self.app.conf.task_publish_retry
        default_policy = self.app.conf.task_publish_retry_policy
        default_delivery_mode = self.app.conf.task_default_delivery_mode
        default_queue = self.default_queue
        queues = self.queues
        send_before_publish = signals.before_task_publish.send
        before_receivers = signals.before_task_publish.receivers
        send_after_publish = signals.after_task_publish.send
        after_receivers = signals.after_task_publish.receivers

        send_task_sent = signals.task_sent.send   # XXX compat
        sent_receivers = signals.task_sent.receivers

        default_evd = self._event_dispatcher
        default_exchange = self.default_exchange

        default_rkey = self.app.conf.task_default_routing_key
        default_serializer = self.app.conf.task_serializer
        default_compressor = self.app.conf.result_compression
```

which tells me it is used for task compression , not results?

Was this ever working at all?
You're all absolutely right, it should be `self.app.conf.task_compression` in this instance.
Welp. It seems like compressing results isn't supported and only the setting exists. Is anyone up for a quick PR to fix that?

Resolving patch:
diff --git a/celery/app/amqp.py b/celery/app/amqp.py
--- a/celery/app/amqp.py
+++ b/celery/app/amqp.py
@@ -447,7 +447,7 @@ def _create_task_sender(self):
 
         default_rkey = self.app.conf.task_default_routing_key
         default_serializer = self.app.conf.task_serializer
-        default_compressor = self.app.conf.result_compression
+        default_compressor = self.app.conf.task_compression
 
         def send_task_message(producer, name, message,
                               exchange=None, routing_key=None, queue=None,


Test patch:
diff --git a/t/unit/app/test_amqp.py b/t/unit/app/test_amqp.py
--- a/t/unit/app/test_amqp.py
+++ b/t/unit/app/test_amqp.py
@@ -205,8 +205,7 @@ def test_as_task_message_without_utc(self):
         self.app.amqp.as_task_v1(uuid(), 'foo', countdown=30, expires=40)
 
 
-class test_AMQP:
-
+class test_AMQP_Base:
     def setup(self):
         self.simple_message = self.app.amqp.as_task_v2(
             uuid(), 'foo', create_sent_event=True,
@@ -215,6 +214,9 @@ def setup(self):
             uuid(), 'foo', create_sent_event=False,
         )
 
+
+class test_AMQP(test_AMQP_Base):
+
     def test_kwargs_must_be_mapping(self):
         with pytest.raises(TypeError):
             self.app.amqp.as_task_v2(uuid(), 'foo', kwargs=[1, 2])
@@ -336,7 +338,7 @@ def update_conf_runtime_for_tasks_queues(self):
         assert router != router_was
 
 
-class test_as_task_v2:
+class test_as_task_v2(test_AMQP_Base):
 
     def test_raises_if_args_is_not_tuple(self):
         with pytest.raises(TypeError):
@@ -368,8 +370,27 @@ def test_eta_to_datetime(self):
         )
         assert m.headers['eta'] == eta.isoformat()
 
-    def test_callbacks_errbacks_chord(self):
+    def test_compression(self):
+        self.app.conf.task_compression = 'gzip'
+
+        prod = Mock(name='producer')
+        self.app.amqp.send_task_message(
+            prod, 'foo', self.simple_message_no_sent_event,
+            compression=None
+        )
+        assert prod.publish.call_args[1]['compression'] == 'gzip'
+
+    def test_compression_override(self):
+        self.app.conf.task_compression = 'gzip'
+
+        prod = Mock(name='producer')
+        self.app.amqp.send_task_message(
+            prod, 'foo', self.simple_message_no_sent_event,
+            compression='bz2'
+        )
+        assert prod.publish.call_args[1]['compression'] == 'bz2'
 
+    def test_callbacks_errbacks_chord(self):
         @self.app.task
         def t(i):
             pass