Repository: dmwm/wmcore
Issue: Wrong gzip compression recognized/identified by header

**Impact of the bug**
Reqmgr2, WorkQueue , maybe some others as well

**Describe the bug**
While validating the April release we have noticed an error coming from the newly introduced `decompress` function at the `pycurl_manager` module. This [[1]](https://github.com/dmwm/WMCore/issues/11036) is the PR with which we have included this function. It serves to decompress zipped streams through HTTP, which are identified by proper negotiation between the server and the client about the data content compression through the HTTP headers. It looks like for some cases the content's compression has been wrongly set to `gzip` [2].  More on the way how the identification of the proper compression happens is discussed in the issue/PR above. One thing is suspicious in the log message for the current bug - at the end it says: 
```
Getting from Cache due to: CouchNotFoundError - reason: Object Not Found, data: {} result: None     
```
which may be a sign for reading from a cache file which is not having the proper compression. 

**How to reproduce it**
Not clear yet. But if it is related to reading gzip data from the cache it might be by trying to provide the proper header from the client and then try to read  the data from an old cache file, which is not having the contents compressed (Still to be confirmed).

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context and error message**
[1]
https://github.com/dmwm/WMCore/issues/11036

[2]

[05/Apr/2022:02:49:24]  Updating request "tivanov_ReReco_Parents_HG2204_Val_220401_123539_5532" with these user-provided args: {'total_jobs': 0, 'input_events': 0, 'input_lumis': 0, 'input_n
um_files': 0, 'RequestName': 'tivanov_ReReco_Parents_HG2204_Val_220401_123539_5532'}
ERROR:root:While processing decompress function with headers: {'Date': 'Tue, 05 Apr 2022 00:49:24 GMT', 'Server': 'CouchDB/1.6.1 (Erlang OTP/R16B03-1)', 'Set-Cookie': 'cms-node=624b9214001f3
6ffe1331bdb2ae875f748469a9f8732fb707323f8b02b6822441ae6cccea426fbbd05c2c42b;path=/;secure;httponly', 'ETag': '"W1AVV/xAOj0Xndl63XAMIA=="', 'Content-Type': 'application/json', 'Content-MD5': 
'W1AVV/xAOj0Xndl63XAMIA==', 'Content-Length': '17414', 'Content-Encoding': 'gzip', 'Cache-Control': 'must-revalidate', 'Accept-Ranges': 'none', 'CMS-Server-Time': 'D=12028 t=1649119764746333
'}, we were unable to decompress gzip content. Details: Not a gzipped file (b'cc')
Traceback (most recent call last):
  File "/data/srv/HG2204b/sw/slc7_amd64_gcc630/cms/reqmgr2/2.0.2.pre2/lib/python3.8/site-packages/WMCore/Services/pycurl_manager.py", line 80, in decompress
    return gzip.decompress(body)
  File "/data/srv/HG2204b/sw/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/gzip.py", line 548, in decompress
    return f.read()
  File "/data/srv/HG2204b/sw/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/gzip.py", line 292, in read
    return self._buffer.read(size)
  File "/data/srv/HG2204b/sw/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/gzip.py", line 479, in read
    if not self._read_gzip_header():
  File "/data/srv/HG2204b/sw/slc7_amd64_gcc630/external/python3/3.8.2-comp/lib/python3.8/gzip.py", line 427, in _read_gzip_header
    raise BadGzipFile('Not a gzipped file (%r)' % magic)
gzip.BadGzipFile: Not a gzipped file (b'cc')
Getting from Cache due to: CouchNotFoundError - reason: Object Not Found, data: {} result: None


Changed files:
src/python/WMCore/Services/pycurl_manager.py

diff --git a/src/python/WMCore/Services/pycurl_manager.py b/src/python/WMCore/Services/pycurl_manager.py
index f840ceead..2050f686d 100644
--- a/src/python/WMCore/Services/pycurl_manager.py
+++ b/src/python/WMCore/Services/pycurl_manager.py
@@ -46,6 +46,7 @@ from future.utils import viewitems
 
 
 # system modules
+import copy
 import json
 import gzip
 import logging
@@ -81,7 +82,8 @@ def decompress(body, headers):
     except Exception as exc:
         logger = logging.getLogger()
         msg = "While processing decompress function with headers: %s, " % headers
-        msg += "we were unable to decompress gzip content. Details: %s" % str(exc)
+        msg += "we were unable to decompress gzip content. Details: %s. " % str(exc)
+        msg += "Considering response body as uncompressed."
         logger.exception(msg)
         return body
 
@@ -202,16 +204,20 @@ class RequestHandler(object):
         curl.setopt(pycurl.FOLLOWLOCATION, self.followlocation)
         curl.setopt(pycurl.MAXREDIRS, self.maxredirs)
 
-        # also accepts encoding/compression algorithms
-        if headers and headers.get("Accept-Encoding"):
-            if isinstance(headers["Accept-Encoding"], basestring):
-                curl.setopt(pycurl.ENCODING, headers["Accept-Encoding"])
+        # If ACCEPT_ENCODING is set to the encoding string, then libcurl
+        # will automatically decode the response object according to the
+        # Content-Enconding received
+        # More info: https://curl.se/libcurl/c/CURLOPT_ACCEPT_ENCODING.html
+        thisHeaders = copy.deepcopy(headers)
+        if thisHeaders and thisHeaders.get("Accept-Encoding"):
+            if isinstance(thisHeaders["Accept-Encoding"], basestring):
+                curl.setopt(pycurl.ACCEPT_ENCODING, thisHeaders.pop("Accept-Encoding"))
             else:
                 logging.warning("Wrong data type for header 'Accept-Encoding': %s",
-                                type(headers["Accept-Encoding"]))
+                                type(thisHeaders["Accept-Encoding"]))
         else:
             # add gzip encoding by default
-            curl.setopt(pycurl.ENCODING, 'gzip')
+            curl.setopt(pycurl.ACCEPT_ENCODING, 'gzip')
 
         if cookie and url in cookie:
             curl.setopt(pycurl.COOKIEFILE, cookie[url])
@@ -242,15 +248,15 @@ class RequestHandler(object):
 
         if verb in ('POST', 'PUT'):
             # only these methods (and PATCH) require this header
-            headers["Content-Length"] = str(len(encoded_data))
+            thisHeaders["Content-Length"] = str(len(encoded_data))
 
         # we must pass url as a string data-type, otherwise pycurl will fail with error
         # TypeError: invalid arguments to setopt
         # see https://curl.haxx.se/mail/curlpython-2007-07/0001.html
         curl.setopt(pycurl.URL, encodeUnicodeToBytes(url))
-        if headers:
+        if thisHeaders:
             curl.setopt(pycurl.HTTPHEADER, \\
-                [encodeUnicodeToBytes("%s: %s" % (k, v)) for k, v in viewitems(headers)])
+                [encodeUnicodeToBytes("%s: %s" % (k, v)) for k, v in viewitems(thisHeaders)])
         bbuf = BytesIO()
         hbuf = BytesIO()
         curl.setopt(pycurl.WRITEFUNCTION, bbuf.write)
@@ -275,12 +281,11 @@ class RequestHandler(object):
         """Debug callback implementation"""
         print("debug(%d): %s" % (debug_type, debug_msg))
 
-    def parse_body(self, data, headers, decode=False):
+    def parse_body(self, data, decode=False):
         """
         Parse body part of URL request (by default use json).
         This method can be overwritten.
         """
-        data = decompress(data, headers)
         if decode:
             try:
                 res = json.loads(data)
@@ -316,7 +321,7 @@ class RequestHandler(object):
             if verb == 'HEAD':
                 data = ''
             else:
-                data = self.parse_body(bbuf.getvalue(), header.header, decode)
+                data = self.parse_body(bbuf.getvalue(), decode)
         else:
             data = bbuf.getvalue()
             data = decompress(data, header.header)