What is the correct way to use requests.Session with celery ??

I have a fairly basic setup of celery running on `4.4.6` and `requests[security]` `2.24.0`  most of the task look like.

```py
@app.task(bind=True)
def get_some_data_from_a_server_and_update_records(self, ...):
   # Make a request using requests via a Session that is maintained by the instance of the SDK
   my_sdk_client = Client()
   
   response = my_sdk_client.session.get(...)

   # Update my database

```

Django celery settings

```py
# Celery
CELERY_BROKER_URL = REDIS_URL
CELERY_RESULT_BACKEND = REDIS_URL

CELERY_TASK_QUEUES = (
    Queue('critical', Exchange('critical'), routing_key='critical'),
    Queue('high', Exchange('high'), routing_key='high'),
    Queue('medium', Exchange('medium'), routing_key='medium'),
    Queue('low', Exchange('low'), routing_key='low'),
)
CELERY_TASK_DEFAULT_QUEUE = 'medium'
CELERY_TASK_IGNORE_RESULT = True
CELERY_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_WORKER_HIJACK_ROOT_LOGGER = False
CELERY_WORKER_POOL = 'threads'
```


Which runs on GKE using multiple pods i.e 8 - 10 pods with each worker reading from a dedicated queue

Unfortunately this results in a number of `ConnectionErrors` as shown below

```py
File /venv/lib/python3.6/site-packages/requests/sessions.py line 578 in post
return self.request('POST', url, data=data, json=json, **kwargs)
File /venv/lib/python3.6/site-packages/requests/sessions.py line 530 in request
resp = self.send(prep, **send_kwargs)
File /venv/lib/python3.6/site-packages/requests/sessions.py line 643 in send
r = adapter.send(request, **kwargs)
File /venv/lib/python3.6/site-packages/requests/adapters.py line 498 in send args locals
raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', OSError(0, 'Error'))
ProtocolError: ('Connection aborted.', OSError(0, 'Error')) (Most recent call last)
```

The current docs don't highlight this which seems like a common use case and is currently a pain point.

Based: https://github.com/psf/requests/issues/4323

Most debate around this has been about the lack of multiprocess safety using requests or ensuring you use a single Session per thread/process which doesn't seem feasible when you interact with more than one endpoint or initialize multiple Sessions in different tasks.
 
We switched our execution pool to use `threads` but this doesn't solve the issue.

# Suggestions
- Expect a better setup documentation that clearly states any limitations running in a distributed application like celery.
- How to work with celery and `requests` or `urllib3`
- A Recommended execution pool based on how celery is intended to be used. i.e `prefork` vs `threads` or `gevent` / `eventlet`.



## Package Versions:
```
celery[redis]==4.4.6      # via coherentpath (setup.py), django-celery-beat, flower
certifi==2019.11.28       # via requests, urllib3
cffi==1.13.2              # via bcrypt, cryptography, pynacl
chardet==3.0.4            # via aiohttp, requests
cryptography==3.3.2       # via django-cryptography, paramiko, pyopenssl, requests, urllib3
django-celery-beat==2.0.0  # via coherentpath (setup.py)
django-timezone-field==4.1.1  # via coherentpath (setup.py), django-celery-beat
future==0.18.2            # via celery, django-json-widget, python-jose
humanize==0.5.1           # via flower
idna-ssl==1.1.0           # via aiohttp
idna==2.6                 # via idna-ssl, jsonschema, requests, urllib3, yarl
importlib-metadata==1.6.0  # via kombu, scout-apm
kombu==4.6.11             # via celery
oauthlib==3.1.0           # via django-oauth-toolkit, requests-oauthlib
psycopg2==2.7.3.1         # via coherentpath (setup.py)
pycparser==2.19           # via cffi
pyopenssl==19.1.0         # via requests, urllib3
pyparsing==2.4.7          # via httplib2, packaging
python-crontab==2.5.1     # via django-celery-beat
python-dateutil==2.8.1    # via jsonmodels, pandas, python-crontab
redis==3.5.3              # via celery
requests-toolbelt==0.9.1  # via zeep
requests[security]==2.24.0 
urllib3[secure]==1.25.11  # via pdpyras, requests, scout-apm, selenium
vine==1.3.0               # via amqp, celery
```


I've worked on products with several hundreds of thousands of tasks per minute running using requests and Sessions, API clients, etc. and never had this problem. 

What exactly should we be highlighting in particular in your opinion? The issue you linked if I remember correctly wanted to share the connection pool across the processes which isn't wise anyway. Are you pre-configuring all of your clients before celery forks? 

Not sure how the clients need to be pre-configured ??

But basically using a client like

```py
class APIClient(object):
    
	def __init__(...):
        ...
        self._session = None


    @property
    def session(...):
        if not self._session:
            self._session = requests.Session()
        
        return self._session

```

Also changed the current execution pool from `prefork` to `thread` or `threads`

https://docs.celeryproject.org/en/v4.4.6/userguide/workers.html#remote-control

After noticing a similar ConnectionError testing with `ProcessPoolExecutor`

Python 2.7

```python
=================================== FAILURES ===================================
  ________________________ ***_multi_process_sessions[2] ________________________
  
  process_pool = <concurrent.futures.process.ProcessPoolExecutor object at 0x7f4f119b6a50>
  
      def ***_multi_process_sessions(process_pool):
          jobs = 3
  >       result = list(process_pool.map(run, range(jobs)))
  
  ***s/integration/***_vault.py:101: 
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  .tox/py27/lib/python2.7/site-packages/concurrent/futures/_base.py:641: in result_iterator
      yield fs.pop().result()
  .tox/py27/lib/python2.7/site-packages/concurrent/futures/_base.py:455: in result
      return self.__get_result()
  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
  
  self = <Future at 0x7f4f118b53d0 state=finished raised ConnectionError>
  
      def __get_result(self):
          if self._exception:
              if isinstance(self._exception, types.InstanceType):
                  # The exception is an instance of an old-style class, which
                  # means type(self._exception) returns types.ClassType instead
                  # of the exception's actual class type.
                  exception_type = self._exception.__class__
              else:
                  exception_type = type(self._exception)
  >           raise exception_type, self._exception, self._traceback
  E           ConnectionError: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',))
```

Python 3

```python
 
  Traceback (most recent call last):
    File "/opt/hostedtoolcache/Python/3.6.13/x64/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
      r = call_item.fn(*call_item.args, **call_item.kwargs)
    File "/opt/hostedtoolcache/Python/3.6.13/x64/lib/python3.6/concurrent/futures/process.py", line 153, in _process_chunk
      return [fn(*args) for args in chunk]
    File "/opt/hostedtoolcache/Python/3.6.13/x64/lib/python3.6/concurrent/futures/process.py", line 153, in <listcomp>
      return [fn(*args) for args in chunk]
    File ".../***s/integration/***y", line 94, in run
		...
    File ".../....py", line 43, in __init__
      self.token = self.adapter.get_token(self)
    File ".../adapters/vault.py", line 53, in get_token
      timeout=config.timeout,
    File ".../.tox/py36/lib/python3.6/site-packages/requests/sessions.py", line 578, in post
      return self.request('POST', url, data=data, json=json, **kwargs)
    File ".../.tox/py36/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
      resp = self.send(prep, **send_kwargs)
    File ".../.tox/py36/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
      r = adapter.send(request, **kwargs)
    File ".../.tox/py36/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
      raise ConnectionError(err, request=request)
  requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',))
  """
  
  The above exception was the direct cause of the following exception:
  
  process_pool = <concurrent.futures.process.ProcessPoolExecutor object at 0x7fa6f6e90a20>
  
      def ***_multi_process_sessions(process_pool):
          jobs = 3
  >       result = list(process_pool.map(run, range(jobs)))
```


This is setup using a single process and concurrency of 1 thread per pod with up to 14 replicas

Using:

```
celery -A myapp worker -l info -c 1 -Q critical,high
```
 
```
[ Session(), Session(), ... ] POD 1, thread x
[ Session(), Session(), ... ] POD 2, thread y
[ Session(), Session(), ... ] POD 3, thread z
```


After much digging into this seems the design of the Session object is one that should be re-designed  because this forces you to start managing the Session object minimizing the number of instances created per thread or process.