Repository: googleapis/google-api-python-client

Issue:
discovery.build( ) raises ResourceWarning: unclosed <ssl.SSLSocket fd=13, 
#### Environment details

  - OS:
Linux hu3br 4.18.16-300.fc29.x86_64 #1 SMP Sat Oct 20 23:24:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  - Python version:
Python 3.6.7
  - pip version:
pip 19.0.1 from /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/pip (python 3.6)

  - `google-api-python-client` version:

Name: google-api-python-client
Version: 1.7.8
Summary: Google API Client Library for Python
Home-page: http://github.com/google/google-api-python-client/
Author: Google Inc.
Author-email: None
License: Apache 2.0
Location: /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages
Requires: google-auth-httplib2, six, uritemplate, google-auth, httplib2
Required-by: 

#### Steps to reproduce

Instantiate a dataflow, v1b3 API with googleapis.discovery
```
> /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(117)__init__()
-> self.api_service = build(API_COMPONENT, API_VERSION, cache_discovery=False).projects().locations()
(Pdb) n
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/google/auth/_default.py:66: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK. We recommend that most server applications use service accounts instead. If your application continues to use end user credentials from Cloud SDK, you might receive a "quota exceeded" or "API not enabled" error. For more information about service accounts, see https://cloud.google.com/docs/authentication/.
  warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
/home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=13, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('10.43.33.41', 48878), raddr=('74.125.133.95', 443)>
  return wrapped(*args, **kwargs)
> /home/ramonm/git/bigtable-backups/.venv/lib/python3.6/site-packages/bigtable_backups/dataflow.py(120)__init__()

```

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!



Discussion:
We're also seeing this while using other API endpoints (other than `dataflow`) when `-Wall` is in effect for `python`.
@berendo Could you elaborate on what `-Wall` is?
> Could you elaborate on what -Wall is?

Sure, it’s running Python with `python -Wall`. See the `-W` option [here](https://docs.python.org/3/using/cmdline.html).
@berendo Thank you!

This is straightforward to reproduce:

```py
from googleapiclient import discovery

client = discovery.build("dataflow", "v1b3")
```

```
$ python3 -Wall make-client.py 
~/.local/lib/python3.6/site-packages/googleapiclient/_helpers.py:130: ResourceWarning: unclosed <ssl.SSLSocket fd=3, family=AddressFamily.AF_INET6, type=2049, proto=6, laddr=('2620:15c:10f:202:e603:43de:a156:cc2', 33710, 0, 0), raddr=('2607:f8b0:400a:809::200a', 443, 0, 0)>
```
The request to fetch the discovery doc results in the ResourceWarning.
```python
from googleapiclient import discovery
import requests
import httplib2

# Doesn't raise socket resource warning
dataflow_discovery_doc = requests.get("https://dataflow.googleapis.com/$discovery/rest?version=v1b3").json()
client = discovery.build_from_document(dataflow_discovery_doc)

# Raises socket resource warning
client = discovery.build("dataflow", "v1b3")

# Function to retrieve public discovery doc
doc = discovery._retrieve_discovery_doc("https://www.googleapis.com/discovery/v1/apis", httplib2.Http(), cache_discovery=False)
```


`httplib2` seems to leave sockets open when requests are successful.
```python
import httplib2

http = httplib2.Http()
http.request("http://example.com")
```

I'm not sure if there's a good way to ask that the connection be closed in httplib2. I filed https://github.com/httplib2/httplib2/issues/148 to ask the maintainers.

From similar discussions on requests (https://github.com/psf/requests/issues/3912) it may be okay to let this be.
Since https://github.com/httplib2/httplib2/issues/148 is now closed, any plans to add `.close()` to discovery? 
@wintermelons Thank you for the ping! It looks like httplib2 hasn't yet made a [release](https://github.com/httplib2/httplib2/releases) with the change.
FYI: httplib2 has made a [new release](https://github.com/httplib2/httplib2/releases/tag/v0.15.0) :slightly_smiling_face: 
Bumping this up for the code sanity, any updates?
This is causing a lot of disruption for us - it causes containers running Airflow tasks against GCP to consistently run out of available sockets.
Hello folks,

Apologies for leaving this unattended for so long.

f this is actively causing pain you can manually call the [`httplib2.http.close()`](https://github.com/httplib2/httplib2/commit/3679f7b99561843ef6add04c725f5d773d3dd6ac) method when you are done with the client.

```py
from googleapiclient import discovery

client = discovery.build("drive", "v3")

client._http.http.close()
```

Ideally the library should clean up `httplib2.http` it creates.  httplib2 instances could be shared across clients so I think it will require some care to get the logic correct.
@ramonmedeiros Can you please elaborate why did you close this issue?

Resolving patch:
diff --git a/googleapiclient/discovery.py b/googleapiclient/discovery.py
--- a/googleapiclient/discovery.py
+++ b/googleapiclient/discovery.py
@@ -261,6 +261,8 @@ def build(
     else:
         discovery_http = http
 
+    service = None
+
     for discovery_url in _discovery_service_uri_options(discoveryServiceUrl, version):
         requested_url = uritemplate.expand(discovery_url, params)
 
@@ -273,7 +275,7 @@ def build(
                 developerKey,
                 num_retries=num_retries,
             )
-            return build_from_document(
+            service = build_from_document(
                 content,
                 base=discovery_url,
                 http=http,
@@ -285,13 +287,22 @@ def build(
                 adc_cert_path=adc_cert_path,
                 adc_key_path=adc_key_path,
             )
+            break  # exit if a service was created
         except HttpError as e:
             if e.resp.status == http_client.NOT_FOUND:
                 continue
             else:
                 raise e
 
-    raise UnknownApiNameOrVersion("name: %s  version: %s" % (serviceName, version))
+    # If discovery_http was created by this function, we are done with it
+    # and can safely close it
+    if http is None:
+        discovery_http.close()
+
+    if service is None:
+        raise UnknownApiNameOrVersion("name: %s  version: %s" % (serviceName, version))
+    else:
+        return service
 
 
 def _discovery_service_uri_options(discoveryServiceUrl, version):
@@ -1309,6 +1320,20 @@ def __setstate__(self, state):
         self._dynamic_attrs = []
         self._set_service_methods()
 
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc, exc_tb):
+        self.close()
+
+    def close(self):
+        """Close httplib2 connections."""
+        # httplib2 leaves sockets open by default.
+        # Cleanup using the `close` method.
+        # https://github.com/httplib2/httplib2/issues/148
+        self._http.http.close()
+
     def _set_service_methods(self):
         self._add_basic_methods(self._resourceDesc, self._rootDesc, self._schema)
         self._add_nested_resources(self._resourceDesc, self._rootDesc, self._schema)
diff --git a/googleapiclient/http.py b/googleapiclient/http.py
--- a/googleapiclient/http.py
+++ b/googleapiclient/http.py
@@ -1720,6 +1720,8 @@ def request(
         self.headers = headers
         return httplib2.Response(self.response_headers), self.data
 
+    def close(self):
+        return None
 
 class HttpMockSequence(object):
     """Mock of httplib2.Http


Test patch:
diff --git a/tests/test_discovery.py b/tests/test_discovery.py
--- a/tests/test_discovery.py
+++ b/tests/test_discovery.py
@@ -437,6 +437,13 @@ def test_ResourceMethodParameters_zoo_animals_patch(self):
         self.assertEqual(parameters.enum_params, {})
 
 
+class Discovery(unittest.TestCase):
+    def test_discovery_http_is_closed(self):
+        http = HttpMock(datafile("malformed.json"), {"status": "200"})
+        service = build("plus", "v1", credentials=mock.sentinel.credentials)
+        http.close.assert_called_once()
+
+
 class DiscoveryErrors(unittest.TestCase):
     def test_tests_should_be_run_with_strict_positional_enforcement(self):
         try:
@@ -572,6 +579,25 @@ def test_building_with_developer_key_skips_adc(self):
         # application default credentials were used.
         self.assertNotIsInstance(plus._http, google_auth_httplib2.AuthorizedHttp)
 
+    def test_building_with_context_manager(self):
+        discovery = read_datafile("plus.json")
+        with mock.patch("httplib2.Http") as http:
+            with build_from_document(discovery, base="https://www.googleapis.com/", credentials=self.MOCK_CREDENTIALS) as plus:
+                self.assertIsNotNone(plus)
+                self.assertTrue(hasattr(plus, "activities"))
+            plus._http.http.close.assert_called_once()
+
+    def test_resource_close(self):
+        discovery = read_datafile("plus.json")
+        with mock.patch("httplib2.Http") as http:
+            plus = build_from_document(
+                discovery,
+                base="https://www.googleapis.com/",
+                credentials=self.MOCK_CREDENTIALS,
+            )
+            plus.close()
+            plus._http.http.close.assert_called_once()
+
     def test_api_endpoint_override_from_client_options(self):
         discovery = read_datafile("plus.json")
         api_endpoint = "https://foo.googleapis.com/"