Repository: oceanprotocol/ocean.py (language: python)

Issue:
"Connection refused" in pay_for_compute_services() while doing c2d README
**Describe the bug**

I was not able to complete the c2d README. The call to `get_c2d_environments()` returned `"Connection refused"`.

**To reproduce**

Go through the steps of [c2d README](https://github.com/oceanprotocol/ocean.py/blob/v4main/READMEs/c2d-flow.md) until the failure is hit. 
- Exception to setup: I was using a `git clone` installation of ocean_lib, rather than a `pip install` version, because the latter currently fails at the very first step (reported in bug [#836](https://github.com/oceanprotocol/ocean.py/issues/836)).

When you get to the step:
```python
datasets, algorithm = ocean.assets.pay_for_compute_service(
    ...
    compute_environment=environments[0]["id"], 
    ...
)
```

It will fail with:
```
KeyError: 0
````

It turns out that it doesn't even make it into `pay_for_compute_service()`. Rather, it can't see item `[0]` in `environments`. That That variable was set a few lines above:
```
environments = ocean.compute.get_c2d_environments(compute_service.service_endpoint)
```

This call _did_ return a value:
```
>>> environments
{'context': None, 'error': "HTTPConnectionPool(host='172.15.0.13', port=31000): Max retries exceeded with url: /api/v1/operator/environments?chainId=8996 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fab4b45a430>: Failed to establish a new connection: [Errno 111] Connection refused'))"}
```

Note that it can't be keyed with `[0]`. And there was an error. Therefore any usage of `environments` fails.


**Barge output**

Here's the output in barge when `get_c2d_environments()` is called:
```console
kindcluster_1      | 172.15.0.12 - - [24/May/2022:04:52:56 +0000] "HEAD /kubernetes-ready HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
kindcluster_1      | 172.15.0.12 - - [24/May/2022:04:53:01 +0000] "HEAD /kubernetes-ready HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
provider_1         | 2022-05-24 04:53:03 ac290fa2c43a ocean_provider.run[27] INFO incoming request = http, GET, 172.15.0.1, /?
provider_1         | 2022-05-24 04:53:03 ac290fa2c43a ocean_provider.run[27] INFO root endpoint called
provider_1         | 2022-05-24 04:53:03 ac290fa2c43a ocean_provider.run[27] INFO root endpoint response = <Response 778 bytes [200 OK]>
provider_1         | 2022-05-24 04:53:03 ac290fa2c43a ocean_provider.run[26] INFO incoming request = http, GET, 172.15.0.1, /?
provider_1         | 2022-05-24 04:53:03 ac290fa2c43a ocean_provider.run[26] INFO root endpoint called
provider_1         | 2022-05-24 04:53:03 ac290fa2c43a ocean_provider.run[26] INFO root endpoint response = <Response 778 bytes [200 OK]>
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a ocean_provider.run[23] INFO incoming request = http, GET, 172.15.0.1, /api/services/computeEnvironments?
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a config[23] INFO Config: loading config file /ocean-provider/config.ini
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a config[23] INFO Config: setting environ network = http://172.15.0.3:8545
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a config[23] INFO Config: setting environ address.file = /ocean-contracts/artifacts/address.json
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a config[23] INFO Config: setting environ aquarius.url = http://172.15.0.5:5000
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a config[23] INFO Config: setting environ operator_service.url = http://172.15.0.13:31000/
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a config[23] INFO Config: setting environ allow_non_public_ip = False
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a urllib3.connectionpool[23] WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fab4b4c4370>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/operator/environments?chainId=8996
provider_1         | 2022-05-24 04:53:04 ac290fa2c43a ocean_provider.run[23] ERROR error: HTTPConnectionPool(host='172.15.0.13', port=31000): Max retries exceeded with url: /api/v1/operator/environments?chainId=8996 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fab4b4a3190>: Failed to establish a new connection: [Errno 111] Connection refused')), payload: b''
provider_1         | Traceback (most recent call last):
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 174, in _new_conn
provider_1         |     conn = connection.create_connection(
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/util/connection.py", line 95, in create_connection
provider_1         |     raise err
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/util/connection.py", line 85, in create_connection
provider_1         |     sock.connect(sa)
provider_1         | ConnectionRefusedError: [Errno 111] Connection refused
provider_1         | 
provider_1         | During handling of the above exception, another exception occurred:
provider_1         | 
provider_1         | Traceback (most recent call last):
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
provider_1         |     httplib_response = self._make_request(
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
provider_1         |     conn.request(method, url, **httplib_request_kw)
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 239, in request
provider_1         |     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
provider_1         |   File "/usr/lib/python3.8/http/client.py", line 1234, in request
provider_1         |     self._send_request(method, url, body, headers, encode_chunked)
provider_1         |   File "/usr/lib/python3.8/http/client.py", line 1280, in _send_request
provider_1         |     self.endheaders(body, encode_chunked=encode_chunked)
provider_1         |   File "/usr/lib/python3.8/http/client.py", line 1229, in endheaders
provider_1         |     self._send_output(message_body, encode_chunked=encode_chunked)
provider_1         |   File "/usr/lib/python3.8/http/client.py", line 1008, in _send_output
provider_1         |     self.send(msg)
provider_1         |   File "/usr/lib/python3.8/http/client.py", line 948, in send
provider_1         |     self.connect()
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 205, in connect
provider_1         |     conn = self._new_conn()
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 186, in _new_conn
provider_1         |     raise NewConnectionError(
provider_1         | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fab4b4a3190>: Failed to establish a new connection: [Errno 111] Connection refused
provider_1         | 
provider_1         | During handling of the above exception, another exception occurred:
provider_1         | 
provider_1         | Traceback (most recent call last):
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 440, in send
provider_1         |     resp = conn.urlopen(
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 813, in urlopen
provider_1         |     return self.urlopen(
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 785, in urlopen
provider_1         |     retries = retries.increment(
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 592, in increment
provider_1         |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
provider_1         | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.15.0.13', port=31000): Max retries exceeded with url: /api/v1/operator/environments?chainId=8996 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fab4b4a3190>: Failed to establish a new connection: [Errno 111] Connection refused'))
provider_1         | 
provider_1         | During handling of the above exception, another exception occurred:
provider_1         | 
provider_1         | Traceback (most recent call last):
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request
provider_1         |     rv = self.dispatch_request()
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request
provider_1         |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
provider_1         |   File "/ocean-provider/ocean_provider/routes/compute.py", line 69, in decorated_function
provider_1         |     return f(*args, **kws)
provider_1         |   File "/ocean-provider/ocean_provider/routes/compute.py", line 565, in computeEnvironments
provider_1         |     response = jsonify(get_c2d_environments())
provider_1         |   File "/ocean-provider/ocean_provider/utils/compute_environments.py", line 23, in get_c2d_environments
provider_1         |     response = requests_session.get(
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 542, in get
provider_1         |     return self.request('GET', url, **kwargs)
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 529, in request
provider_1         |     resp = self.send(prep, **send_kwargs)
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 645, in send
provider_1         |     r = adapter.send(request, **kwargs)
provider_1         |   File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 519, in send
provider_1         |     raise ConnectionError(e, request=request)
provider_1         | requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.15.0.13', port=31000): Max retries exceeded with url: /api/v1/operator/environments?chainId=8996 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fab4b4a3190>: Failed to establish a new connection: [Errno 111] Connection refused'))
aquarius_1         | 2022-05-24 04:53:04 052e57d7b569 root[19] INFO Stable connection to ES.
ganache_1          | eth_blockNumber
kindcluster_1      | 172.15.0.12 - - [24/May/2022:04:53:06 +0000] "HEAD /kubernetes-ready HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
kindcluster_1      | 172.15.0.12 - - [24/May/2022:04:53:11 +0000] "HEAD /kubernetes-ready HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
aquarius_1         | 2022-05-24 04:53:14 052e57d7b569 root[19] INFO Stable connection to ES.
ganache_1          | eth_blockNumber
kindcluster_1      | 172.15.0.12 - - [24/May/2022:04:53:16 +0000] "HEAD /kubernetes-ready HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
kindcluster_1      | 172.15.0.12 - - [24/May/2022:04:53:22 +0000] "HEAD /kubernetes-ready HTTP/1.1" 404 0 "-" "curl/7.68.0" "-"
aquarius_1         | 2022-05-24 04:53:25 052e57d7b569 root[19] INFO Stable connection to ES.
```

----

After I'd closed all of the above down, I cleared and re-ran barge. Here's the output. I notice there are some errors.
[log.txt](https://github.com/oceanprotocol/ocean.py/files/8759978/log.txt)



Discussion / hints:
@trentmc did you wait for the c2d to boot up? It sometimes takes a lot of time but will lead to failure in tests. I have encountered this error before and 100% of the time it was due to the barge c2d not being up and running.

You specifically have to wait for the barge logs to say something like "C2d is up and running, have fun!". Do you have any logs from docker indicating c2d is up? As an empirical observation, I've seen that kindcluster logs stop once c2d is up, so I'm pretty much convinced that the c2d container is not up.

On the other hand, let's discuss how we can mitigate this. Should we add some instructions to the readme that make the user aware they have to wait longer sometimes? Or that if the c2d barge deployment is failing for some reason or the other, then they should restart the process (prune & rerun barge start_ocean)?
Yes, I waited for c2d to boot up. I gave it lots of time. >10 min.

And, I tried a second time, end-to-end. Same results.

After I'd closed all of the above down, I cleared and re-ran barge. Here's the output. I notice there are some errors. 

[log.txt](https://github.com/oceanprotocol/ocean.py/files/8759978/log.txt)

I'd help to debug right now but I have to work on a talk that I'm giving in 1.5h.
@calina-c There is a script in github-actions we need to add to the Compute-related READMEs.

Fix patch:
diff --git a/READMEs/consume-flow.md b/READMEs/consume-flow.md
index d97ec416a..516f01443 100644
--- a/READMEs/consume-flow.md
+++ b/READMEs/consume-flow.md
@@ -97,8 +97,8 @@ print(f"file_path = '{file_path}'")  # e.g. datafile.0xAf07...
 
 In console:
 
-```
-#verify that the file is downloaded
+```console
+# verify that the file is downloaded
 cd my_project/datafile.did:op:0xAf07...
 ls branin.arff
 ```
@@ -107,7 +107,7 @@ Congrats to Bob for buying and consuming a data asset!
 
 _Note_. The file is in ARFF format, used by some AI/ML tools. In this case there are two input variables (x0, x1) and one output.
 
-```
+```console
 % 1. Title: Branin Function
 % 3. Number of instances: 225
 % 6. Number of attributes: 2
diff --git a/READMEs/data-nfts-and-datatokens-flow.md b/READMEs/data-nfts-and-datatokens-flow.md
index 2704d23eb..d0d78d9c1 100644
--- a/READMEs/data-nfts-and-datatokens-flow.md
+++ b/READMEs/data-nfts-and-datatokens-flow.md
@@ -31,6 +31,15 @@ docker system prune -a --volumes
 ./start_ocean.sh --with-c2d
 ```
 
+When running barge with c2d enabled, use the following bash script to wait until
+the `artifacts` directory and `address.json` file are successfully created and c2d is fully deployed:
+```console
+for i in $(seq 1 50); do
+    sleep 5
+    [ -f "$HOME/.ocean/ocean-contracts/artifacts/ready" -a -f "$HOME/.ocean/ocean/c2d/ready" ] && break
+    done
+```
+
 ## Install the library from v4 sources
 
 In a new console:


Test patch:
diff --git a/.github/workflows/pytest.yml b/.github/workflows/pytest.yml
index 1bcda8ad0..1d88e1063 100644
--- a/.github/workflows/pytest.yml
+++ b/.github/workflows/pytest.yml
@@ -41,7 +41,7 @@ jobs:
           pip install -r requirements_dev.txt
       - name: Test with pytest
         run: |
-          mkcodes --github --output tests/generated-readmes/test_{name}.py READMEs
+          mkcodes --github --output tests/generated-readmes/test_{name}.{ext} READMEs
           coverage run --source ocean_lib -m py.test -m "not slow"
           coverage report
           coverage xml
diff --git a/tests/test_readmes.py b/tests/test_readmes.py
index a0b82f1f1..8bda0d1b8 100644
--- a/tests/test_readmes.py
+++ b/tests/test_readmes.py
@@ -13,7 +13,7 @@
 @pytest.mark.parametrize("script", scripts)
 def test_script_execution(script, monkeypatch):
     # README generation command:
-    # mkcodes --github --output tests/generated-readmes/test_{name}.py READMEs
+    # mkcodes --github --output tests/generated-readmes/test_{name}.{ext} READMEs
 
     if (
         "developers" in script.name