Repository: ethereum/web3.py

Issue:
gethdev AutoProvider does not consider the WEB3_PROVIDER_URI environment variable
* Version: 5.13.0
* Python: 3.8.5
* OS: Linux (Ubuntu 20.04.1 LTS)

### What was wrong?

When using `web3.auto.gethdev`, the `WEB3_PROVIDER_URI` environment variable does not appear to be honored.

```
$ export WEB3_PROVIDER_URI=file:///path/to/node/geth.ipc

$ python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from web3.auto.gethdev import w3
>>> w3.eth.filter({'fromBlock': 'latest'})

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/<path>/.local/lib/python3.8/site-packages/web3/eth.py", line 491, in filter
    _filter_id = self.web3.manager.request_blocking(
  File "/<path>/.local/lib/python3.8/site-packages/web3/manager.py", line 154, in request_blocking
    response = self._make_request(method, params)
  File "/<path>/.local/lib/python3.8/site-packages/web3/manager.py", line 133, in _make_request
    return request_func(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 74, in apply_formatters
    response = make_request(method, formatted_params)
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/gas_price_strategy.py", line 34, in middleware
    return make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 74, in apply_formatters
    response = make_request(method, formatted_params)
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/attrdict.py", line 33, in middleware
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 74, in apply_formatters
    response = make_request(method, formatted_params)
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/normalize_errors.py", line 25, in middleware
    result = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 76, in apply_formatters
    response = make_request(method, params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 74, in apply_formatters
    response = make_request(method, formatted_params)
  File "cytoolz/functoolz.pyx", line 250, in cytoolz.functoolz.curry.__call__
  File "/<path>/.local/lib/python3.8/site-packages/web3/middleware/formatting.py", line 76, in apply_formatters
    response = make_request(method, params)
  File "/<path>/.local/lib/python3.8/site-packages/web3/providers/ipc.py", line 234, in make_request
    with self._lock, self._socket as sock:
  File "/<path>/.local/lib/python3.8/site-packages/web3/providers/ipc.py", line 55, in __enter__
    raise FileNotFoundError("cannot connect to IPC socket at path: %r" % self.ipc_path)
FileNotFoundError: cannot connect to IPC socket at path: None
```

### How can it be fixed?

Apply this environment variable to the gethdev autoprovider.


Discussion:
Thanks for the report! I'll take a look.
I think the issue is not related to the env vars.
The real problem is that get_default_ipc_path() is returning None because the user doesn't have the geth.ipc in the expected location.
I would be glad to contribute with a fix for this, but I'll need some guidance on what the way to go is.
Would it be the best to raise an exception with an adequate message when get_default_ipc_path() returns None?
@AngelGris A PR would be great! I think the reported problem is an environment variable issue, although it's you're right in seeing that if the path doesn't exist, `None` gets returned. I think it would be good to add an error message instead of just silently not connecting, although I'm not sure what the thinking was originally to return None instead of an error. 

I'd approach the env var fix by adding a check at the top of `get_dev_ipc_path` [here](https://github.com/ethereum/web3.py/blob/master/web3/providers/ipc.py#L159) for the environment variable, and then if the path doesn't exist, we can do the `sys.platform` check. So something like:

```
if os.environ.get('WEB3_PROVIDER_URI'):
    if os.path.exists(ipc_path):
        return ipc_path
elif sys.platform == 'dawin':
    ...
```
And then if you're up for it, I'd open a second PR to raise an error if `os.path.exists` returns `False`, since it's technically a breaking change. 

That way, if they're in separate PRs, we can get the environment variable fix in and then talk about whether we want to pull the error change into v5 or start on a v6 branch. The breaking change would only apply to the gethdev auto provider, so I think it would probably be fine to add the error change into v5 but someone else may have a stronger opinion. I also think if we're raising an error when one provider fails to connect, we should probably do it with all of them, but I'd definitely want to wait until v6 to start mucking around with breaking changes in the other providers.

Let me know if anything was unclear or if you have any other questions!
From my understanding, web3.auto.gethdev would call [get_default_ipc_path](https://github.com/ethereum/web3.py/blob/06803e26903e4f8d94e4a6db7f98a56988664f6a/web3/providers/ipc.py#L82) because of the [IPCProvider](https://github.com/ethereum/web3.py/blob/06803e26903e4f8d94e4a6db7f98a56988664f6a/web3/auto/gethdev.py#L12) call which, in its turn, would call [get_default_ipc_path](https://github.com/ethereum/web3.py/blob/06803e26903e4f8d94e4a6db7f98a56988664f6a/web3/providers/ipc.py#L215), if [get_dev_ipc_path](https://github.com/ethereum/web3.py/blob/06803e26903e4f8d94e4a6db7f98a56988664f6a/web3/providers/ipc.py#L159) returns None because of non-existing ipc file, that returns a [string](https://github.com/ethereum/web3.py/blob/master/web3/providers/ipc.py#L82) object that represents the absolute path for the `jsonrpc.ipc` or the `geth.ipc` file on the machine. Since the method doesn't handle the case of non existing file, it will return a None type object. Therefore, you need to check if `geth.ipc` or `jsonrpc.ipc` exist first.

```sh
$ locate geth.ipc
/home/<USER>/.ethereum/geth.ipc

$ locate jsonrpc.ipc
/home/<USER>/.local/share/io.parity.ethereum/jsonrpc.ipc
/home/<USER>/.local/share/trinity/mainnet/ipcs-eth1/jsonrpc.ipc  # <--- running a trinity client will generate this file i guess?
```

Additionally, I think @AngelGris is right by his comment that the problem is not related to the env variable, but rather to a non-existing ipc file.
@Morgon, to help you with your situation, follow the steps below:

1. Begin by starting your ethereum node such as [Geth](https://geth.ethereum.org/docs/install-and-build/installing-geth#install-on-ubuntu-via-ppas) server by running the following cmd:

```sh
$ geth console # this will generate a geth.ipc file when the node starts up
```

2. Open another terminal, and set the env variable:

```sh
$ export WEB3_PROVIDER_URI=/home/<USER>/.ethereum/geth.ipc  # <--- replace <USER> by your actual user.
```

3.  And then proceed with the python statements.

```python
>>> from web3.auto.gethdev import w3
>>> w3.eth.filter({'fromBlock': 'latest'})
<web3._utils.filters.LogFilter object at 0x7fd9f2bd3850>
```
Just to be clear on the conversation, you are asking whether I've provided an incorrect path? Unfortunately not - I'm using the same path to the IPC socket as I do to attach a geth instance to my currently-running environment. 

`WEB3_PROVIDER_URI` is supposed to override the default without regard to the existence of that default, is it not? Or are you suggesting that not having the IPC in the default path is what is causing the bug?
I'll try using a direct path instead of a file URI, per your example:

```bash
$ pwd
/home/bsc

$ ls /home/bsc/bsc/node/geth.ipc
/home/bsc/bsc/node/geth.ipc
$ export WEB3_PROVIDER_URI=/home/bsc/bsc/node/geth.ipc

$ bsc/geth attach /home/bsc/bsc/node/geth.ipc  # To show this is a valid socket
Welcome to the Geth JavaScript console!

instance: Geth/v1.1.0-beta-032970b2/linux-amd64/go1.15.5
at block: 8183235 (Thu Jun 10 2021 14:59:25 GMT-0400 (EDT))
[snip]

```
```py
$ python3
Python 3.9.5 (default, May 11 2021, 08:20:37)
[snip]
>>> from web3.auto.gethdev import w3
>>> w3.eth.filter({'fromBlock': 'latest'})
Traceback (most recent call last):
[snip]
FileNotFoundError: cannot connect to IPC socket at path: None
```

This is after updating to web3 5.20.0
As another test, I `touch`ed `~/.ethereum/geth.ipc` so that we wouldn't get a file not found error. Executing through gethdev showed 'Connection Refused'

This confirms my claim that `WEB3_PROVIDER_URI` is not being honored at all and gethdev only considers the default path.

Resolving patch:
diff --git a/web3/providers/ipc.py b/web3/providers/ipc.py
--- a/web3/providers/ipc.py
+++ b/web3/providers/ipc.py
@@ -157,7 +157,11 @@ def get_default_ipc_path() -> str:  # type: ignore
 
 # type ignored b/c missing return statement is by design here
 def get_dev_ipc_path() -> str:  # type: ignore
-    if sys.platform == 'darwin':
+    if os.environ.get('WEB3_PROVIDER_URI', ''):
+        ipc_path = os.environ.get('WEB3_PROVIDER_URI')
+        if os.path.exists(ipc_path):
+            return ipc_path
+    elif sys.platform == 'darwin':
         tmpdir = os.environ.get('TMPDIR', '')
         ipc_path = os.path.expanduser(os.path.join(
             tmpdir,


Test patch:
diff --git a/tests/core/providers/test_auto_provider.py b/tests/core/providers/test_auto_provider.py
--- a/tests/core/providers/test_auto_provider.py
+++ b/tests/core/providers/test_auto_provider.py
@@ -17,6 +17,9 @@
 from web3.providers.auto import (
     load_provider_from_environment,
 )
+from web3.providers.ipc import (
+    get_dev_ipc_path,
+)
 
 # Ugly hack to import infura now that API KEY is required
 os.environ['WEB3_INFURA_API_KEY'] = 'test'
@@ -51,6 +54,13 @@ def test_load_provider_from_env(monkeypatch, uri, expected_type, expected_attrs)
         assert getattr(provider, attr) == val
 
 
+def test_get_dev_ipc_path(monkeypatch, tmp_path):
+    uri = str(tmp_path)
+    monkeypatch.setenv('WEB3_PROVIDER_URI', uri)
+    path = get_dev_ipc_path()
+    assert path == uri
+
+
 @pytest.mark.parametrize('environ_name', ['WEB3_INFURA_API_KEY', 'WEB3_INFURA_PROJECT_ID'])
 def test_web3_auto_infura_empty_key(monkeypatch, caplog, environ_name):
     monkeypatch.setenv('WEB3_INFURA_SCHEME', 'https')