Repository: netbox-community/netbox
Pull request: Add support for socks connection to Git backend
### Fixes: #17639

**Note:** will also need to install the python_socks library along with the Dulwich library to use this:
```
pip install dulwich
pip install python_socks
```

To test locally, start a socks proxy in a docker container. Make a HTTP request and observe it using the proxy:

```
$ docker run -d --name socks5 -p 1080:1080 serjs/go-socks5-proxy
$ docker logs -f socks5
```

Specify the local proxy in Netbox configuration.py:
```
HTTP_PROXIES = {
   'http': 'socks5h://localhost:1080',
   'https': 'socks5h://localhost:1080',
}
```

Add a remote GIT datasource and you should see the traffic logs in the terminal. Setup a DataSource in Netbox, I was using the following in the form:

```
Name = GitSync
Type = Git
URL = https://github.com/bogdancordos/netbox_scripts.git
```
The URL just has a selection of scripts so you can see that it actually syncs the data source.  Press the [Sync] button and make sure it actually syncs.

For SOCKs auth scenario it can be setup as follows:

```
docker run -d --name socks5 -p 1080:1080 -e PROXY_USER=aaa -e PROXY_PASSWORD=bbb  serjs/go-socks5-proxy
```

Then in configuration.py:
```
HTTP_PROXIES = {
    'http': 'socks5h://aaa:bbb@localhost:1080',
    'https': 'socks5h://aaa:bbb@localhost:1080',
}
```

Problem statement:
Add SOCKS support to proxy settings for Git remote data source
### NetBox version

v4.1.2

### Feature type

Data model extension

### Proposed functionality

Add socks ('socks5:') support for Git remote data sources.

Currently Dulwich (the git client API library) doesn't support passing in HTTP_PROXIES with 'socks5:', this works fine on other proxy requests like WebHooks.

### Use case

This is needed if using a secure SOCKS proxy and accessing a remote Git data source.

### Database changes

n/a

### External dependencies

Requires the python_socks library if used.

Hints from issue discussion:

Commit message: fix missing comma

Patch to review:
diff --git a/docs/features/synchronized-data.md b/docs/features/synchronized-data.md
index 8c95c87791f..95656f6d13b 100644
--- a/docs/features/synchronized-data.md
+++ b/docs/features/synchronized-data.md
@@ -13,6 +13,9 @@ To enable remote data synchronization, the NetBox administrator first designates
 !!! info
     Data backends which connect to external sources typically require the installation of one or more supporting Python libraries. The Git backend requires the [`dulwich`](https://www.dulwich.io/) package, and the S3 backend requires the [`boto3`](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html) package. These must be installed within NetBox's environment to enable these backends.
 
+!!! info
+    If you are configuring Git and have HTTP_PROXIES set to use the SOCKS protocol, you will also need to install the [`python_socks`](https://pypi.org/project/python-socks/) package.
+
 Each type of remote source has its own configuration parameters. For instance, a git source will ask the user to specify a branch and authentication credentials. Once the source has been created, a synchronization job is run to automatically replicate remote files in the local database.
 
 The following NetBox models can be associated with replicated data files:
diff --git a/netbox/core/data_backends.py b/netbox/core/data_backends.py
index 1b64f5f5c89..72c7bae5386 100644
--- a/netbox/core/data_backends.py
+++ b/netbox/core/data_backends.py
@@ -12,6 +12,7 @@
 
 from netbox.data_backends import DataBackend
 from netbox.utils import register_data_backend
+from utilities.socks import ProxyPoolManager
 from .exceptions import SyncError
 
 __all__ = (
@@ -67,11 +68,14 @@ def init_config(self):
 
         # Initialize backend config
         config = ConfigDict()
+        self.use_socks = False
 
         # Apply HTTP proxy (if configured)
         if settings.HTTP_PROXIES and self.url_scheme in ('http', 'https'):
             if proxy := settings.HTTP_PROXIES.get(self.url_scheme):
                 config.set("http", "proxy", proxy)
+                if urlparse(proxy).scheme in ['socks4', 'socks4a', 'socks4h', 'socks5', 'socks5a', 'socks5h']:
+                    self.use_socks = True
 
         return config
 
@@ -87,6 +91,10 @@ def fetch(self):
             "errstream": porcelain.NoneStream(),
         }
 
+        # check if using socks for proxy - if so need to use custom pool_manager
+        if self.use_socks:
+            clone_args['pool_manager'] = ProxyPoolManager(settings.HTTP_PROXIES.get(self.url_scheme))
+
         if self.url_scheme in ('http', 'https'):
             if self.params.get('username'):
                 clone_args.update(
diff --git a/netbox/utilities/socks.py b/netbox/utilities/socks.py
new file mode 100644
index 00000000000..0b5864e449f
--- /dev/null
+++ b/netbox/utilities/socks.py
@@ -0,0 +1,76 @@
+from urllib.parse import urlparse
+from urllib3 import PoolManager, HTTPConnectionPool, HTTPSConnectionPool
+from urllib3.connection import HTTPConnection, HTTPSConnection
+
+
+# These Proxy Methods are for handling SOCKS connection proxy
+class ProxyHTTPConnection(HTTPConnection):
+    use_rdns = False
+
+    def __init__(self, *args, **kwargs):
+        socks_options = kwargs.pop('_socks_options')
+        self._proxy_url = socks_options['proxy_url']
+        super().__init__(*args, **kwargs)
+
+    def _new_conn(self):
+        from python_socks.sync import Proxy
+        proxy = Proxy.from_url(self._proxy_url, rdns=self.use_rdns)
+        return proxy.connect(
+            dest_host=self.host,
+            dest_port=self.port,
+            timeout=self.timeout
+        )
+
+
+class ProxyHTTPSConnection(ProxyHTTPConnection, HTTPSConnection):
+    pass
+
+
+class RdnsProxyHTTPConnection(ProxyHTTPConnection):
+    use_rdns = True
+
+
+class RdnsProxyHTTPSConnection(ProxyHTTPSConnection):
+    use_rdns = True
+
+
+class ProxyHTTPConnectionPool(HTTPConnectionPool):
+    ConnectionCls = ProxyHTTPConnection
+
+
+class ProxyHTTPSConnectionPool(HTTPSConnectionPool):
+    ConnectionCls = ProxyHTTPSConnection
+
+
+class RdnsProxyHTTPConnectionPool(HTTPConnectionPool):
+    ConnectionCls = RdnsProxyHTTPConnection
+
+
+class RdnsProxyHTTPSConnectionPool(HTTPSConnectionPool):
+    ConnectionCls = RdnsProxyHTTPSConnection
+
+
+class ProxyPoolManager(PoolManager):
+    def __init__(self, proxy_url, timeout=5, num_pools=10, headers=None, **connection_pool_kw):
+        # python_socks uses rdns param to denote remote DNS parsing and
+        # doesn't accept the 'h' or 'a' in the proxy URL
+        cleaned_proxy_url = proxy_url
+        if use_rdns := urlparse(cleaned_proxy_url).scheme in ['socks4h', 'socks4a', 'socks5h', 'socks5a']:
+            cleaned_proxy_url = cleaned_proxy_url.replace('socks5h:', 'socks5:').replace('socks5a:', 'socks5:')
+            cleaned_proxy_url = cleaned_proxy_url.replace('socks4h:', 'socks4:').replace('socks4a:', 'socks4:')
+
+        connection_pool_kw['_socks_options'] = {'proxy_url': cleaned_proxy_url}
+        connection_pool_kw['timeout'] = timeout
+
+        super().__init__(num_pools, headers, **connection_pool_kw)
+
+        if use_rdns:
+            self.pool_classes_by_scheme = {
+                'http': RdnsProxyHTTPConnectionPool,
+                'https': RdnsProxyHTTPSConnectionPool,
+            }
+        else:
+            self.pool_classes_by_scheme = {
+                'http': ProxyHTTPConnectionPool,
+                'https': ProxyHTTPSConnectionPool,
+            }


Code review comments:
diff_hunk: @@ -0,0 +1,76 @@
+from urllib.parse import urlparse
+from urllib3 import PoolManager, HTTPConnectionPool, HTTPSConnectionPool
+from urllib3.connection import HTTPConnection, HTTPSConnection
+
+
+# These Proxy Methods are for handling SOCKS connection proxy
+class ProxyHTTPConnection(HTTPConnection):
+    use_rdns = False
+
+    def __init__(self, *args, **kwargs):
+        socks_options = kwargs.pop('_socks_options')
+        self._proxy_url = socks_options['proxy_url']
+        super().__init__(*args, **kwargs)
+
+    def _new_conn(self):
+        from python_socks.sync import Proxy
+        proxy = Proxy.from_url(self._proxy_url, rdns=self.use_rdns)
+        return proxy.connect(
+            dest_host=self.host,
+            dest_port=self.port,
+            timeout=self.timeout
+        )
+
+
+class ProxyHTTPSConnection(ProxyHTTPConnection, HTTPSConnection):
+    pass
+
+
+class RdnsProxyHTTPConnection(ProxyHTTPConnection):
+    use_rdns = True
+
+
+class RdnsProxyHTTPSConnection(ProxyHTTPSConnection):
+    use_rdns = True
+
+
+class ProxyHTTPConnectionPool(HTTPConnectionPool):
+    ConnectionCls = ProxyHTTPConnection
+
+
+class ProxyHTTPSConnectionPool(HTTPSConnectionPool):
+    ConnectionCls = ProxyHTTPSConnection
+
+
+class RdnsProxyHTTPConnectionPool(HTTPConnectionPool):
+    ConnectionCls = RdnsProxyHTTPConnection
+
+
+class RdnsProxyHTTPSConnectionPool(HTTPSConnectionPool):
+    ConnectionCls = RdnsProxyHTTPSConnection
+
+
+class ProxyPoolManager(PoolManager):
+    def __init__(self, proxy_url, timeout=5, num_pools=10, headers=None, **connection_pool_kw):
+        # python_socks uses rdns param to denote remote DNS parsing and
+        # doesn't accept the 'h' or 'a' in the proxy URL
+        cleaned_proxy_url = proxy_url
+        if use_rdns := urlparse(cleaned_proxy_url).scheme in ['socks4h', 'socks4a', 'socks5h', 'socks5a']:
+            cleaned_proxy_url = cleaned_proxy_url.replace('socks5h:', 'socks5:').replace('socks5a:', 'socks5:')
+            cleaned_proxy_url = cleaned_proxy_url.replace('socks4h:', 'socks4:').replace('socks4a:', 'socks4:')
path: netbox/utilities/socks.py
text: @user1:
Should we just raise an `ImproperlyConfigured` exception if the scheme is unsupported? I suspect it would help reduce troubleshooting effort.

@author:
This is in the background task queue so the error is never propagated to the front-end.  We could potentially put in a check in the settings.py / config code but I don't know if we want to disallow any proxy of another type for any type of proxy?  The old code works fine for SOCKS proxy for everything but Git / dulwich.

diff_hunk: @@ -67,11 +68,14 @@ def init_config(self):
 
         # Initialize backend config
         config = ConfigDict()
+        self.use_socks = False
 
         # Apply HTTP proxy (if configured)
         if settings.HTTP_PROXIES and self.url_scheme in ('http', 'https'):
             if proxy := settings.HTTP_PROXIES.get(self.url_scheme):
                 config.set("http", "proxy", proxy)
+                if urlparse(proxy).scheme in ['socks4', 'socks4a', 'socks4h', 'socks5', 'socks5a', 'socks5h']:
path: netbox/core/data_backends.py
text: @user1:
Feels like we should define these in `constants.py`.

diff_hunk: @@ -0,0 +1,76 @@
+from urllib.parse import urlparse
+from urllib3 import PoolManager, HTTPConnectionPool, HTTPSConnectionPool
+from urllib3.connection import HTTPConnection, HTTPSConnection
+
+
+# These Proxy Methods are for handling SOCKS connection proxy
+class ProxyHTTPConnection(HTTPConnection):
+    use_rdns = False
+
+    def __init__(self, *args, **kwargs):
+        socks_options = kwargs.pop('_socks_options')
+        self._proxy_url = socks_options['proxy_url']
+        super().__init__(*args, **kwargs)
+
+    def _new_conn(self):
+        from python_socks.sync import Proxy
path: netbox/utilities/socks.py
text: @user1:
We should catch & re-raise `ImportError` here to include a helpful error message.

@author:
It's in the code that is executed by the background task queue so the error is never propagated to the front-end.  In the RQ log there is the message:
```
Fetching remote data failed (ModuleNotFoundError): No module named 'python_socks'
```
Which should be similar to dulwich and any other missing library so I think a different error message might be more confusing?

@user1:
We should log an error which hints as to the root cause; something like "Configuring an HTTP proxy using SOCKS requires the python_socks library. Check that it has been installed."

diff_hunk: @@ -0,0 +1,76 @@
+from urllib.parse import urlparse
+from urllib3 import PoolManager, HTTPConnectionPool, HTTPSConnectionPool
+from urllib3.connection import HTTPConnection, HTTPSConnection
+
+
+# These Proxy Methods are for handling SOCKS connection proxy
+class ProxyHTTPConnection(HTTPConnection):
+    use_rdns = False
+
+    def __init__(self, *args, **kwargs):
+        socks_options = kwargs.pop('_socks_options')
+        self._proxy_url = socks_options['proxy_url']
+        super().__init__(*args, **kwargs)
+
+    def _new_conn(self):
+        from python_socks.sync import Proxy
+        proxy = Proxy.from_url(self._proxy_url, rdns=self.use_rdns)
+        return proxy.connect(
+            dest_host=self.host,
+            dest_port=self.port,
+            timeout=self.timeout
+        )
+
+
+class ProxyHTTPSConnection(ProxyHTTPConnection, HTTPSConnection):
+    pass
+
+
+class RdnsProxyHTTPConnection(ProxyHTTPConnection):
+    use_rdns = True
+
+
+class RdnsProxyHTTPSConnection(ProxyHTTPSConnection):
+    use_rdns = True
+
+
+class ProxyHTTPConnectionPool(HTTPConnectionPool):
+    ConnectionCls = ProxyHTTPConnection
+
+
+class ProxyHTTPSConnectionPool(HTTPSConnectionPool):
+    ConnectionCls = ProxyHTTPSConnection
+
+
+class RdnsProxyHTTPConnectionPool(HTTPConnectionPool):
+    ConnectionCls = RdnsProxyHTTPConnection
+
+
+class RdnsProxyHTTPSConnectionPool(HTTPSConnectionPool):
+    ConnectionCls = RdnsProxyHTTPSConnection
+
+
+class ProxyPoolManager(PoolManager):
+    def __init__(self, proxy_url, timeout=5, num_pools=10, headers=None, **connection_pool_kw):
+        # python_socks uses rdns param to denote remote DNS parsing and
+        # doesn't accept the 'h' or 'a' in the proxy URL
+        cleaned_proxy_url = proxy_url
path: netbox/utilities/socks.py
text: @user1:
If we keep this approach, we can just refer to this variable as `proxy_url` consistently.