Repository: StackStorm/st2

Issue:
no_proxy is not respected during index searches
## SUMMARY

This isn't exactly a bug (kinda?) but maybe a solution to consider in how implement this.

Basically no_proxy is ignored during pack index searches.
https://github.com/StackStorm/st2/blob/15dcd3b3642e3a6af0dc82f2b746d98fa46467fa/st2common/st2common/services/packs.py#L74-L84

This makes obvious sense because if you're behind a proxy you want to be able to search the st2 Exchange Index. The issue comes in when you have a local index like I do. The local index is hosted on our controller node and the proxy doesn't have access to this URL due to FW rules. Proxy isn't local to the datacenter either. So because no_proxy isn't respected in an index search my custom index is unreachable.

To further complicate things, I also search the Exchange Index, meaning I have specified both internet based and a local based index. The internet based obviously needs to go though the proxy, but the local based one can't.

The only real way to by pass this at this point is to specify our internal git URL directly and the pack will install since its not searching the index.

Not sure if anyone else has this issue, it might be extremely specific to me with our locked down environment.

### STACKSTORM VERSION

Paste the output of ``st2 --version``: 
st2 3.6.0, on Python 3.6.8

##### OS, environment, install method

3 node HA cluster: 2 nodes, 1 controller all three behind a load balancer





Discussion:
Actually now that I'm thinking about it the whole point of no_proxy is to do exceptions. So I guess I dont see a reason why the index search also couldn't do no_proxy as well.

Resolving patch:
diff --git a/st2common/st2common/services/packs.py b/st2common/st2common/services/packs.py
--- a/st2common/st2common/services/packs.py
+++ b/st2common/st2common/services/packs.py
@@ -20,6 +20,7 @@
 
 import os
 import requests
+from requests.utils import should_bypass_proxies
 import six
 from six.moves import range
 from oslo_config import cfg
@@ -83,6 +84,7 @@ def _fetch_and_compile_index(index_urls, logger=None, proxy_config=None):
     if proxy_config:
         https_proxy = proxy_config.get("https_proxy", None)
         http_proxy = proxy_config.get("http_proxy", None)
+        no_proxy = proxy_config.get("no_proxy", None)
         ca_bundle_path = proxy_config.get("proxy_ca_bundle_path", None)
 
         if https_proxy:
@@ -92,7 +94,17 @@ def _fetch_and_compile_index(index_urls, logger=None, proxy_config=None):
         if http_proxy:
             proxies_dict["http"] = http_proxy
 
+        if no_proxy:
+            proxies_dict["no"] = no_proxy
+
     for index_url in index_urls:
+
+        # TODO:
+        # Bug  in requests doesn't bypass proxies, so we do it ourselves
+        # If this issue ever gets fixed then we can remove it
+        # https://github.com/psf/requests/issues/4871
+        bypass_proxy = should_bypass_proxies(index_url, proxies_dict.get("no"))
+
         index_status = {
             "url": index_url,
             "packs": 0,
@@ -102,7 +114,11 @@ def _fetch_and_compile_index(index_urls, logger=None, proxy_config=None):
         index_json = None
 
         try:
-            request = requests.get(index_url, proxies=proxies_dict, verify=verify)
+            request = requests.get(
+                index_url,
+                proxies=proxies_dict if not bypass_proxy else None,
+                verify=verify if not bypass_proxy else True,
+            )
             request.raise_for_status()
             index_json = request.json()
         except ValueError as e:


Test patch: