Repository: Azure/WALinuxAgent
Pull request #2205 (2021-03-12 18:17:29)

logger.warn when http fetch with exception happened.
<!-- DO NOT DELETE THIS TEMPLATE -->

## Description

logger.warn when http fetch with exception happened.

---

### PR information
- [ ] The title of the PR is clear and informative.
- [ ] There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
- [ ] If applicable, the PR references the bug/issue that it fixes in the description.
- [ ] New Unit tests were added for the changes made

### Quality of Code and Contribution Guidelines
- [ ] I have read the [contribution guidelines](https://github.com/Azure/WALinuxAgent/blob/master/.github/CONTRIBUTING.md).

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205?src=pr&el=h1) Report
> Merging [#2205](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205?src=pr&el=desc) (f5cdf90) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/49dcf6d5290a2990efb53336a01ad5bbcd5a8827?el=desc) (49dcf6d) will **increase** coverage by `0.11%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/graphs/tree.svg?width=650&height=150&src=pr&token=ae90E2KYCw)](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #2205      +/-   ##
===========================================
+ Coverage    70.33%   70.44%   +0.11%     
===========================================
  Files           96       96              
  Lines        13395    13538     +143     
  Branches      1896     1910      +14     
===========================================
+ Hits          9421     9537     +116     
- Misses        3563     3591      +28     
+ Partials       411      410       -1     
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/protocol/wire.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC93aXJlLnB5) | `78.30% <ø> (+0.59%)` | :arrow_up: |
| [azurelinuxagent/common/persist\_firewall\_rules.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wZXJzaXN0X2ZpcmV3YWxsX3J1bGVzLnB5) | `75.18% <0.00%> (-8.99%)` | :arrow_down: |
| [azurelinuxagent/common/osutil/default.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9vc3V0aWwvZGVmYXVsdC5weQ==) | `59.39% <0.00%> (-0.28%)` | :arrow_down: |
| [azurelinuxagent/common/cgroupconfigurator.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9jZ3JvdXBjb25maWd1cmF0b3IucHk=) | `71.01% <0.00%> (-0.25%)` | :arrow_down: |
| [azurelinuxagent/ga/exthandlers.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL2V4dGhhbmRsZXJzLnB5) | `86.72% <0.00%> (-0.11%)` | :arrow_down: |
| [azurelinuxagent/ga/update.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL3VwZGF0ZS5weQ==) | `88.37% <0.00%> (-0.08%)` | :arrow_down: |
| [azurelinuxagent/ga/env.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL2Vudi5weQ==) | `63.57% <0.00%> (ø)` | |
| [azurelinuxagent/common/version.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi92ZXJzaW9uLnB5) | `72.28% <0.00%> (ø)` | |
| [azurelinuxagent/ga/send\_telemetry\_events.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL3NlbmRfdGVsZW1ldHJ5X2V2ZW50cy5weQ==) | `95.12% <0.00%> (ø)` | |
| [azurelinuxagent/common/agent\_supported\_feature.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9hZ2VudF9zdXBwb3J0ZWRfZmVhdHVyZS5weQ==) | `100.00% <0.00%> (ø)` | |
| ... and [9 more](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205?src=pr&el=footer). Last update [49dcf6d...f5cdf90](https://codecov.io/gh/Azure/WALinuxAgent/pull/2205?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index 726a3324..3c1f7cc7 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -751,7 +751,13 @@ class WireClient(object):
                     host_plugin.report_fetch_health(uri, source='WireClient')
 
         except (HttpError, ProtocolError, IOError) as error:
-            logger.verbose("Fetch failed from [{0}]: {1}", uri, error)
+            msg = "Fetch failed: {0}".format(error)
+            logger.warn(msg)
+            report_event(op=WALAEventOperation.Download,
+                            is_success=False,
+                            message=msg,
+                            log_event=False)
+
             if isinstance(error, (InvalidContainerError, ResourceGoneError)):
                 # These are retryable errors that should force a goal state refresh in the host plugin
                 raise
diff --git a/azurelinuxagent/common/utils/restutil.py b/azurelinuxagent/common/utils/restutil.py
index 427157b0..15dd3470 100644
--- a/azurelinuxagent/common/utils/restutil.py
+++ b/azurelinuxagent/common/utils/restutil.py
@@ -172,6 +172,19 @@ def _parse_url(url):
         secure = True
     return o.hostname, o.port, secure, rel_uri
 
+def _trim_url_parameters(url):
+    """
+    Parse URL and return scheme://hostname:port/path
+    """
+    o = urlparse(url)
+
+    if o.hostname:
+        if o.port:
+            return "{0}://{1}:{2}{3}".format(o.scheme, o.hostname, o.port, o.path)
+        else:
+            return "{0}://{1}{2}".format(o.scheme, o.hostname, o.path)
+
+    return url
 
 def is_valid_cidr(string_network):
     """
@@ -450,7 +463,7 @@ def http_request(method,
             return resp
 
         except httpclient.HTTPException as e:
-            clean_url = redact_sas_tokens_in_urls(url)
+            clean_url = _trim_url_parameters(url)
             msg = '[HTTP Failed] {0} {1} -- HttpException {2}'.format(method, clean_url, e)
             if _is_retry_exception(e):
                 continue
@@ -458,7 +471,7 @@ def http_request(method,
 
         except IOError as e:
             IOErrorCounter.increment(host=host, port=port)
-            clean_url = redact_sas_tokens_in_urls(url)
+            clean_url = _trim_url_parameters(url)
             msg = '[HTTP Failed] {0} {1} -- IOError {2}'.format(method, clean_url, e)
             continue
 


Test patch:
diff --git a/tests/utils/test_rest_util.py b/tests/utils/test_rest_util.py
index d3ddbb34..02431b62 100644
--- a/tests/utils/test_rest_util.py
+++ b/tests/utils/test_rest_util.py
@@ -115,6 +115,25 @@ class TestHttpOperations(AgentTestCase):
         self.assertEqual(None, host) 
         self.assertEqual(rel_uri, "None") 
 
+    def test_trim_url_parameters(self):
+        test_uri = "http://abc.def/ghi#hash?jkl=mn"
+        rel_uri = restutil._trim_url_parameters(test_uri)
+        self.assertEqual("http://abc.def/ghi", rel_uri)
+
+        test_uri = "https://abc.def/ghi?jkl=mn"
+        rel_uri = restutil._trim_url_parameters(test_uri)
+        self.assertEqual("https://abc.def/ghi", rel_uri)
+
+        test_uri = "https://abc.def:8443/ghi?jkl=mn"
+        rel_uri = restutil._trim_url_parameters(test_uri)
+        self.assertEqual("https://abc.def:8443/ghi", rel_uri) 
+
+        rel_uri = restutil._trim_url_parameters("")
+        self.assertEqual(rel_uri, "")
+
+        rel_uri = restutil._trim_url_parameters("None")
+        self.assertEqual(rel_uri, "None")
+
     def test_cleanup_sas_tokens_from_urls_for_normal_cases(self):
         test_url = "http://abc.def/ghi#hash?jkl=mn"
         filtered_url = restutil.redact_sas_tokens_in_urls(test_url)