Repository: Azure/WALinuxAgent
Pull request #1564 (2019-06-23 22:26:35)

Adding unittests to test the collect_and_send_events call
<!-- DO NOT DELETE THIS TEMPLATE -->

## Description
The collect_and_send_events code needed unittests to verify its workings.

<!--
Please add an informative description that covers that changes made by the pull request. 
This checklist is used to make sure that common issues in a pull request are addressed.
This will expedite the process of getting your pull request merged and avoid extra work on your part to fix issues discovered during the review process.
-->

---

### PR information
- [x] The title of the PR is clear and informative.
- [x] There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md).
- [x] Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
- [ ] If applicable, the PR references the bug/issue that it fixes in the description.
- [x] New Unit tests were added for the changes made and Travis.CI is passing.

### Quality of Code and Contribution Guidelines
- [x] I have read the [contribution guidelines](https://github.com/Azure/WALinuxAgent/blob/master/.github/CONTRIBUTING.md).

<!-- Reviewable:start -->
---
This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/azure/walinuxagent/1564)
<!-- Reviewable:end -->


codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=h1) Report
> Merging [#1564](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=desc) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/8761775464a6f6cb5de073d7bfe6a66eb8312cad?src=pr&el=desc) will **increase** coverage by `0.73%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/graphs/tree.svg?width=650&token=ae90E2KYCw&height=150&src=pr)](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #1564      +/-   ##
===========================================
+ Coverage    63.38%   64.11%   +0.73%     
===========================================
  Files           78       78              
  Lines        11056    11056              
  Branches      1559     1559              
===========================================
+ Hits          7008     7089      +81     
+ Misses        3709     3625      -84     
- Partials       339      342       +3
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/exception.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9leGNlcHRpb24ucHk=) | `92.59% <0%> (+1.23%)` | :arrow_up: |
| [azurelinuxagent/common/protocol/wire.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC93aXJlLnB5) | `72.76% <0%> (+3.56%)` | :arrow_up: |
| [azurelinuxagent/ga/monitor.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL21vbml0b3IucHk=) | `81.31% <0%> (+14.53%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=footer). Last update [8761775...54f8d71](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

codecov[bot]: # [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=h1) Report
> Merging [#1564](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=desc) into [develop](https://codecov.io/gh/Azure/WALinuxAgent/commit/8761775464a6f6cb5de073d7bfe6a66eb8312cad?src=pr&el=desc) will **increase** coverage by `0.73%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/graphs/tree.svg?width=650&token=ae90E2KYCw&height=150&src=pr)](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           develop    #1564      +/-   ##
===========================================
+ Coverage    63.38%   64.11%   +0.73%     
===========================================
  Files           78       78              
  Lines        11056    11056              
  Branches      1559     1559              
===========================================
+ Hits          7008     7089      +81     
+ Misses        3709     3625      -84     
- Partials       339      342       +3
```


| [Impacted Files](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [azurelinuxagent/common/exception.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9leGNlcHRpb24ucHk=) | `92.59% <0%> (+1.23%)` | :arrow_up: |
| [azurelinuxagent/common/protocol/wire.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2NvbW1vbi9wcm90b2NvbC93aXJlLnB5) | `72.76% <0%> (+3.56%)` | :arrow_up: |
| [azurelinuxagent/ga/monitor.py](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564/diff?src=pr&el=tree#diff-YXp1cmVsaW51eGFnZW50L2dhL21vbml0b3IucHk=) | `81.31% <0%> (+14.53%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=footer). Last update [8761775...54f8d71](https://codecov.io/gh/Azure/WALinuxAgent/pull/1564?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

narrieta: @vrdmr - how did we address reducing the frequency of those messages to the log (since it is echoed to the serial console)?
vrdmr: > @vrdmr - how did we address reducing the frequency of those messages to the log (since it is echoed to the serial console)?

I have changed the warning to use `periodic_warn()` and it is reset every 30 minutes. Thus we are reducing both the content written on the console as well as the count of the logs written.

Patch:
diff --git a/azurelinuxagent/common/protocol/wire.py b/azurelinuxagent/common/protocol/wire.py
index afdcc5e9..35464531 100644
--- a/azurelinuxagent/common/protocol/wire.py
+++ b/azurelinuxagent/common/protocol/wire.py
@@ -20,29 +20,21 @@ import json
 import os
 import random
 import re
-import sys
 import time
-import traceback
 import xml.sax.saxutils as saxutils
-
 from datetime import datetime
 
 import azurelinuxagent.common.conf as conf
-import azurelinuxagent.common.utils.fileutil as fileutil
 import azurelinuxagent.common.utils.textutil as textutil
-
 from azurelinuxagent.common.exception import ProtocolNotFoundError, \
     ResourceGoneError, ExtensionDownloadError
 from azurelinuxagent.common.future import httpclient, bytebuffer
-from azurelinuxagent.common.protocol.hostplugin import HostPluginProtocol, URI_FORMAT_GET_EXTENSION_ARTIFACT, \
-    HOST_PLUGIN_PORT
+from azurelinuxagent.common.protocol.hostplugin import HostPluginProtocol
 from azurelinuxagent.common.protocol.restapi import *
 from azurelinuxagent.common.utils.archive import StateFlusher
 from azurelinuxagent.common.utils.cryptutil import CryptUtil
 from azurelinuxagent.common.utils.textutil import parse_doc, findall, find, \
     findtext, getattrib, gettext, remove_bom, get_bytes_from_pem, parse_json
-from azurelinuxagent.common.version import AGENT_NAME
-from azurelinuxagent.common.osutil import get_osutil
 
 VERSION_INFO_URI = "http://{0}/?comp=versions"
 GOAL_STATE_URI = "http://{0}/machine/?comp=goalstate"
@@ -70,6 +62,8 @@ ENDPOINT_FINE_NAME = "WireServer"
 
 SHORT_WAITING_INTERVAL = 1  # 1 second
 
+MAX_EVENT_BUFFER_SIZE = 2**16 - 2**10
+
 
 class UploadError(HttpError):
     pass
@@ -1136,10 +1130,14 @@ class WireClient(object):
             if event.providerId not in buf:
                 buf[event.providerId] = ""
             event_str = event_to_v1(event)
-            if len(event_str) >= 63 * 1024:
-                logger.warn("Single event too large: {0}", event_str[300:])
+            if len(event_str) >= MAX_EVENT_BUFFER_SIZE:
+                details_of_event = [ustr(x.name) + ":" + ustr(x.value) for x in event.parameters if x.name in
+                                    ["Name", "Version", "Operation", "OperationSuccess"]]
+                logger.periodic_warn(logger.EVERY_HALF_HOUR,
+                                     "Single event too large: {0}, with the length: {1} more than the limit({2})"
+                                     .format(str(details_of_event), len(event_str), MAX_EVENT_BUFFER_SIZE))
                 continue
-            if len(buf[event.providerId] + event_str) >= 63 * 1024:
+            if len(buf[event.providerId] + event_str) >= MAX_EVENT_BUFFER_SIZE:
                 self.send_event(event.providerId, buf[event.providerId])
                 buf[event.providerId] = ""
             buf[event.providerId] = buf[event.providerId] + event_str
diff --git a/azurelinuxagent/ga/monitor.py b/azurelinuxagent/ga/monitor.py
index 99ff3ae9..a69b7946 100644
--- a/azurelinuxagent/ga/monitor.py
+++ b/azurelinuxagent/ga/monitor.py
@@ -176,7 +176,7 @@ class MonitorHandler(object):
             self.sysinfo.append(TelemetryEventParam("RAM", ram))
             self.sysinfo.append(TelemetryEventParam("Processors", processors))
         except OSUtilError as e:
-            logger.warn("Failed to get system info: {0}", e)
+            logger.warn("Failed to get system info: {0}", ustr(e))
 
         try:
             vminfo = self.protocol.get_vminfo()
@@ -191,7 +191,7 @@ class MonitorHandler(object):
             self.sysinfo.append(TelemetryEventParam("ContainerId",
                                                     vminfo.containerId))
         except ProtocolError as e:
-            logger.warn("Failed to get system info: {0}", e)
+            logger.warn("Failed to get system info: {0}", ustr(e))
 
         try:
             vminfo = self.imds_client.get_compute()
@@ -206,18 +206,20 @@ class MonitorHandler(object):
             self.sysinfo.append(TelemetryEventParam('ImageOrigin',
                                                     vminfo.image_origin))
         except (HttpError, ValueError) as e:
-            logger.warn("failed to get IMDS info: {0}", e)
+            logger.warn("failed to get IMDS info: {0}", ustr(e))
 
-    def collect_event(self, evt_file_name):
+    @staticmethod
+    def collect_event(evt_file_name):
         try:
             logger.verbose("Found event file: {0}", evt_file_name)
             with open(evt_file_name, "rb") as evt_file:
                 # if fail to open or delete the file, throw exception
-                data_str = evt_file.read().decode("utf-8", 'ignore')
+                data_str = evt_file.read().decode("utf-8")
             logger.verbose("Processed event file: {0}", evt_file_name)
             os.remove(evt_file_name)
             return data_str
-        except IOError as e:
+        except (IOError, UnicodeDecodeError) as e:
+            os.remove(evt_file_name)
             msg = "Failed to process {0}, {1}".format(evt_file_name, e)
             raise EventError(msg)
 
@@ -237,7 +239,7 @@ class MonitorHandler(object):
                     try:
                         data_str = self.collect_event(event_file_path)
                     except EventError as e:
-                        logger.error("{0}", e)
+                        logger.error("{0}", ustr(e))
                         continue
 
                     try:
@@ -245,7 +247,7 @@ class MonitorHandler(object):
                         self.add_sysinfo(event)
                         event_list.events.append(event)
                     except (ValueError, ProtocolError) as e:
-                        logger.warn("Failed to decode event file: {0}", e)
+                        logger.warn("Failed to decode event file: {0}", ustr(e))
                         continue
 
                 if len(event_list.events) == 0:
@@ -254,9 +256,9 @@ class MonitorHandler(object):
                 try:
                     self.protocol.report_event(event_list)
                 except ProtocolError as e:
-                    logger.error("{0}", e)
+                    logger.error("{0}", ustr(e))
             except Exception as e:
-                logger.warn("Failed to send events: {0}", e)
+                logger.warn("Failed to send events: {0}", ustr(e))
 
             self.last_event_collection = datetime.datetime.utcnow()
 
@@ -404,7 +406,7 @@ class MonitorHandler(object):
                         message=msg,
                         log_event=False)
             except Exception as e:
-                logger.warn("Failed to send heartbeat: {0}", e)
+                logger.warn("Failed to send heartbeat: {0}", ustr(e))
 
             self.last_telemetry_heartbeat = datetime.datetime.utcnow()
 


Test patch:
diff --git a/tests/common/test_event.py b/tests/common/test_event.py
index 23b84757..8a8ae923 100644
--- a/tests/common/test_event.py
+++ b/tests/common/test_event.py
@@ -17,12 +17,16 @@
 
 from __future__ import print_function
 
+import json
 from datetime import datetime, timedelta
 
 from azurelinuxagent.common.event import add_event, \
     WALAEventOperation, elapsed_milliseconds
+from azurelinuxagent.common.exception import EventError
 from azurelinuxagent.common.future import ustr
+from azurelinuxagent.common.utils.processutil import read_output
 from azurelinuxagent.common.version import CURRENT_VERSION
+from azurelinuxagent.ga.monitor import MonitorHandler
 
 from tests.tools import *
 
@@ -78,7 +82,6 @@ class TestEvent(AgentTestCase):
         self.assertTrue(event.should_emit_event("Foo", "1.2", "FauxOperation", True))
         self.assertTrue(event.should_emit_event("Foo", "1.2", "FauxOperation", False))
 
-
     def test_should_emit_event_handles_known_operations(self):
         event.__event_status__ = event.EventStatus()
 
@@ -219,6 +222,41 @@ class TestEvent(AgentTestCase):
         for filename in os.listdir(self.tmp_dir):
             self.assertEqual(".tld", filename[-4:])
 
+    def test_save_event_message_with_non_ascii_characters(self):
+        test_data_dir = os.path.join(data_dir, "events", "collect_and_send_extension_stdout_stderror")
+        msg = ""
+
+        with open(os.path.join(test_data_dir, "dummy_stdout_with_non_ascii_characters"), mode="r+b") as stdout:
+            with open(os.path.join(test_data_dir, "dummy_stderr_with_non_ascii_characters"), mode="r+b") as stderr:
+                msg = read_output(stdout, stderr)
+
+        duration = elapsed_milliseconds(datetime.utcnow())
+        log_msg = "{0}\n{1}".format("DummyCmd", "\n".join([line for line in msg.split('\n') if line != ""]))
+
+        add_event('test_extension', message=log_msg, duration=duration)
+
+        for tld_file in os.listdir(self.tmp_dir):
+            event_str = MonitorHandler.collect_event(os.path.join(self.tmp_dir, tld_file))
+            event_json = json.loads(event_str)
+
+            self.assertEqual(len(event_json["parameters"]), 8)
+
+            for i in event_json["parameters"]:
+                if i["name"] == "Name":
+                    self.assertEqual(i["value"], "test_extension")
+                if i["name"] == "Message":
+                    self.assertEqual(i["value"], log_msg)
+
+    def test_save_event_message_with_decode_errors(self):
+        tmp_file = os.path.join(self.tmp_dir, "tmp_file")
+        fileutil.write_file(tmp_file, "This is not JSON data", encoding="utf-16")
+
+        for tld_file in os.listdir(self.tmp_dir):
+            try:
+                MonitorHandler.collect_event(os.path.join(self.tmp_dir, tld_file))
+            except Exception as e:
+                self.assertIsInstance(e, EventError)
+
     def test_save_event_rollover(self):
         # We keep 1000 events only, and the older ones are removed.
 
diff --git a/tests/data/events/collect_and_send_events_invalid_data/1560752429123264-1.tld b/tests/data/events/collect_and_send_events_invalid_data/1560752429123264-1.tld
new file mode 100755
index 00000000..78e1189a
--- /dev/null
+++ b/tests/data/events/collect_and_send_events_invalid_data/1560752429123264-1.tld
@@ -0,0 +1,32 @@
+ {
+            "eventId": 1,
+            "providerId": "69B669B9-4AF8-4C50-BDC4-6006FA76E975",
+            "parameters": [
+                        {
+                        "name": "Name",
+                        "value": "Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux"
+                },
+                        {
+
+            			"name": "Version",
+            			"value": "1.11.5"
+             		},
+
+                        {
+                        "name": "Operation",
+                        "value": "DscPerformConsistency"
+                },
+                        {
+                        "name": "OperationSuccess",
+                        "value": false
+                },
+                        {
+                        "name": "Message",
+                        "value": "[0]....{u'PluginName': MI_STRING: u'Common', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'PatchManagement', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'Antimalware', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SecurityBaseline', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'DockerBaseline', u'Ensure': MI_STRING: u'Absent'}....\n....{u'PluginName': MI_STRING: u'ChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SoftwareChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'ServiceChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n[-1]: error msg: "Error occurred processing (0, u'nxOMSPlugin', {u'WorkspaceID': MI_STRING: u'5a46ca0a-e748-4262-9f0a-f55a05513d9e', u'Name': MI_STRING: u'SimpleOMSPluginConfiguration', u'Plugins': MI_INSTANCEA: [{u'PluginName': 'Common', u'Ensure': 'Present'}, {u'PluginName': 'PatchManagement', u'Ensure': 'Present'}, {u'PluginName': 'Antimalware', u'Ensure': 'Present'}, {u'PluginName': 'SecurityBaseline', u'Ensure': 'Present'}, {u'PluginName': 'DockerBaseline', u'Ensure': 'Absent'}, {u'PluginName': 'ChangeTracking', u'Ensure': 'Present'}, {u'PluginName': 'SoftwareChangeTracking', u'Ensure': 'Present'}, {u'PluginName': 'ServiceChangeTracking', u'Ensure': 'Present'}]})"\n....{u'PluginName': MI_STRING: u'Common', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'PatchManagement', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'Antimalware', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SecurityBaseline', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'DockerBaseline', u'Ensure': MI_STRING: u'Absent'}....\n....{u'PluginName': MI_STRING: u'ChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SoftwareChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'ServiceChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n[0][0][0]Exiting - closing socket\nExiting - closing socket\nExiting - closing socket\nExiting - closing socket\n\n---LOG---\n2019/06/17 06:14:49: DEBUG: Scripts/nxPackage.pyc(251):\nPackageGroup type is <type 'bool'>\n2019/06/17 06:15:31: WARNING: null(0): EventId=2 Priority=WARNING Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDisplaying messages from built-in DSC resources:\n\t WMI channel 1 \n\t ResourceID:  \n\t Message : []:                            [] Starting consistency engine.\n2019/06/17 06:15:31: WARNING: null(0): EventId=2 Priority=WARNING Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDisplaying messages from built-in DSC resources:\n\t WMI channel 1 \n\t ResourceID:  \n\t Message : []:                            [] Checking consistency for current configuration.\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/Common/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/PatchManagement/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/Antimalware/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/SecurityBaseline/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/SoftwareChangeTracking/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/ServiceChangeTracking/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:32: WARNING: null(0): EventId=2 Priority=WARNING Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDisplaying messages from built-in DSC resources:\n\t WMI channel 1 \n\t ResourceID:  \n\t Message : []:                            [] Consistency check completed.\n2019/06/17 06:15:32: ERROR: null(0): EventId=1 Priority=ERROR Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDSC Engine Error : \n\t Error Message cURL failed to perform on this base url: https://scus-agentservice-prod-1.azure-automation.net/Accounts/5a46ca0a-e748-4262-9f0a-f55a05513d9e/Nodes(AgentId='6446114e-b8a9-410b-af0e-41d4d0ce83b6')/SendReport with this error message: Problem with the local SSL certificate. Make sure cURL and SSL libraries are up to date. \n\tError Code : 1 \n2019/06/17 06:15:32: ERROR: null(0): EventId=1 Priority=ERROR Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDSC Engine Error : \n\t Error Message Failed to report to all reporting servers.  Last server failed with HTTP response code: 0. \n\tError Code : 1 \n"
+                },
+                        {
+                        "name": "Duration",
+                        "value": 300000
+                }
+            ]
+            }
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_events_invalid_data/1560752429123264.tld b/tests/data/events/collect_and_send_events_invalid_data/1560752429123264.tld
new file mode 100755
index 00000000..78e1189a
--- /dev/null
+++ b/tests/data/events/collect_and_send_events_invalid_data/1560752429123264.tld
@@ -0,0 +1,32 @@
+ {
+            "eventId": 1,
+            "providerId": "69B669B9-4AF8-4C50-BDC4-6006FA76E975",
+            "parameters": [
+                        {
+                        "name": "Name",
+                        "value": "Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux"
+                },
+                        {
+
+            			"name": "Version",
+            			"value": "1.11.5"
+             		},
+
+                        {
+                        "name": "Operation",
+                        "value": "DscPerformConsistency"
+                },
+                        {
+                        "name": "OperationSuccess",
+                        "value": false
+                },
+                        {
+                        "name": "Message",
+                        "value": "[0]....{u'PluginName': MI_STRING: u'Common', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'PatchManagement', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'Antimalware', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SecurityBaseline', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'DockerBaseline', u'Ensure': MI_STRING: u'Absent'}....\n....{u'PluginName': MI_STRING: u'ChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SoftwareChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'ServiceChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n[-1]: error msg: "Error occurred processing (0, u'nxOMSPlugin', {u'WorkspaceID': MI_STRING: u'5a46ca0a-e748-4262-9f0a-f55a05513d9e', u'Name': MI_STRING: u'SimpleOMSPluginConfiguration', u'Plugins': MI_INSTANCEA: [{u'PluginName': 'Common', u'Ensure': 'Present'}, {u'PluginName': 'PatchManagement', u'Ensure': 'Present'}, {u'PluginName': 'Antimalware', u'Ensure': 'Present'}, {u'PluginName': 'SecurityBaseline', u'Ensure': 'Present'}, {u'PluginName': 'DockerBaseline', u'Ensure': 'Absent'}, {u'PluginName': 'ChangeTracking', u'Ensure': 'Present'}, {u'PluginName': 'SoftwareChangeTracking', u'Ensure': 'Present'}, {u'PluginName': 'ServiceChangeTracking', u'Ensure': 'Present'}]})"\n....{u'PluginName': MI_STRING: u'Common', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'PatchManagement', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'Antimalware', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SecurityBaseline', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'DockerBaseline', u'Ensure': MI_STRING: u'Absent'}....\n....{u'PluginName': MI_STRING: u'ChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'SoftwareChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n....{u'PluginName': MI_STRING: u'ServiceChangeTracking', u'Ensure': MI_STRING: u'Present'}....\n[0][0][0]Exiting - closing socket\nExiting - closing socket\nExiting - closing socket\nExiting - closing socket\n\n---LOG---\n2019/06/17 06:14:49: DEBUG: Scripts/nxPackage.pyc(251):\nPackageGroup type is <type 'bool'>\n2019/06/17 06:15:31: WARNING: null(0): EventId=2 Priority=WARNING Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDisplaying messages from built-in DSC resources:\n\t WMI channel 1 \n\t ResourceID:  \n\t Message : []:                            [] Starting consistency engine.\n2019/06/17 06:15:31: WARNING: null(0): EventId=2 Priority=WARNING Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDisplaying messages from built-in DSC resources:\n\t WMI channel 1 \n\t ResourceID:  \n\t Message : []:                            [] Checking consistency for current configuration.\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/Common/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/PatchManagement/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/Antimalware/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/SecurityBaseline/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/SoftwareChangeTracking/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:31: ERROR: Scripts/nxOMSPlugin.pyc(366):\ncopy_all_files failed for src: /opt/microsoft/omsconfig/modules/nxOMSPlugin/DSCResources/MSFT_nxOMSPluginResource/Plugins/ServiceChangeTracking/conf dest: /etc/opt/microsoft/omsagent/conf/omsagent.d/ with error <type 'exceptions.IOError'>\n2019/06/17 06:15:32: WARNING: null(0): EventId=2 Priority=WARNING Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDisplaying messages from built-in DSC resources:\n\t WMI channel 1 \n\t ResourceID:  \n\t Message : []:                            [] Consistency check completed.\n2019/06/17 06:15:32: ERROR: null(0): EventId=1 Priority=ERROR Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDSC Engine Error : \n\t Error Message cURL failed to perform on this base url: https://scus-agentservice-prod-1.azure-automation.net/Accounts/5a46ca0a-e748-4262-9f0a-f55a05513d9e/Nodes(AgentId='6446114e-b8a9-410b-af0e-41d4d0ce83b6')/SendReport with this error message: Problem with the local SSL certificate. Make sure cURL and SSL libraries are up to date. \n\tError Code : 1 \n2019/06/17 06:15:32: ERROR: null(0): EventId=1 Priority=ERROR Job 174DEE2C-0E97-4EEA-9519-6F05A27715B5 : \nDSC Engine Error : \n\t Error Message Failed to report to all reporting servers.  Last server failed with HTTP response code: 0. \n\tError Code : 1 \n"
+                },
+                        {
+                        "name": "Duration",
+                        "value": 300000
+                }
+            ]
+            }
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_events_invalid_data/1560752429133818-1.tld b/tests/data/events/collect_and_send_events_invalid_data/1560752429133818-1.tld
new file mode 100755
index 00000000..3581eb17
--- /dev/null
+++ b/tests/data/events/collect_and_send_events_invalid_data/1560752429133818-1.tld
@@ -0,0 +1,32 @@
+ {
+            "eventId": 1,
+            "providerId": "69B669B9-4AF8-4C50-BDC4-6006FA76E975",
+            "parameters": [
+                        {
+                        "name": "Name",
+                        "value": "Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux"
+                },
+                        {
+
+            			"name": "Version",
+            			"value": "1.11.5"
+             		},
+
+                        {
+                        "name": "Operation",
+                        "value": "DscPerformInventory"
+                },
+                        {
+                        "name": "OperationSuccess",
+                        "value": false
+                },
+                        {
+                        "name": "Message",
+                        "value": "(0, {'__Inventory': MI_INSTANCEA: [{'Name': MI_STRING: 'gcc-8-base', 'Version': MI_STRING: '8.3.0-6ubuntu1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libgcc1', 'Version': MI_STRING: '1:8.3.0-6ubuntu1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libstdc++6', 'Version': MI_STRING: '8.3.0-6ubuntu1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libnss-systemd', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libsystemd0', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpam-systemd', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'systemd', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'udev', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libudev1', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'dbus', 'Version': MI_STRING: '1.12.2-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libdbus-1-3', 'Version': MI_STRING: '1.12.2-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'systemd-sysv', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libapt-pkg5.0', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libapt-inst2.0', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libdb5.3', 'Version': MI_STRING: '5.3.28-13.1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'apt', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'apt-utils', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython3.6', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libssl1.1', 'Version': MI_STRING: '1.1.1-1ubuntu2.1~18.04.2', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3.6', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3.6-minimal', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython3.6-stdlib', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython3.6-minimal', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython2.7', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python2.7', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython2.7-stdlib', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python2.7-minimal', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython2.7-minimal', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'update-notifier-common', 'Version': MI_STRING: '3.192.1.7', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libdw1', 'Version': MI_STRING: '0.170-0.4ubuntu0.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libelf1', 'Version': MI_STRING: '0.170-0.4ubuntu0.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libglib2.0-0', 'Version': MI_STRING: '2.56.4-0ubuntu0.18.04.3', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libglib2.0-data', 'Version': MI_STRING: '2.56.4-0ubuntu0.18.04.3', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'openssl', 'Version': MI_STRING: '1.1.1-1ubuntu2.1~18.04.2', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'xxd', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim-tiny', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim-runtime', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim-common', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3-gdbm', 'Version': MI_STRING: '3.6.8-1~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libcups2', 'Version': MI_STRING: '2.2.7-1ubuntu2.6', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-modules-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-image-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-image-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure-headers-4.18.0-1019', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-headers-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-headers-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure-tools-4.18.0-1019', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-tools-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-tools-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure-cloud-tools-4.18.0-1019', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-cloud-tools-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-cloud-tools-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3-cryptography', 'Version': MI_STRING: '2.1.4-1ubuntu1.3', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3-jinja2', 'Version': MI_STRING: '2.10-1ubuntu0.18.04.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}]})Exiting - closing socket\n\n---LOG---\n2019/06/17 06:16:55: DEBUG: Scripts/nxAvailableUpdates.py(62):\nCompleted checking Available Updates\n2019/06/17 06:17:56: DEBUG: Scripts/nxAvailableUpdates.pyc(51):\nStarting to check Available Updates\n2019/06/17 06:17:58: DEBUG: Scripts/nxAvailableUpdates.pyc(80):\nRetrieving update package list using cmd:LANG=en_US.UTF8 apt-get -s dist-upgrade | grep "^Inst"\n2019/06/17 06:17:59: DEBUG: Scripts/nxAvailableUpdates.pyc(96):\nNumber of packages being written to the XML: 56\n2019/06/17 06:17:59: DEBUG: Scripts/nxAvailableUpdates.pyc(62):\nCompleted checking Available Updates\n"
+                },
+                        {
+                        "name": "Duration",
+                        "value": 300000
+                }
+            ]
+            }
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_events_invalid_data/1560752429133818.tld b/tests/data/events/collect_and_send_events_invalid_data/1560752429133818.tld
new file mode 100755
index 00000000..3581eb17
--- /dev/null
+++ b/tests/data/events/collect_and_send_events_invalid_data/1560752429133818.tld
@@ -0,0 +1,32 @@
+ {
+            "eventId": 1,
+            "providerId": "69B669B9-4AF8-4C50-BDC4-6006FA76E975",
+            "parameters": [
+                        {
+                        "name": "Name",
+                        "value": "Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux"
+                },
+                        {
+
+            			"name": "Version",
+            			"value": "1.11.5"
+             		},
+
+                        {
+                        "name": "Operation",
+                        "value": "DscPerformInventory"
+                },
+                        {
+                        "name": "OperationSuccess",
+                        "value": false
+                },
+                        {
+                        "name": "Message",
+                        "value": "(0, {'__Inventory': MI_INSTANCEA: [{'Name': MI_STRING: 'gcc-8-base', 'Version': MI_STRING: '8.3.0-6ubuntu1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libgcc1', 'Version': MI_STRING: '1:8.3.0-6ubuntu1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libstdc++6', 'Version': MI_STRING: '8.3.0-6ubuntu1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libnss-systemd', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libsystemd0', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpam-systemd', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'systemd', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'udev', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libudev1', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'dbus', 'Version': MI_STRING: '1.12.2-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libdbus-1-3', 'Version': MI_STRING: '1.12.2-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'systemd-sysv', 'Version': MI_STRING: '237-3ubuntu10.22', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libapt-pkg5.0', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libapt-inst2.0', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libdb5.3', 'Version': MI_STRING: '5.3.28-13.1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'apt', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'apt-utils', 'Version': MI_STRING: '1.6.11', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython3.6', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libssl1.1', 'Version': MI_STRING: '1.1.1-1ubuntu2.1~18.04.2', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3.6', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3.6-minimal', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython3.6-stdlib', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython3.6-minimal', 'Version': MI_STRING: '3.6.8-1~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython2.7', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python2.7', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython2.7-stdlib', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python2.7-minimal', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libpython2.7-minimal', 'Version': MI_STRING: '2.7.15-4ubuntu4~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'update-notifier-common', 'Version': MI_STRING: '3.192.1.7', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libdw1', 'Version': MI_STRING: '0.170-0.4ubuntu0.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libelf1', 'Version': MI_STRING: '0.170-0.4ubuntu0.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libglib2.0-0', 'Version': MI_STRING: '2.56.4-0ubuntu0.18.04.3', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libglib2.0-data', 'Version': MI_STRING: '2.56.4-0ubuntu0.18.04.3', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'openssl', 'Version': MI_STRING: '1.1.1-1ubuntu2.1~18.04.2', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'xxd', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim-tiny', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim-runtime', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'vim-common', 'Version': MI_STRING: '2:8.0.1453-1ubuntu1.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3-gdbm', 'Version': MI_STRING: '3.6.8-1~18.04', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'libcups2', 'Version': MI_STRING: '2.2.7-1ubuntu2.6', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-modules-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-image-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-image-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure-headers-4.18.0-1019', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-headers-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-headers-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure-tools-4.18.0-1019', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-tools-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-tools-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-azure-cloud-tools-4.18.0-1019', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-cloud-tools-4.18.0-1019-azure', 'Version': MI_STRING: '4.18.0-1019.19~18.04.1', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'linux-cloud-tools-azure', 'Version': MI_STRING: '4.18.0.1019.18', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3-cryptography', 'Version': MI_STRING: '2.1.4-1ubuntu1.3', 'Architecture': MI_STRING: 'amd64', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates', 'BuildDate': MI_STRING: ''}, {'Name': MI_STRING: 'python3-jinja2', 'Version': MI_STRING: '2.10-1ubuntu0.18.04.1', 'Architecture': MI_STRING: 'all', 'Repository': MI_STRING: 'Ubuntu:18.04/bionic-updates, Ubuntu:18.04/bionic-security', 'BuildDate': MI_STRING: ''}]})Exiting - closing socket\n\n---LOG---\n2019/06/17 06:16:55: DEBUG: Scripts/nxAvailableUpdates.py(62):\nCompleted checking Available Updates\n2019/06/17 06:17:56: DEBUG: Scripts/nxAvailableUpdates.pyc(51):\nStarting to check Available Updates\n2019/06/17 06:17:58: DEBUG: Scripts/nxAvailableUpdates.pyc(80):\nRetrieving update package list using cmd:LANG=en_US.UTF8 apt-get -s dist-upgrade | grep "^Inst"\n2019/06/17 06:17:59: DEBUG: Scripts/nxAvailableUpdates.pyc(96):\nNumber of packages being written to the XML: 56\n2019/06/17 06:17:59: DEBUG: Scripts/nxAvailableUpdates.pyc(62):\nCompleted checking Available Updates\n"
+                },
+                        {
+                        "name": "Duration",
+                        "value": 300000
+                }
+            ]
+            }
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_events_unreadable_data/IncorrectExtension.tmp b/tests/data/events/collect_and_send_events_unreadable_data/IncorrectExtension.tmp
new file mode 100755
index 00000000..28f08445
--- /dev/null
+++ b/tests/data/events/collect_and_send_events_unreadable_data/IncorrectExtension.tmp
@@ -0,0 +1,32 @@
+ {
+            "eventId": 1,
+            "providerId": "69B669B9-4AF8-4C50-BDC4-6006FA76E975",
+            "parameters": [
+                        {
+                        "name": "Name",
+                        "value": "Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux"
+                },
+                        {
+
+            			"name": "Version",
+            			"value": "1.11.5"
+             		},
+
+                        {
+                        "name": "Operation",
+                        "value": "Install"
+                },
+                        {
+                        "name": "OperationSuccess",
+                        "value": false
+                },
+                        {
+                        "name": "Message",
+                        "value": "HelloWorld"
+                },
+                        {
+                        "name": "Duration",
+                        "value": 300000
+                }
+            ]
+            }
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_events_unreadable_data/UnreadableFile.tld b/tests/data/events/collect_and_send_events_unreadable_data/UnreadableFile.tld
new file mode 100755
index 00000000..28f08445
--- /dev/null
+++ b/tests/data/events/collect_and_send_events_unreadable_data/UnreadableFile.tld
@@ -0,0 +1,32 @@
+ {
+            "eventId": 1,
+            "providerId": "69B669B9-4AF8-4C50-BDC4-6006FA76E975",
+            "parameters": [
+                        {
+                        "name": "Name",
+                        "value": "Microsoft.EnterpriseCloud.Monitoring.OmsAgentForLinux"
+                },
+                        {
+
+            			"name": "Version",
+            			"value": "1.11.5"
+             		},
+
+                        {
+                        "name": "Operation",
+                        "value": "Install"
+                },
+                        {
+                        "name": "OperationSuccess",
+                        "value": false
+                },
+                        {
+                        "name": "Message",
+                        "value": "HelloWorld"
+                },
+                        {
+                        "name": "Duration",
+                        "value": 300000
+                }
+            ]
+            }
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_stderr_with_non_ascii_characters b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_stderr_with_non_ascii_characters
new file mode 100644
index 00000000..6f64ff61
--- /dev/null
+++ b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_stderr_with_non_ascii_characters
@@ -0,0 +1,1 @@
+STDERR Worldעיות אחרותआज""
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_stdout_with_non_ascii_characters b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_stdout_with_non_ascii_characters
new file mode 100644
index 00000000..a472c675
--- /dev/null
+++ b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_stdout_with_non_ascii_characters
@@ -0,0 +1,1 @@
+STDOUT Worldעיות אחרותआज""
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_valid_stderr b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_valid_stderr
new file mode 100644
index 00000000..9428735b
--- /dev/null
+++ b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_valid_stderr
@@ -0,0 +1,1 @@
+The five boxing wizards jump quickly.
\ No newline at end of file
diff --git a/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_valid_stdout b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_valid_stdout
new file mode 100644
index 00000000..8fe2a4b5
--- /dev/null
+++ b/tests/data/events/collect_and_send_extension_stdout_stderror/dummy_valid_stdout
@@ -0,0 +1,1 @@
+The quick brown fox jumps over the lazy dog.
\ No newline at end of file
diff --git a/tests/ga/test_monitor.py b/tests/ga/test_monitor.py
index 4595fae5..ac767fe2 100644
--- a/tests/ga/test_monitor.py
+++ b/tests/ga/test_monitor.py
@@ -15,16 +15,65 @@
 # Requires Python 2.6+ and Openssl 1.0+
 #
 import random
+import string
 from datetime import timedelta
 
 from azurelinuxagent.common.cgroup import CGroup
+from azurelinuxagent.common.event import EventLogger
+from azurelinuxagent.common.protocol.restapi import get_properties
 from azurelinuxagent.common.protocol.wire import WireProtocol
+from azurelinuxagent.common.utils import restutil
 from azurelinuxagent.ga.monitor import *
 from tests.common.test_cgroupstelemetry import make_new_cgroup, consume_cpu_time, consume_memory
 from tests.ga.test_extension import i_am_root, CGroupConfigurator
+from tests.protocol.mockwiredata import WireProtocolData, DATA_FILE
 from tests.tools import *
 
 
+class ResponseMock(Mock):
+    def __init__(self, status=restutil.httpclient.OK, response=None, reason=None):
+        Mock.__init__(self)
+        self.status = status
+        self.reason = reason
+        self.response = response
+
+    def read(self):
+        return self.response
+
+
+def random_generator(size=6, chars=string.ascii_uppercase + string.digits + string.ascii_lowercase):
+    return ''.join(random.choice(chars) for x in range(size))
+
+
+def create_event_message(size,
+                         name="DummyExtension",
+                         op=WALAEventOperation.Unknown,
+                         is_success=True,
+                         duration=0,
+                         version=CURRENT_VERSION,
+                         is_internal=False,
+                         evt_type="",
+                         invalid_chars=False):
+    return get_event_message(name=size, op=op, is_success=is_success, duration=duration,
+                             version=version, message=random_generator(size), evt_type=evt_type,
+                             is_internal=is_internal)
+
+
+def get_event_message(duration, evt_type, is_internal, is_success, message, name, op, version, eventId=1):
+    event = TelemetryEvent(eventId, "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX")
+    event.parameters.append(TelemetryEventParam('Name', name))
+    event.parameters.append(TelemetryEventParam('Version', str(version)))
+    event.parameters.append(TelemetryEventParam('IsInternal', is_internal))
+    event.parameters.append(TelemetryEventParam('Operation', op))
+    event.parameters.append(TelemetryEventParam('OperationSuccess', is_success))
+    event.parameters.append(TelemetryEventParam('Message', message))
+    event.parameters.append(TelemetryEventParam('Duration', duration))
+    event.parameters.append(TelemetryEventParam('ExtensionType', evt_type))
+
+    data = get_properties(event)
+    return json.dumps(data)
+
+
 @patch('azurelinuxagent.common.event.EventLogger.add_event')
 @patch('azurelinuxagent.common.osutil.get_osutil')
 @patch('azurelinuxagent.common.protocol.get_protocol_util')
@@ -32,7 +81,6 @@ from tests.tools import *
 @patch("azurelinuxagent.common.protocol.healthservice.HealthService._report")
 @patch("azurelinuxagent.common.utils.restutil.http_get")
 class TestMonitor(AgentTestCase):
-
     def test_parse_xml_event(self, *args):
         data_str = load_data('ext/event.xml')
         event = parse_xml_event(data_str)
@@ -229,6 +277,196 @@ class TestMonitor(AgentTestCase):
         monitor_handler.stop()
 
 
+@patch('azurelinuxagent.common.event.EventLogger.add_event')
+@patch('azurelinuxagent.common.osutil.get_osutil')
+@patch("azurelinuxagent.common.protocol.healthservice.HealthService._report")
+@patch("azurelinuxagent.common.protocol.wire.CryptUtil")
+@patch("azurelinuxagent.common.utils.restutil.http_get")
+class TestEventMonitoring(AgentTestCase):
+    def setUp(self):
+        AgentTestCase.setUp(self)
+        self.lib_dir = tempfile.mkdtemp()
+
+        self.event_logger = EventLogger()
+        self.event_logger.event_dir = os.path.join(self.lib_dir, "events")
+
+    def tearDown(self):
+        fileutil.rm_dirs(self.lib_dir)
+
+    def _create_mock(self, test_data, mock_http_get, MockCryptUtil, *args):
+        """Test enable/disable/uninstall of an extension"""
+        monitor_handler = get_monitor_handler()
+
+        # Mock protocol to return test data
+        mock_http_get.side_effect = test_data.mock_http_get
+        MockCryptUtil.side_effect = test_data.mock_crypt_util
+
+        protocol = WireProtocol("foo.bar")
+        protocol.detect()
+        protocol.report_ext_status = MagicMock()
+        protocol.report_vm_status = MagicMock()
+
+        monitor_handler.protocol_util.get_protocol = Mock(return_value=protocol)
+        return monitor_handler, protocol
+
+    @patch("azurelinuxagent.common.protocol.wire.WireClient.send_event")
+    @patch("azurelinuxagent.common.conf.get_lib_dir")
+    def test_collect_and_send_events_with_small_events(self, mock_lib_dir, patch_send_event, *args):
+        mock_lib_dir.return_value = self.lib_dir
+
+        test_data = WireProtocolData(DATA_FILE)
+        monitor_handler, protocol = self._create_mock(test_data, *args)
+        monitor_handler.init_protocols()
+
+        sizes = [15, 15, 15, 15]  # get the powers of 2 - 2**16 is the limit
+
+        for power in sizes:
+            size = 2 ** power
+            self.event_logger.save_event(create_event_message(size))
+        monitor_handler.collect_and_send_events()
+
+        # The send_event call would be called each time, as we are filling up the buffer up to the brim for each call.
+
+        self.assertEqual(4, patch_send_event.call_count)
+
+    @patch("azurelinuxagent.common.protocol.wire.WireClient.send_event")
+    @patch("azurelinuxagent.common.conf.get_lib_dir")
+    def test_collect_and_send_events_with_large_events(self, mock_lib_dir, patch_send_event, *args):
+        mock_lib_dir.return_value = self.lib_dir
+
+        test_data = WireProtocolData(DATA_FILE)
+        monitor_handler, protocol = self._create_mock(test_data, *args)
+        monitor_handler.init_protocols()
+
+        sizes = [17, 17, 17]  # get the powers of 2
+
+        for power in sizes:
+            size = 2 ** power
+            self.event_logger.save_event(create_event_message(size))
+
+        with patch("azurelinuxagent.common.logger.periodic_warn") as patch_periodic_warn:
+            monitor_handler.collect_and_send_events()
+            self.assertEqual(3, patch_periodic_warn.call_count)
+
+        # The send_event call should never be called as the events are larger than 2**16.
+        self.assertEqual(0, patch_send_event.call_count)
+
+    @patch("azurelinuxagent.common.protocol.wire.WireClient.send_event")
+    @patch("azurelinuxagent.common.conf.get_lib_dir")
+    def test_collect_and_send_events_with_invalid_events(self, mock_lib_dir, patch_send_event, *args):
+        mock_lib_dir.return_value = self.lib_dir
+        dummy_events_dir = os.path.join(data_dir, "events", "collect_and_send_events_invalid_data")
+        fileutil.mkdir(self.event_logger.event_dir)
+
+        test_data = WireProtocolData(DATA_FILE)
+        monitor_handler, protocol = self._create_mock(test_data, *args)
+        monitor_handler.init_protocols()
+
+        for filename in os.listdir(dummy_events_dir):
+            shutil.copy(os.path.join(dummy_events_dir, filename), self.event_logger.event_dir)
+
+        monitor_handler.collect_and_send_events()
+
+        # Invalid events
+        self.assertEqual(0, patch_send_event.call_count)
+
+    @patch("azurelinuxagent.common.protocol.wire.WireClient.send_event")
+    @patch("azurelinuxagent.common.conf.get_lib_dir")
+    def test_collect_and_send_events_cannot_read_events(self, mock_lib_dir, patch_send_event, *args):
+        mock_lib_dir.return_value = self.lib_dir
+        dummy_events_dir = os.path.join(data_dir, "events", "collect_and_send_events_unreadable_data")
+        fileutil.mkdir(self.event_logger.event_dir)
+
+        test_data = WireProtocolData(DATA_FILE)
+        monitor_handler, protocol = self._create_mock(test_data, *args)
+        monitor_handler.init_protocols()
+
+        for filename in os.listdir(dummy_events_dir):
+            shutil.copy(os.path.join(dummy_events_dir, filename), self.event_logger.event_dir)
+            fileutil.chmod(os.path.join(self.event_logger.event_dir, filename), 111)
+
+        monitor_handler.collect_and_send_events()
+
+        # Invalid events
+        self.assertEqual(0, patch_send_event.call_count)
+
+    @patch("azurelinuxagent.common.conf.get_lib_dir")
+    def test_collect_and_send_with_http_post_returning_503(self, mock_lib_dir, *args):
+        mock_lib_dir.return_value = self.lib_dir
+        fileutil.mkdir(self.event_logger.event_dir)
+
+        test_data = WireProtocolData(DATA_FILE)
+        monitor_handler, protocol = self._create_mock(test_data, *args)
+        monitor_handler.init_protocols()
+
+        sizes = [1, 2, 3]  # get the powers of 2, and multiple by 1024.
+
+        for power in sizes:
+            size = 2 ** power * 1024
+            self.event_logger.save_event(create_event_message(size))
+
+        with patch("azurelinuxagent.common.logger.error") as mock_error:
+            with patch("azurelinuxagent.common.utils.restutil.http_post") as mock_http_post:
+                mock_http_post.return_value = ResponseMock(
+                    status=restutil.httpclient.SERVICE_UNAVAILABLE,
+                    response="")
+                monitor_handler.collect_and_send_events()
+                self.assertEqual(1, mock_error.call_count)
+                self.assertEqual("[ProtocolError] [Wireserver Exception] [ProtocolError] [Wireserver Failed] "
+                                 "URI http://foo.bar/machine?comp=telemetrydata  [HTTP Failed] Status Code 503",
+                                 mock_error.call_args[0][1])
+                self.assertEqual(0, len(os.listdir(self.event_logger.event_dir)))
+
+    @patch("azurelinuxagent.common.conf.get_lib_dir")
+    def test_collect_and_send_with_send_event_generating_exception(self, mock_lib_dir, *args):
+        mock_lib_dir.return_value = self.lib_dir
+        fileutil.mkdir(self.event_logger.event_dir)
+
+        test_data = WireProtocolData(DATA_FILE)
+        monitor_handler, protocol = self._create_mock(test_data, *args)
+        monitor_handler.init_protocols()
+
+        sizes = [1, 2, 3]  # get the powers of 2, and multiple by 1024.
+
+        for power in sizes:
+            size = 2 ** power * 1024
+            self.event_logger.save_event(create_event_message(size))
+
+        monitor_handler.last_event_collection = datetime.datetime.utcnow() - timedelta(hours=1)
+        # This test validates that if we hit an issue while sending an event, we never send it again.
+        with patch("azurelinuxagent.common.logger.warn") as mock_warn:
+            with patch("azurelinuxagent.common.protocol.wire.WireClient.send_event") as patch_send_event:
+                patch_send_event.side_effect = Exception()
+                monitor_handler.collect_and_send_events()
+
+                self.assertEqual(1, mock_warn.call_count)
+                self.assertEqual(0, len(os.listdir(self.event_logger.event_dir)))
+
+    @patch("azurelinuxagent.common.conf.get_lib_dir")
+    def test_collect_and_send_with_call_wireserver_returns_http_error(self, mock_lib_dir, *args):
+        mock_lib_dir.return_value = self.lib_dir
+        fileutil.mkdir(self.event_logger.event_dir)
+
+        test_data = WireProtocolData(DATA_FILE)
+        monitor_handler, protocol = self._create_mock(test_data, *args)
+        monitor_handler.init_protocols()
+
+        sizes = [1, 2, 3]  # get the powers of 2, and multiple by 1024.
+
+        for power in sizes:
+            size = 2 ** power * 1024
+            self.event_logger.save_event(create_event_message(size))
+
+        monitor_handler.last_event_collection = datetime.datetime.utcnow() - timedelta(hours=1)
+        with patch("azurelinuxagent.common.logger.error") as mock_error:
+            with patch("azurelinuxagent.common.protocol.wire.WireClient.call_wireserver") as patch_call_wireserver:
+                patch_call_wireserver.side_effect = HttpError
+                monitor_handler.collect_and_send_events()
+
+                self.assertEqual(1, mock_error.call_count)
+                self.assertEqual(0, len(os.listdir(self.event_logger.event_dir)))
+
+
 @patch('azurelinuxagent.common.osutil.get_osutil')
 @patch('azurelinuxagent.common.protocol.get_protocol_util')
 @patch('azurelinuxagent.common.protocol.util.ProtocolUtil.get_protocol')
@@ -323,6 +561,7 @@ class TestExtensionMetricsDataTelemetry(AgentTestCase):
 
         monitor_handler.stop()
 
+    @patch("azurelinuxagent.common.osutil.default.DefaultOSUtil._get_proc_stat")
     def test_generate_extension_metrics_telemetry_dictionary(self, *args):
         num_polls = 10
         num_extensions = 1
diff --git a/tests/protocol/test_wire.py b/tests/protocol/test_wire.py
index 0ce67772..cadb1baa 100644
--- a/tests/protocol/test_wire.py
+++ b/tests/protocol/test_wire.py
@@ -24,6 +24,7 @@ from azurelinuxagent.common import event
 from azurelinuxagent.common.protocol.wire import *
 from azurelinuxagent.common.utils.shellutil import run_get_output
 from tests.common.osutil.test_default import running_under_travis
+from tests.ga.test_monitor import get_event_message, random_generator
 from tests.protocol.mockwiredata import *
 
 data_with_bom = b'\xef\xbb\xbfhehe'
@@ -31,6 +32,20 @@ testurl = 'http://foo'
 testtype = 'BlockBlob'
 wireserver_url = '168.63.129.16'
 
+
+def get_event(message, duration=30000, evt_type="", is_internal=False, is_success=True,
+              name="", op="Unknown", version=CURRENT_VERSION, eventId=1):
+    event = TelemetryEvent(eventId, "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX")
+    event.parameters.append(TelemetryEventParam('Name', name))
+    event.parameters.append(TelemetryEventParam('Version', str(version)))
+    event.parameters.append(TelemetryEventParam('IsInternal', is_internal))
+    event.parameters.append(TelemetryEventParam('Operation', op))
+    event.parameters.append(TelemetryEventParam('OperationSuccess', is_success))
+    event.parameters.append(TelemetryEventParam('Message', message))
+    event.parameters.append(TelemetryEventParam('Duration', duration))
+    event.parameters.append(TelemetryEventParam('ExtensionType', evt_type))
+    return event
+
 @patch("time.sleep")
 @patch("azurelinuxagent.common.protocol.wire.CryptUtil")
 @patch("azurelinuxagent.common.protocol.healthservice.HealthService._report")
@@ -505,6 +520,52 @@ class TestWireProtocol(AgentTestCase):
         # the body is not encoded, just check for equality
         self.assertIn(event_str, body_received)
 
+    @patch("azurelinuxagent.common.protocol.wire.WireClient.send_event")
+    def test_report_event_small_event(self, patch_send_event, *args):
+        event_list = TelemetryEventList()
+        client = WireProtocol(wireserver_url).client
+
+        event_str = random_generator(10)
+        event_list.events.append(get_event(message=event_str))
+
+        event_str = random_generator(100)
+        event_list.events.append(get_event(message=event_str))
+
+        event_str = random_generator(1000)
+        event_list.events.append(get_event(message=event_str))
+
+        event_str = random_generator(10000)
+        event_list.events.append(get_event(message=event_str))
+
+        client.report_event(event_list)
+
+        # It merges the messages into one message
+        self.assertEqual(patch_send_event.call_count, 1)
+
+    @patch("azurelinuxagent.common.protocol.wire.WireClient.send_event")
+    def test_report_event_multiple_events_to_fill_buffer(self, patch_send_event, *args):
+        event_list = TelemetryEventList()
+        client = WireProtocol(wireserver_url).client
+
+        event_str = random_generator(2**15)
+        event_list.events.append(get_event(message=event_str))
+        event_list.events.append(get_event(message=event_str))
+
+        client.report_event(event_list)
+
+        # It merges the messages into one message
+        self.assertEqual(patch_send_event.call_count, 2)
+
+    @patch("azurelinuxagent.common.protocol.wire.WireClient.send_event")
+    def test_report_event_large_event(self, patch_send_event, *args):
+        event_list = TelemetryEventList()
+        event_str = random_generator(2 ** 18)
+        event_list.events.append(get_event(message=event_str))
+        client = WireProtocol(wireserver_url).client
+        client.report_event(event_list)
+
+        self.assertEqual(patch_send_event.call_count, 0)
+
 
 class MockResponse:
     def __init__(self, body, status_code):