Make DataDog create alerts on submitted events

[#50018413]

DataDog was not designed for this. In order to make an event fire we
change the 'value' of the event and then alert when it crosses a certain
threshold. Unfortunately DataDog is not completely real time and there
seems to be some weirdness in how they calculate their rolling averages.
The process we use to trigger the event is as follows:

1. Submit a regular DataDog event so that it shows up in the event
   stream.

2. Submit a new set of metrics with an initial high value (10) to
   trigger the alert which is set to go off when the rolling average over
   the last 5 minutes is over a threshold (1).

3. Wait 2 minutes to make sure that DataDog has had time to catch the
   change and trigger the alert.

4. After the 2 minutes DataDog will continue through the other metrics
   in the initial set which contain a low value (0) and which fire 5 times
   over the next 10 minutes in order to aggressively bring the rolling
   average back to under 1 to end the alert.

The high initial value is required so that we alway trigger the
threshold crossing even if we have had recent '0' values which will
interfere with the rolling average. The multiple tear down values are
required to make sure that the rolling average comes down below the
threshold inside the rolling window otherwise the alert never ends.
Revert "Make DataDog create alerts on submitted events"

This reverts commit 471a0762532dda267aed0f0c8d3b64b878d9ac35.
Add a Datadog client that pages people upon events

[#50018413]

This is implemented as a decorator object to limit the leaky abstraction
that Datadog forces upon us. At the moment the health monitor is
required to know that we are using PagerDuty for our alerting. In the
future this change should be easy to remove so we can use the vanilla
Datadog client again.