Repository: TwiN/gatus

Issue:
Allow alerting on external endpoints that do not receive a push within a configurable time frame
> For step 3. , will it also include the negative "Said endpoint must automatically check if an alert has **_not_** been triggered" i.e. raise an alert if no result received within a specified period ? 
(aka push-based monitoring)

_Originally posted by @r3mi in https://github.com/TwiN/gatus/issues/722#issuecomment-2041015420_
            

Discussion:
@TwiN: Thank you for creating the issue.

Now that external endpoints have been implemented (#722, #724), this should probably be the next feature related to external endpoints that gets implemented as they kind of go hand in hand, especially for those using external endpoints to test connectivity. If there's no connectivity, Gatus' API won't be reachable, which means that Gatus wouldn't be able to trigger an alert without this feature.

**The feature in question should allow the user to configure a duration under which an update is expected to be received.** 

Should the duration elapse with no new status update, a status should be created to indicate a failure to receive an update within the expected time frame. 

This should in turn cause https://github.com/TwiN/gatus/blob/28339684bfd9bb0e13bed4409f88e64bff31f3b2/watchdog/alerting.go#L13-L22 to be called, which would then lead to `handleAlertsToTrigger` being called (due to the new result indicating failure to receive an update having its `Success` field set to `false`), incrementing `NumberOfFailuresInARow` https://github.com/TwiN/gatus/blob/28339684bfd9bb0e13bed4409f88e64bff31f3b2/watchdog/alerting.go#L24-L27 & triggering whichever alerts should be triggered.


The only proper name I can think of for this feature is "dead man's switch", but as silly as it may sound, I don't like how that'd look on the configuration:
```yaml
external-endpoints:
  - name: ...
    dead-man-switch:
      blackout-duration-until-automatic-failure: 1h
    alerts: 
      - type: slack
        send-on-resolved: true
```

Another consideration to make is the interaction between this feature and maintenance. While the maintenance period should prevent alerts from being triggered, should failure status be pushed anyways? Perhaps this should be an additional parameter on the maintenance configuration (e.g. `maintenance.silence-dead-man-switch`)?

Some food for thoughts.@onedr0p: > The only proper name I can think of for this feature is "dead man's switch", but as silly as it may sound, I don't like how that'd look on the configuration:

I've seen other services call this `heartbeat` instead of `dead man's switch`, and they also have a grace period that is configurable.

```yaml
external-endpoints:
  - name: ...
    heartbeat:
      interval: 5m
      grace-period: 5m
    alerts: 
      - type: slack
        send-on-resolved: true
```@atch24: I would love to see a feature similar to this feature request, just like implemented in healthchecks.io

This would allow to provide a cron schedule and then monitor crons and alert for crons that do not run or jobs that take too long to complete.

An example configuration might look like this:

```
external-endpoints:
  - name: ...
    heartbeat:
      schedule: 0 0 15 * *
      grace-period: 2m
```
@flohoss: If you look at betterstack.com or Healthchecks.io, they both implement similar settings, although betterstack does not really go into much detail. They offer a separate maintenance setting per heartbeat, but this is already [covered](https://github.com/TwiN/gatus?tab=readme-ov-file#maintenance).

Seems like the settings that make most sense to me would be:

```yaml
external-endpoints:
  - name: ...
    heartbeat:
      # Define the scheduling method to use: "interval" or "cron".
      # "interval" runs based on a fixed duration between heartbeats.
      # "cron" runs at specific times defined by a cron expression.
      method: 'cron' # Choose either "cron" or "interval"

      # Use this field if the method is "interval".
      # Defines the duration between heartbeats. Valid units are:
      # "ns" (nanoseconds), "us"/"µs" (microseconds), "ms" (milliseconds),
      # "s" (seconds), "m" (minutes), "h" (hours).
      # -> https://pkg.go.dev/time#ParseDuration
      interval: '24h'

      # Use these fields if the method is "cron".
      # "cron" defines the schedule using a cron expression.
      # For example, "0 0 * * *" means midnight every day.
      cron: '0 0 * * *'

      # Specifies the timezone to use when evaluating the cron expression.
      # This is required only for "cron" and is ignored for "interval".
      timezone: 'Europe/Berlin'

      # Specifies the grace period to tolerate delayed heartbeats.
      # This applies to both "interval" and "cron".
      grace-period: '1h'
```

This would be awesome as i am currently running healthchecks and gatus but only need healthchecks for 2 pings from a backup every night. Not really feeling the need of running Django for this...
@pswilde: I'd also really love this feature
I'm currently using Uptime Kuma, which does the job well, but I prefer Gatus' way of configuration

Uptime Kuma also uses the term "heartbeat" which fits well I think, which I use for backups and I have set for every 90000 seconds - a day plus an hour grace time), so previous comments talking about a grace-period would also be massively beneficial

![Image](https://github.com/user-attachments/assets/06f1d7e4-231c-4877-b233-b6706a0b4b81)

Thanks for the hard work on Gatus - great stuff!@TwiN: I think heartbeat makes sense. It's definitely on my radar, especially given how many :+1: this issue has amassed in the past year.@a-waider: Is there any progress on this topic?@TwiN: I started having a look at a possible implementation. 

Just thinking out loud here (rubber duck ~~debugging~~ programming): let's say we configure the heartbeat of an external endpoint with an interval of 1h, and no new result comes in for an hour, do we want to push a failed result/check? This would be the easiest way to implement this, as pushing a failed result would automatically trigger the alerts.

The sneaky thing with this implementation is that we could leverage the existing implementation for normal/internal endpoints here: https://github.com/TwiN/gatus/blob/00419a4b4a1322417d681ff606d88abc6bedb72e/watchdog/watchdog.go#L30-L34

As I think through this problem and go through the code, of course, past me is there to answer future me's questions: https://github.com/TwiN/gatus/blob/00419a4b4a1322417d681ff606d88abc6bedb72e/watchdog/watchdog.go#L55-L57

The statement above would no longer be true, as we'd now want to check external endpoints periodically -- but we'd do it using `external-endpoints[].heartbeat.interval` (if interval > 0)

@spencerhughes: > let's say we configure the heartbeat of an external endpoint with an interval of 1h, and no new result comes in for an hour, do we want to push a failed result/check?

I think that this makes the most sense. The success/failure of a heartbeat check is, to me, saying "this is considered a success if we receive a ping before the heartbeat interval timer reaches zero, and if that timer reaches zero that is considered a failure." If your heart is supposed to beat once every second and it doesn't beat for >1 second, your heart has transitioned to a failure state.

As other commenters have mentioned as well, I think we would want to have a grace period configuration for heartbeat checks as well. This is helpful to prevent alert spam in the event that intermittent connectivity or system load delays the health check.@TwiN: > As other commenters have mentioned as well, I think we would want to have a grace period configuration for heartbeat checks as well. This is helpful to prevent alert spam in the event that intermittent connectivity or system load delays the health check.

@spencerhughes I'm thinking of just relying on the `failure-threshold` of the configured alert for now, as its main purpose _is_ to prevent alert spam.

This implementation will likely be pretty minimal. I expect people will want more features, such as different ways of configuring the interval as one of the previous comments mentioned (cron), or maybe even add a grace period configuration like you requested, but right now, I'm focusing on getting this feature out ASAP. 

When I implemented external endpoints, I knew there was _some_ interest in it, but given the number of votes a simple feature complementing external endpoints (this issue) received, it's become clear to me that a lot more people I anticipated are using external endpoints, hence why I'd rather focus on getting something out faster and perfecting it later.@spencerhughes: > I'm thinking of just relying on the failure-threshold of the configured alert for now

I think that makes sense. Re-using an existing parameter is cleaner/simpler than implementing something new that is effectively analogous, and it doesn't take a huge amount of brain power to do *fail if I don't get a heartbeat in 5 minutes* * *let this alert fail 3 times before alerting me* == *15 minute grace period* (though maybe I'm undermining my own point here because I can't 100% decide if that would be a 15 or 20 minute grace period :sweat_smile:).@xXluki98Xx: @TwiN Thanks for working on this feature!

I can already cover one of our use cases with it, but I'm particularly interested in the CRON support.

Our scenario: some pipelines run every weekday around 2 AM, and we want to monitor their successful execution — and be notified if they fail.
The challenge is daylight saving time, which can cause slight shifts. (GitLab.com runs on UTC, but our software runs locally on MEST/CEST.)

So if the Feature is ready, should I open a new Issue for the Cron Support?@TwiN: @xXluki98Xx The feature isn't ready, it's still a WIP I'm working on during my free time, but sure, once it's merged, you can absolutely create an issue suggesting cron support.@TwiN: So, I've got good news for the many interested. I just finished the first implementation for this - #1173 

The bad news is that before I merge this into master, I would really appreciate it if a few people could confirm that it works as intended on their end.

Docker command: `docker pull ghcr.io/twin/gatus:heartbeat` (created using https://github.com/TwiN/gatus/actions/runs/16601219374)@markormesher: I gave it a very quick test with a 1m timeout and it seems to be working! I'll set it up properly today for a real job and see how it goes.@TwiN: This is now available under `latest`. 
Let me know if there's any issues.

P.S. I'll be deleting `ghcr.io/twin/gatus:heartbeat` as it is no longer necessary@darkrain-nl: Quick test here works great. Thanks for the great work.@TwiN: Awesome, thank you folks. I'll be releasing this in v5.21.0 today or tomorrow.@ghanjiboy: Hi @TwiN , glad to see this feature implemented.  I have a question, I have about 20 monitors that fall in this category and they have different heartbeat intervals as they are jobs - some are hourly, some daily and some weekly.   I noticed that the monitor doesnt show up in the dashboard until at least one of the intervals has passed -this seems like a bug - but I wanted to check with you first.  

Here's an example:

```
external-endpoints:
  - name: BackupPostgres
    token: JvCWBHNScw
    heartbeat:
      interval: 180h
    group: job
  - name: BackupNFSServer
    token: EsyPeGcgtA
    heartbeat:
      interval: 25h
    group: job
```
@TwiN: @ghanjiboy This isn't a bug. Essentially, if there's nothing to display, nothing should be displayed.

Until the first check, nothing is displayed, and this does not just apply to external endpoints.@ghanjiboy: Perfect, makes sense - thanks for the quick response and explanation!

Solution patch:
diff --git a/README.md b/README.md
index 62a54464..32d3332b 100644
--- a/README.md
+++ b/README.md
@@ -305,14 +305,16 @@ For instance:
 - You can monitor services that are not supported by Gatus
 - You can implement your own monitoring system while using Gatus as the dashboard
 
-| Parameter                      | Description                                                                                                            | Default       |
-|:-------------------------------|:-----------------------------------------------------------------------------------------------------------------------|:--------------|
-| `external-endpoints`           | List of endpoints to monitor.                                                                                          | `[]`          |
-| `external-endpoints[].enabled` | Whether to monitor the endpoint.                                                                                       | `true`        |
-| `external-endpoints[].name`    | Name of the endpoint. Can be anything.                                                                                 | Required `""` |
-| `external-endpoints[].group`   | Group name. Used to group multiple endpoints together on the dashboard. <br />See [Endpoint groups](#endpoint-groups). | `""`          |
-| `external-endpoints[].token`   | Bearer token required to push status to.                                                                               | Required `""` |
-| `external-endpoints[].alerts`  | List of all alerts for a given endpoint. <br />See [Alerting](#alerting).                                              | `[]`          |
+| Parameter                                 | Description                                                                                                                       | Default        |
+|:------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------|:---------------|
+| `external-endpoints`                      | List of endpoints to monitor.                                                                                                     | `[]`           |
+| `external-endpoints[].enabled`            | Whether to monitor the endpoint.                                                                                                  | `true`         |
+| `external-endpoints[].name`               | Name of the endpoint. Can be anything.                                                                                            | Required `""`  |
+| `external-endpoints[].group`              | Group name. Used to group multiple endpoints together on the dashboard. <br />See [Endpoint groups](#endpoint-groups).            | `""`           |
+| `external-endpoints[].token`              | Bearer token required to push status to.                                                                                          | Required `""`  |
+| `external-endpoints[].alerts`             | List of all alerts for a given endpoint. <br />See [Alerting](#alerting).                                                         | `[]`           |
+| `external-endpoints[].heartbeat`          | Heartbeat configuration for monitoring when the external endpoint stops sending updates.                                          | `{}`           |
+| `external-endpoints[].heartbeat.interval` | Expected interval between updates. If no update is received within this interval, alerts will be triggered. Must be at least 10s. | `0` (disabled) |
 
 Example:
 ```yaml
@@ -320,6 +322,8 @@ external-endpoints:
   - name: ext-ep-test
     group: core
     token: "potato"
+    heartbeat:
+      interval: 30m  # Automatically create a failure if no update is received within 30 minutes
     alerts:
       - type: discord
         description: "healthcheck failed"
diff --git a/api/external_endpoint.go b/api/external_endpoint.go
index 18c08802..c2c0fce6 100644
--- a/api/external_endpoint.go
+++ b/api/external_endpoint.go
@@ -7,6 +7,7 @@ import (
 
 	"github.com/TwiN/gatus/v5/config"
 	"github.com/TwiN/gatus/v5/config/endpoint"
+	"github.com/TwiN/gatus/v5/metrics"
 	"github.com/TwiN/gatus/v5/storage/store"
 	"github.com/TwiN/gatus/v5/storage/store/common"
 	"github.com/TwiN/gatus/v5/watchdog"
@@ -72,6 +73,9 @@ func CreateExternalEndpointResult(cfg *config.Config) fiber.Handler {
 			externalEndpoint.NumberOfSuccessesInARow = convertedEndpoint.NumberOfSuccessesInARow
 			externalEndpoint.NumberOfFailuresInARow = convertedEndpoint.NumberOfFailuresInARow
 		}
+		if cfg.Metrics {
+			metrics.PublishMetricsForEndpoint(convertedEndpoint, result)
+		}
 		// Return the result
 		return c.Status(200).SendString("")
 	}
diff --git a/config/endpoint/external_endpoint.go b/config/endpoint/external_endpoint.go
index 58f37fed..3563c54e 100644
--- a/config/endpoint/external_endpoint.go
+++ b/config/endpoint/external_endpoint.go
@@ -2,13 +2,19 @@ package endpoint
 
 import (
 	"errors"
+	"time"
 
 	"github.com/TwiN/gatus/v5/alerting/alert"
+	"github.com/TwiN/gatus/v5/config/endpoint/heartbeat"
+	"github.com/TwiN/gatus/v5/config/maintenance"
 )
 
 var (
 	// ErrExternalEndpointWithNoToken is the error with which Gatus will panic if an external endpoint is configured without a token.
 	ErrExternalEndpointWithNoToken = errors.New("you must specify a token for each external endpoint")
+
+	// ErrExternalEndpointHeartbeatIntervalTooLow is the error with which Gatus will panic if an external endpoint's heartbeat interval is less than 10 seconds.
+	ErrExternalEndpointHeartbeatIntervalTooLow = errors.New("heartbeat interval must be at least 10 seconds")
 )
 
 // ExternalEndpoint is an endpoint whose result is pushed from outside Gatus, which means that
@@ -30,6 +36,12 @@ type ExternalEndpoint struct {
 	// Alerts is the alerting configuration for the endpoint in case of failure
 	Alerts []*alert.Alert `yaml:"alerts,omitempty"`
 
+	// MaintenanceWindow is the configuration for per-endpoint maintenance windows
+	MaintenanceWindows []*maintenance.Config `yaml:"maintenance-windows,omitempty"`
+
+	// Heartbeat is the configuration that checks if the external endpoint has received new results when it should have.
+	Heartbeat heartbeat.Config `yaml:"heartbeat,omitempty"`
+
 	// NumberOfFailuresInARow is the number of unsuccessful evaluations in a row
 	NumberOfFailuresInARow int `yaml:"-"`
 
@@ -45,6 +57,10 @@ func (externalEndpoint *ExternalEndpoint) ValidateAndSetDefaults() error {
 	if len(externalEndpoint.Token) == 0 {
 		return ErrExternalEndpointWithNoToken
 	}
+	if externalEndpoint.Heartbeat.Interval != 0 && externalEndpoint.Heartbeat.Interval < 10*time.Second {
+		// If the heartbeat interval is set (non-0), it must be at least 10 seconds.
+		return ErrExternalEndpointHeartbeatIntervalTooLow
+	}
 	return nil
 }
 
diff --git a/config/endpoint/heartbeat/heartbeat.go b/config/endpoint/heartbeat/heartbeat.go
new file mode 100644
index 00000000..c27b5717
--- /dev/null
+++ b/config/endpoint/heartbeat/heartbeat.go
@@ -0,0 +1,11 @@
+package heartbeat
+
+import "time"
+
+// Config used to check if the external endpoint has received new results when it should have.
+// This configuration is used to trigger alerts when an external endpoint has no new results for a defined period of time
+type Config struct {
+	// Interval is the time interval at which Gatus verifies whether the external endpoint has received new results
+	// If no new result is received within the interval, the endpoint is marked as failed and alerts are triggered
+	Interval time.Duration `yaml:"interval"`
+}
diff --git a/storage/store/memory/memory.go b/storage/store/memory/memory.go
index 41714dcd..e47b9ee8 100644
--- a/storage/store/memory/memory.go
+++ b/storage/store/memory/memory.go
@@ -211,6 +211,23 @@ func (s *Store) DeleteAllTriggeredAlertsNotInChecksumsByEndpoint(ep *endpoint.En
 	return 0
 }
 
+// HasEndpointStatusNewerThan checks whether an endpoint has a status newer than the provided timestamp
+func (s *Store) HasEndpointStatusNewerThan(key string, timestamp time.Time) (bool, error) {
+	s.RLock()
+	defer s.RUnlock()
+	endpointStatus := s.cache.GetValue(key)
+	if endpointStatus == nil {
+		// If no endpoint exists, there's no newer status, so return false instead of an error
+		return false, nil
+	}
+	for _, result := range endpointStatus.(*endpoint.Status).Results {
+		if result.Timestamp.After(timestamp) {
+			return true, nil
+		}
+	}
+	return false, nil
+}
+
 // Clear deletes everything from the store
 func (s *Store) Clear() {
 	s.cache.Clear()
diff --git a/storage/store/sql/sql.go b/storage/store/sql/sql.go
index cb0dd058..8d3312d2 100644
--- a/storage/store/sql/sql.go
+++ b/storage/store/sql/sql.go
@@ -514,6 +514,24 @@ func (s *Store) DeleteAllTriggeredAlertsNotInChecksumsByEndpoint(ep *endpoint.En
 	return int(rowsAffects)
 }
 
+// HasEndpointStatusNewerThan checks whether an endpoint has a status newer than the provided timestamp
+func (s *Store) HasEndpointStatusNewerThan(key string, timestamp time.Time) (bool, error) {
+	if timestamp.IsZero() {
+		return false, errors.New("timestamp is zero")
+	}
+	var count int
+	err := s.db.QueryRow(
+		"SELECT COUNT(*) FROM endpoint_results WHERE endpoint_id = (SELECT endpoint_id FROM endpoints WHERE endpoint_key = $1 LIMIT 1) AND timestamp > $2",
+		key,
+		timestamp.UTC(),
+	).Scan(&count)
+	if err != nil {
+		// If the endpoint doesn't exist, we return false instead of an error
+		return false, nil
+	}
+	return count > 0, nil
+}
+
 // Clear deletes everything from the store
 func (s *Store) Clear() {
 	_, _ = s.db.Exec("DELETE FROM endpoints")
diff --git a/storage/store/store.go b/storage/store/store.go
index f7b581ec..7fb0ae06 100644
--- a/storage/store/store.go
+++ b/storage/store/store.go
@@ -57,6 +57,9 @@ type Store interface {
 	// This prevents triggered alerts that have been removed or modified from lingering in the database.
 	DeleteAllTriggeredAlertsNotInChecksumsByEndpoint(ep *endpoint.Endpoint, checksums []string) int
 
+	// HasEndpointStatusNewerThan checks whether an endpoint has a status newer than the provided timestamp
+	HasEndpointStatusNewerThan(key string, timestamp time.Time) (bool, error)
+
 	// Clear deletes everything from the store
 	Clear()
 
diff --git a/watchdog/watchdog.go b/watchdog/watchdog.go
index 7c5a1548..1a8f14c4 100644
--- a/watchdog/watchdog.go
+++ b/watchdog/watchdog.go
@@ -34,6 +34,14 @@ func Monitor(cfg *config.Config) {
 			go monitor(endpoint, cfg.Alerting, cfg.Maintenance, cfg.Connectivity, cfg.DisableMonitoringLock, cfg.Metrics, ctx)
 		}
 	}
+	for _, externalEndpoint := range cfg.ExternalEndpoints {
+		// Check if the external endpoint is enabled and is using heartbeat
+		// If the external endpoint does not use heartbeat, then it does not need to be monitored periodically, because
+		// alerting is checked every time an external endpoint is pushed to Gatus, unlike normal endpoints.
+		if externalEndpoint.IsEnabled() && externalEndpoint.Heartbeat.Interval > 0 {
+			go monitorExternalEndpointHeartbeat(externalEndpoint, cfg.Alerting, cfg.Maintenance, cfg.Connectivity, cfg.DisableMonitoringLock, cfg.Metrics, ctx)
+		}
+	}
 }
 
 // monitor a single endpoint in a loop
@@ -96,6 +104,76 @@ func execute(ep *endpoint.Endpoint, alertingConfig *alerting.Config, maintenance
 	logr.Debugf("[watchdog.execute] Waiting for interval=%s before monitoring group=%s endpoint=%s (key=%s) again", ep.Interval, ep.Group, ep.Name, ep.Key())
 }
 
+func monitorExternalEndpointHeartbeat(ee *endpoint.ExternalEndpoint, alertingConfig *alerting.Config, maintenanceConfig *maintenance.Config, connectivityConfig *connectivity.Config, disableMonitoringLock bool, enabledMetrics bool, ctx context.Context) {
+	ticker := time.NewTicker(ee.Heartbeat.Interval)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			logr.Warnf("[watchdog.monitorExternalEndpointHeartbeat] Canceling current execution of group=%s; endpoint=%s; key=%s", ee.Group, ee.Name, ee.Key())
+			return
+		case <-ticker.C:
+			executeExternalEndpointHeartbeat(ee, alertingConfig, maintenanceConfig, connectivityConfig, disableMonitoringLock, enabledMetrics)
+		}
+	}
+}
+
+func executeExternalEndpointHeartbeat(ee *endpoint.ExternalEndpoint, alertingConfig *alerting.Config, maintenanceConfig *maintenance.Config, connectivityConfig *connectivity.Config, disableMonitoringLock bool, enabledMetrics bool) {
+	if !disableMonitoringLock {
+		// By placing the lock here, we prevent multiple endpoints from being monitored at the exact same time, which
+		// could cause performance issues and return inaccurate results
+		monitoringMutex.Lock()
+		defer monitoringMutex.Unlock()
+	}
+	// If there's a connectivity checker configured, check if Gatus has internet connectivity
+	if connectivityConfig != nil && connectivityConfig.Checker != nil && !connectivityConfig.Checker.IsConnected() {
+		logr.Infof("[watchdog.monitorExternalEndpointHeartbeat] No connectivity; skipping execution")
+		return
+	}
+	logr.Debugf("[watchdog.monitorExternalEndpointHeartbeat] Checking heartbeat for group=%s; endpoint=%s; key=%s", ee.Group, ee.Name, ee.Key())
+	convertedEndpoint := ee.ToEndpoint()
+	hasReceivedResultWithinHeartbeatInterval, err := store.Get().HasEndpointStatusNewerThan(ee.Key(), time.Now().Add(-ee.Heartbeat.Interval))
+	if err != nil {
+		logr.Errorf("[watchdog.monitorExternalEndpointHeartbeat] Failed to check if endpoint has received a result within the heartbeat interval: %s", err.Error())
+		return
+	}
+	if hasReceivedResultWithinHeartbeatInterval {
+		// If we received a result within the heartbeat interval, we don't want to create a successful result, so we
+		// skip the rest. We don't have to worry about alerting or metrics, because if the previous heartbeat failed
+		// while this one succeeds, it implies that there was a new result pushed, and that result being pushed
+		// should've resolved the alert.
+		logr.Infof("[watchdog.monitorExternalEndpointHeartbeat] Checked heartbeat for group=%s; endpoint=%s; key=%s; success=%v; errors=%d", ee.Group, ee.Name, ee.Key(), hasReceivedResultWithinHeartbeatInterval, 0)
+		return
+	}
+	// All code after this point assumes the heartbeat failed
+	result := &endpoint.Result{
+		Timestamp: time.Now(),
+		Success:   false,
+		Errors:    []string{"heartbeat: no update received within " + ee.Heartbeat.Interval.String()},
+	}
+	if enabledMetrics {
+		metrics.PublishMetricsForEndpoint(convertedEndpoint, result)
+	}
+	UpdateEndpointStatuses(convertedEndpoint, result)
+	logr.Infof("[watchdog.monitorExternalEndpointHeartbeat] Checked heartbeat for group=%s; endpoint=%s; key=%s; success=%v; errors=%d; duration=%s", ee.Group, ee.Name, ee.Key(), result.Success, len(result.Errors), result.Duration.Round(time.Millisecond))
+	inEndpointMaintenanceWindow := false
+	for _, maintenanceWindow := range ee.MaintenanceWindows {
+		if maintenanceWindow.IsUnderMaintenance() {
+			logr.Debug("[watchdog.monitorExternalEndpointHeartbeat] Under endpoint maintenance window")
+			inEndpointMaintenanceWindow = true
+		}
+	}
+	if !maintenanceConfig.IsUnderMaintenance() && !inEndpointMaintenanceWindow {
+		HandleAlerting(convertedEndpoint, result, alertingConfig)
+		// Sync the failure/success counters back to the external endpoint
+		ee.NumberOfSuccessesInARow = convertedEndpoint.NumberOfSuccessesInARow
+		ee.NumberOfFailuresInARow = convertedEndpoint.NumberOfFailuresInARow
+	} else {
+		logr.Debug("[watchdog.monitorExternalEndpointHeartbeat] Not handling alerting because currently in the maintenance window")
+	}
+	logr.Debugf("[watchdog.monitorExternalEndpointHeartbeat] Waiting for interval=%s before checking heartbeat for group=%s endpoint=%s (key=%s) again", ee.Heartbeat.Interval, ee.Group, ee.Name, ee.Key())
+}
+
 // UpdateEndpointStatuses updates the slice of endpoint statuses
 func UpdateEndpointStatuses(ep *endpoint.Endpoint, result *endpoint.Result) {
 	if err := store.Get().Insert(ep, result); err != nil {



Test patch:
diff --git a/storage/store/memory/memory_test.go b/storage/store/memory/memory_test.go
index 544577bf..2f21528d 100644
--- a/storage/store/memory/memory_test.go
+++ b/storage/store/memory/memory_test.go
@@ -84,6 +84,7 @@ var (
 // This test is simply an extra sanity check
 func TestStore_SanityCheck(t *testing.T) {
 	store, _ := NewStore(storage.DefaultMaximumNumberOfResults, storage.DefaultMaximumNumberOfEvents)
+	defer store.Clear()
 	defer store.Close()
 	store.Insert(&testEndpoint, &testSuccessfulResult)
 	endpointStatuses, _ := store.GetAllEndpointStatuses(paging.NewEndpointStatusParams())
@@ -134,3 +135,30 @@ func TestStore_Save(t *testing.T) {
 	store.Clear()
 	store.Close()
 }
+
+func TestStore_HasEndpointStatusNewerThan(t *testing.T) {
+	store, _ := NewStore(storage.DefaultMaximumNumberOfResults, storage.DefaultMaximumNumberOfEvents)
+	defer store.Clear()
+	defer store.Close()
+	// Insert a result
+	err := store.Insert(&testEndpoint, &testSuccessfulResult)
+	if err != nil {
+		t.Fatalf("expected no error while inserting result, got %v", err)
+	}
+	// Check with a timestamp in the past
+	hasNewerStatus, err := store.HasEndpointStatusNewerThan(testEndpoint.Key(), time.Now().Add(-time.Hour))
+	if err != nil {
+		t.Fatalf("expected no error, got %v", err)
+	}
+	if !hasNewerStatus {
+		t.Fatal("expected to have a newer status, but didn't")
+	}
+	// Check with a timestamp in the future
+	hasNewerStatus, err = store.HasEndpointStatusNewerThan(testEndpoint.Key(), time.Now().Add(time.Hour))
+	if err != nil {
+		t.Fatalf("expected no error, got %v", err)
+	}
+	if hasNewerStatus {
+		t.Fatal("expected not to have a newer status, but did")
+	}
+}
diff --git a/storage/store/sql/sql_test.go b/storage/store/sql/sql_test.go
index 5dbfb899..a9eb9ed1 100644
--- a/storage/store/sql/sql_test.go
+++ b/storage/store/sql/sql_test.go
@@ -853,3 +853,36 @@ func TestStore_DeleteAllTriggeredAlertsNotInChecksumsByEndpoint(t *testing.T) {
 		t.Error("expected alert3 to exist for ep2")
 	}
 }
+
+func TestStore_HasEndpointStatusNewerThan(t *testing.T) {
+	store, _ := NewStore("sqlite", t.TempDir()+"/TestStore_HasEndpointStatusNewerThan.db", false, storage.DefaultMaximumNumberOfResults, storage.DefaultMaximumNumberOfEvents)
+	defer store.Close()
+	// Insert an endpoint status
+	if err := store.Insert(&testEndpoint, &testSuccessfulResult); err != nil {
+		t.Fatal("expected no error, got", err.Error())
+	}
+	// Check if it has a status newer than 1 hour ago
+	hasNewerStatus, err := store.HasEndpointStatusNewerThan(testEndpoint.Key(), time.Now().Add(-time.Hour))
+	if err != nil {
+		t.Fatal("expected no error, got", err.Error())
+	}
+	if !hasNewerStatus {
+		t.Error("expected to have a newer status")
+	}
+	// Check if it has a status newer than 2 days ago
+	hasNewerStatus, err = store.HasEndpointStatusNewerThan(testEndpoint.Key(), time.Now().Add(-48*time.Hour))
+	if err != nil {
+		t.Fatal("expected no error, got", err.Error())
+	}
+	if !hasNewerStatus {
+		t.Error("expected to have a newer status")
+	}
+	// Check if there's a status newer than 1 hour in the future (silly test, but it should work)
+	hasNewerStatus, err = store.HasEndpointStatusNewerThan(testEndpoint.Key(), time.Now().Add(time.Hour))
+	if err != nil {
+		t.Fatal("expected no error, got", err.Error())
+	}
+	if hasNewerStatus {
+		t.Error("expected not to have a newer status in the future")
+	}
+}