Repository: czerwonk/ping_exporter

Issue:
[target] allow for canceling dns lookups if resolver is failing or unreachable
in kubernetes, if our `config.dns.nameserver` isn't working, our liveness checks will fail on deploy due to the dns lookups not returning in time and the collector not starting.

logs from kubernetes with default config except for a bogus `config.dns.nameserver`:

```
time="2025-03-31T19:28:49Z" level=info msg="rtt units: 2" func=main.main file="/go/ping_exporter/main.go:93"
time="2025-03-31T19:28:49Z" level=info msg="adding target for host 8.8.8.8 ({8.8.8.8 })" func="main.(*target).add" file="/go/ping_exporter/target.go:141"
time="2025-03-31T19:28:49Z" level=info msg="adding target for host 2001:4860:4860::8844 ({2001:4860:4860::8844 })" func="main.(*target).add" file="/go/ping_exporter/target.go:141"
time="2025-03-31T19:28:49Z" level=info msg="adding target for host 2001:4860:4860::8888 ({2001:4860:4860::8888 })" func="main.(*target).add" file="/go/ping_exporter/target.go:141"
time="2025-03-31T19:28:49Z" level=info msg="adding target for host 8.8.4.4 ({8.8.4.4 })" func="main.(*target).add" file="/go/ping_exporter/target.go:141"
```

config for posterity:
```
dns:
  nameserver: kubernetes.default.svc.cluster.local
  refresh: 2m15s
ping:
  history-size: 42
  interval: 2s
  payload-size: 120
  timeout: 3s
targets:
- 8.8.8.8
- 8.8.4.4
- 2001:4860:4860::8888
- 2001:4860:4860::8844
- google.com
```

now if we were to add a measly timeout to our target lookup, we can properly get to the collector and pass our liveness probe

the question is, should this be gated by config flag and should the timeout be tunable in config?

Hints from issue comments:
@daveoy: tested the fix in #135 in kind using the `--dns.timeout=3s` option and added the timeout field to the config testdata

output now, also the pod is no longer crashlooping due to livenessprobe failures:
```
time="2025-03-31T21:07:12Z" level=info msg="rtt units: 2" func=main.main file="/go/ping_exporter/main.go:94"
time="2025-03-31T21:07:12Z" level=info msg="DNS timeout enabled: using 3000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:07:12Z" level=info msg="DNS timeout enabled: using 3000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:07:12Z" level=info msg="DNS timeout enabled: using 3000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:07:12Z" level=info msg="adding target for host 2001:4860:4860::8844 ({2001:4860:4860::8844 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:07:12Z" level=info msg="DNS timeout enabled: using 3000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:07:12Z" level=info msg="adding target for host 2001:4860:4860::8888 ({2001:4860:4860::8888 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:07:12Z" level=info msg="adding target for host 8.8.8.8 ({8.8.8.8 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:07:12Z" level=info msg="DNS timeout enabled: using 3000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:07:12Z" level=info msg="adding target for host 8.8.4.4 ({8.8.4.4 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:07:15Z" level=error msg="failed to setup target: error resolving target 'google.com': lookup google.com: i/o timeout" func=main.upsertTargets.func1 file="/go/ping_exporter/main.go:202"
time="2025-03-31T21:07:15Z" level=info msg="Starting ping exporter (Version: 1.1.3)" func=main.startServer file="/go/ping_exporter/main.go:297"
time="2025-03-31T21:07:15Z" level=info msg="Listening for /metrics on :9427 (HTTP)" func=main.startServer file="/go/ping_exporter/main.go:333"
```@daveoy: also tested with the following config file (via helm / skaffold):
```
dns:
  nameserver: kubernetes.default.svc.cluster.local
  refresh: 2m15s
  timeout: 6s
ping:
  history-size: 42
  interval: 2s
  payload-size: 120
  timeout: 3s
targets:
- 8.8.8.8
- 8.8.4.4
- 2001:4860:4860::8888
- 2001:4860:4860::8844
- google.com
```

produces the following output:
```
time="2025-03-31T21:17:59Z" level=info msg="rtt units: 2" func=main.main file="/go/ping_exporter/main.go:94"
time="2025-03-31T21:17:59Z" level=info msg="DNS timeout enabled: using 6000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:17:59Z" level=info msg="DNS timeout enabled: using 6000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:17:59Z" level=info msg="DNS timeout enabled: using 6000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:17:59Z" level=info msg="adding target for host 2001:4860:4860::8888 ({2001:4860:4860::8888 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:17:59Z" level=info msg="adding target for host 8.8.4.4 ({8.8.4.4 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:17:59Z" level=info msg="DNS timeout enabled: using 6000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:17:59Z" level=info msg="adding target for host 8.8.8.8 ({8.8.8.8 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:17:59Z" level=info msg="DNS timeout enabled: using 6000000000" func="main.(*target).addOrUpdateMonitor" file="/go/ping_exporter/target.go:93"
time="2025-03-31T21:17:59Z" level=info msg="adding target for host 2001:4860:4860::8844 ({2001:4860:4860::8844 })" func="main.(*target).add" file="/go/ping_exporter/target.go:149"
time="2025-03-31T21:18:05Z" level=error msg="failed to setup target: error resolving target 'google.com': lookup google.com: i/o timeout" func=main.upsertTargets.func1 file="/go/ping_exporter/main.go:202"
time="2025-03-31T21:18:05Z" level=info msg="Starting ping exporter (Version: 1.1.3)" func=main.startServer file="/go/ping_exporter/main.go:297"
time="2025-03-31T21:18:05Z" level=info msg="Listening for /metrics on :9427 (HTTP)" func=main.startServer file="/go/ping_exporter/main.go:333"
```

and no crashlooping livenessprobe failures

Solution patch:
diff --git a/config/config.go b/config/config.go
index 5ee55ad..ee9ecd1 100644
--- a/config/config.go
+++ b/config/config.go
@@ -23,6 +23,7 @@ type Config struct {
 	DNS struct {
 		Refresh    duration `yaml:"refresh"`
 		Nameserver string   `yaml:"nameserver"`
+		Timeout    duration `yaml:"timeout"`
 	} `yaml:"dns"`
 
 	Options struct {
diff --git a/main.go b/main.go
index e36095d..9af0c18 100644
--- a/main.go
+++ b/main.go
@@ -45,6 +45,7 @@ var (
 	historySize             = kingpin.Flag("ping.history-size", "Number of results to remember per target").Default("10").Int()
 	dnsRefresh              = kingpin.Flag("dns.refresh", "Interval for refreshing DNS records and updating targets accordingly (0 if disabled)").Default("1m").Duration()
 	dnsNameServer           = kingpin.Flag("dns.nameserver", "DNS server used to resolve hostname of targets").Default("").String()
+	dnsLookupTimeout        = kingpin.Flag("dns.timeout", "Timeout for DNS resolution").Default("0s").Duration()
 	disableIPv6             = kingpin.Flag("options.disable-ipv6", "Disable DNS from resolving IPv6 AAAA records").Default().Bool()
 	disableIPv4             = kingpin.Flag("options.disable-ipv4", "Disable DNS from resolving IPv4 A records").Default().Bool()
 	logLevel                = kingpin.Flag("log.level", "Only log messages with the given severity or above. Valid levels: [debug, info, warn, error, fatal]").Default("info").String()
@@ -196,7 +197,7 @@ func upsertTargets(globalTargets *targets, resolver *net.Resolver, cfg *config.C
 			err := newTarget.addOrUpdateMonitor(monitor, targetOpts{
 				disableIPv4: cfg.Options.DisableIPv4,
 				disableIPv6: cfg.Options.DisableIPv6,
-			})
+			}, cfg)
 			if err != nil {
 				log.Errorf("failed to setup target: %v", err)
 			}
@@ -283,7 +284,7 @@ func refreshDNS(tar *targets, monitor *mon.Monitor, cfg *config.Config) {
 			err := ta.addOrUpdateMonitor(monitor, targetOpts{
 				disableIPv4: cfg.Options.DisableIPv4,
 				disableIPv6: cfg.Options.DisableIPv6,
-			})
+			}, cfg)
 			if err != nil {
 				log.Errorf("could not refresh dns: %v", err)
 			}
@@ -466,6 +467,9 @@ func addFlagToConfig(cfg *config.Config) {
 	if !cfg.Options.DisableIPv4 {
 		cfg.Options.DisableIPv4 = *disableIPv4
 	}
+	if cfg.DNS.Timeout == 0 {
+		cfg.DNS.Timeout.Set(*dnsLookupTimeout)
+	}
 }
 
 const indexHTML = `<!doctype html>
diff --git a/target.go b/target.go
index c589bb5..dc67cb9 100644
--- a/target.go
+++ b/target.go
@@ -10,6 +10,7 @@ import (
 	"sync"
 	"time"
 
+	"github.com/czerwonk/ping_exporter/config"
 	mon "github.com/digineo/go-ping/monitor"
 	log "github.com/sirupsen/logrus"
 )
@@ -83,11 +84,18 @@ func (t *target) removeFromMonitor(monitor *mon.Monitor) {
 	}
 }
 
-func (t *target) addOrUpdateMonitor(monitor *mon.Monitor, opts targetOpts) error {
+func (t *target) addOrUpdateMonitor(monitor *mon.Monitor, opts targetOpts, cfg *config.Config) error {
 	t.mutex.Lock()
 	defer t.mutex.Unlock()
 
-	addrs, err := t.resolver.LookupIPAddr(context.Background(), t.host)
+	ctx := context.Background()
+	if cfg.DNS.Timeout.Duration() != time.Duration(0*time.Second) {
+		log.Infof("DNS timeout enabled: using %+v", cfg.DNS.Timeout)
+		var cancel context.CancelFunc
+		ctx, cancel = context.WithTimeout(context.Background(), cfg.DNS.Timeout.Duration())
+		defer cancel()
+	}
+	addrs, err := t.resolver.LookupIPAddr(ctx, t.host)
 	if err != nil {
 		return fmt.Errorf("error resolving target '%s': %w", t.host, err)
 	}



Test patch:
diff --git a/config/config_test.go b/config/config_test.go
index a1839b7..c4bee1c 100644
--- a/config/config_test.go
+++ b/config/config_test.go
@@ -46,6 +46,9 @@ func TestParseConfig(t *testing.T) {
 	if expected := 2*time.Minute + 15*time.Second; time.Duration(c.DNS.Refresh) != expected {
 		t.Errorf("expected dns.refresh to be %v, got %v", expected, c.DNS.Refresh)
 	}
+	if expected := 5 * time.Second; time.Duration(c.DNS.Timeout) != expected {
+		t.Errorf("expected dns.timeout to be %v, got %v", expected, c.DNS.Timeout)
+	}
 	if expected := "1.1.1.1"; c.DNS.Nameserver != expected {
 		t.Errorf("expected dns.nameserver to be %q, got %q", expected, c.DNS.Nameserver)
 	}
diff --git a/config/testdata/config_test.yml b/config/testdata/config_test.yml
index 3535517..8aaa1e3 100644
--- a/config/testdata/config_test.yml
+++ b/config/testdata/config_test.yml
@@ -9,6 +9,7 @@ targets:
 dns:
   refresh: 2m15s
   nameserver: 1.1.1.1
+  timeout: 5s
 
 ping:
   interval: 2s