Repository: prometheus/prometheus (language: go)

Issue:
Breaking change in DNS Windows name resolution in 2.39.1
### What did you do?

Some context first:

Most of our target are stored with their short name `hostname01` instead of their full DNS name `hostname01.site.domain`
The main reason is that the target are retrieved from our CMDB via scripts and that in our environment the conversion from short name to FQDN is not fully predictable and depend on the site ; It is the “suffix list” in the LAN interface that usually take care of this.

Prometheus never had issues using short name as targets, but as soon as we upgraded our staging server from **2.38.0** to **2.39.1** it stopped working


### What did you expect to see?

Same behavior as in **2.38**, been able to use short host name and let the OS handle the correct resolution

### What did you see instead? Under which circumstances?

As soon as we upgraded to **2.39.1** all the targets using short name when down with message looking like this :
`Get "http://HOSTNAME01:18895/metrics": dial tcp: lookup HOSTNAME01 on 10.1.1.1:53: no such host`
`Get "http://HOSTNAME02:18895/metrics": dial tcp: lookup HOSTNAME02 on 10.1.1.1:53: no such host`

10.1.1.1 been the local Prometheus LAN IP


### System information

Windows 2019, latest patch

### Prometheus version

```text
Prometheus 2.39.1 > Not Worning
Prometheus 2.38.0 > Working
```


### Prometheus configuration file

_No response_

### Alertmanager version

_No response_

### Alertmanager configuration file

_No response_

### Logs

```text
Get "http://HOSTNAME01:18895/metrics": dial tcp: lookup HOSTNAME01 on 10.1.1.1:53: no such host
Get "http://HOSTNAME02:18895/metrics": dial tcp: lookup HOSTNAME02 on 10.1.1.1:53: no such host
```



Discussion / hints:
This is a change from go1.19:


> Resolver.PreferGo is now implemented on Windows and Plan 9. It previously only worked on Unix platforms. Combined with Dialer.Resolver and Resolver.Dial, it's now possible to write portable programs and be in control of all DNS name lookups when dialing.

> The net package now has initial support for the netgo build tag on Windows. When used, the package uses the Go DNS client (as used by Resolver.PreferGo) instead of asking Windows for DNS results. The upstream DNS server it discovers from Windows may not yet be correct with complex system network configurations, however.

I don't know what is the correct course of action here, as we use netgo on every platform and consistency is great.


Thank you for finding the root cause!

In our infrastructure we should find a way to migrate from sort names to FQND for most use cases but for some edge one this will definitely be an issue, as the name are designed to change depending on site.

I feel the same as you about consistency though and I have not come to a good idea (yet) to handle the situation. But I fell like a warning should be added to the 2.39 release as it’s a huge paradigm shift for any major windows deployment.

I don’t have a version of Prometheus running on Linux at the moment, but does the linux version 2.39 respect the `search` “order” in the `etc/resolv.conf` file?
If yes, not respecting the same search order in Windows is a break in consistency, no?

I've the same issue. I downgraded to 2.37.2 for the moment.
2.38 should still work with short names.

We started migrating our targets to FQDN without affecting existing series.
We changed our `relabel_configs` for the `instance` label from 
```
    - source_labels: [__address__]
      regex: '([^:]+):\d+'
      target_label: instance
```
to
```
    - source_labels: [__address__]
      regex: '([^.:]*).*'
      target_label: instance
```

But it will only works for 90% of our targets, the rest will be an issue and we are yet to figure out a correct way to deal with them
Does using the env variable 

> GODEBUG=netdns=cgo

Help here?
Maybe it is GODEBUG=netdns=go; any feedback would be welcome
Sorry I'm unable to test this for the moment

Fix patch:
diff --git a/.promu.yml b/.promu.yml
index 50c0be25752..233295f852a 100644
--- a/.promu.yml
+++ b/.promu.yml
@@ -10,7 +10,13 @@ build:
           path: ./cmd/prometheus
         - name: promtool
           path: ./cmd/promtool
-    flags: -a -tags netgo,builtinassets
+    tags:
+        all:
+            - netgo
+            - builtinassets
+        windows:
+            - builtinassets
+    flags: -a
     ldflags: |
         -X github.com/prometheus/common/version.Version={{.Version}}
         -X github.com/prometheus/common/version.Revision={{.Revision}}
diff --git a/Makefile.common b/Makefile.common
index 7642c4485c1..e358db69c5d 100644
--- a/Makefile.common
+++ b/Makefile.common
@@ -55,7 +55,7 @@ ifneq ($(shell which gotestsum),)
 endif
 endif
 
-PROMU_VERSION ?= 0.13.0
+PROMU_VERSION ?= 0.14.0
 PROMU_URL     := https://github.com/prometheus/promu/releases/download/v$(PROMU_VERSION)/promu-$(PROMU_VERSION).$(GO_BUILD_PLATFORM).tar.gz
 
 SKIP_GOLANGCI_LINT :=


Test patch: