Repository: prometheus/prometheus (language: go)

Issue:
Azure SD with Public IP Address: panic: runtime error: invalid memory address or nil pointer dereference
## Version
Docker image `prom/prometheus`, tags `latest` and `main`, both have the same problem.

## Problem
We're receiving a panic from Azure SD plugin whenever it tries to enumerate our VMs. The problem is caused by an incorrect nil check as seen below:

https://github.com/prometheus/prometheus/blob/579331446a04503b2b1515767e7ceaceab2c6342/discovery/azure/azure.go#L374-L376

## Fix
The current code is checking `ip.PublicIPAddress.PublicIPAddressPropertiesFormat` but is dereferencing `ip.PublicIPAddress.IPAddress`. We should nil check `ip.PublicIPAddress.IPAddress` instead.

## Stacktrace
```
Feb 08 17:14:17 prometheus docker[8302]: panic: runtime error: invalid memory address or nil pointer dereference
Feb 08 17:14:17 prometheus docker[8302]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xeb0ac1]
Feb 08 17:14:17 prometheus docker[8302]: goroutine 776 [running]:
Feb 08 17:14:17 prometheus docker[8302]: github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh.func1(0xd9bebd0f7c815613, {{0xc00194db80, 0x92}, {0xc001950228, 0x11}, {0xc001950240, 0x11}, {0xc0019726f0, 0x21}, {0xc001a62590, ...}, ...})
Feb 08 17:14:17 prometheus docker[8302]:         /app/discovery/azure/azure.go:375 +0x7a1
Feb 08 17:14:17 prometheus docker[8302]: created by github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh
Feb 08 17:14:17 prometheus docker[8302]:         /app/discovery/azure/azure.go:321 +0xd52
```


Discussion / hints:
Thanks for your report. This was theoretically fixed in https://github.com/prometheus/prometheus/pull/5587, and https://github.com/prometheus/prometheus/pull/5587#discussion_r286171167 explains why `ip.PublicIPAddress.PublicIPAddressPropertiesFormat` is used instead. I don't think that check works though, as can be seen in this example (and the bug you found): https://go.dev/play/p/qDzVx0mnoXL.
We've started receiving the same panic with our prometheus implementation today, out the blue with no changes. 

````
Feb 10 14:40:34 xxxxx prometheus: panic: runtime error: invalid memory address or nil pointer dereference
Feb 10 14:40:34 xxxxx prometheus: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xb419b8]
Feb 10 14:40:34 xxxx prometheus: goroutine 9930314 [running]:
Feb 10 14:40:34 xxxx  prometheus: github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh.func1(0xc0cf006aa0, 0xc0e7a0b050, 0xc15b820840, 0xc0d7ff5400, 0x2a1a460, 0xc13b79a300, 0x7, 0xc112e50e60, 0x94, 0xc14b363f80, ...)
Feb 10 14:40:34 xxxx prometheus: /app/discovery/azure/azure.go:339 +0xb38````

prometheus, version 2.13.0 (branch: HEAD, revision: 6ea4252299f542669aca11860abc2192bdc7bede)
  build user:       root@f30bdad2c3fd
  build date:       20191004-11:25:34
  go version:       go1.13.1
  
  
That's interesting because this bug was last seen around 2019, but only now it's showing up. Maybe something changed on Azure's side?
I've opened #10280 to fix this.
This will be fixed in 2.33.2.
We also discovered this today out of the blue. We were running on an older version. Upgrading to the latest version 2.33.1 did not help to solve the issue.

Here are the logs
```
ts=2022-02-11T12:18:17.324Z caller=main.go:896 level=info msg="Server is ready to receive web requests."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xef9b81]

goroutine 580 [running]:
github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh.func1(0x0, {{0xc001eae790, 0xa2}, {0xc000036340, 0x19}, {0xc000036420, 0x19}, {0xc001dbdd10, 0x21}, {0xc001b46d80, ...}, ...})
        /app/discovery/azure/azure.go:365 +0x7a1
created by github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh
        /app/discovery/azure/azure.go:311 +0xd52
```
I guess azure changed something in their API's.
Yeah, unfortunately upgrading won't help as the fix hasn't made a release yet. We're hoping to get 2.33.2, which has the fix, out as soon as possible. In the meantime, you can build the patched version yourself if you wish from the `release-2.33` branch.
A new release is out.
Thanks for your work getting the release out so quickly!
Great effort thank you for the patch release!

Unfortunately, the problem persists:

```
./prometheus-2.33.2.linux-amd64/prometheus --config.file=/<path>/prometheus.yml
ts=2022-02-11T16:59:14.328Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2022-02-11T16:59:14.328Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.2, branch=HEAD, revision=b4714db0a6108aa7ef9c13daa12ac8eb6b5504a1)"
ts=2022-02-11T16:59:14.328Z caller=main.go:517 level=info build_context="(go=go1.17.6, user=root@841e87177e21, date=20220211-14:52:07)"
ts=2022-02-11T16:59:14.328Z caller=main.go:518 level=info host_details="(Linux 5.8.0-1036-azure #38~20.04.1-Ubuntu SMP Thu Jun 17 14:14:18 UTC 2021 x86_64 prometheus (none))"
ts=2022-02-11T16:59:14.328Z caller=main.go:519 level=info fd_limits="(soft=1024, hard=1048576)"
ts=2022-02-11T16:59:14.328Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-02-11T16:59:14.330Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-02-11T16:59:14.330Z caller=main.go:923 level=info msg="Starting TSDB ..."
ts=2022-02-11T16:59:14.330Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/home/<path>/data/lock
ts=2022-02-11T16:59:14.331Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-02-11T16:59:14.333Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-02-11T16:59:14.333Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=244.106µs
ts=2022-02-11T16:59:14.333Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-02-11T16:59:14.333Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=9
ts=2022-02-11T16:59:14.450Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=9
ts=2022-02-11T16:59:14.450Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=9
ts=2022-02-11T16:59:14.451Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=9
ts=2022-02-11T16:59:14.462Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=9
ts=2022-02-11T16:59:14.462Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=9
ts=2022-02-11T16:59:14.462Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=9
ts=2022-02-11T16:59:14.462Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=9
ts=2022-02-11T16:59:14.462Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=9
ts=2022-02-11T16:59:14.462Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=9 maxSegment=9
ts=2022-02-11T16:59:14.462Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=59.402µs wal_replay_duration=129.399411ms total_replay_duration=129.73912ms
ts=2022-02-11T16:59:14.463Z caller=main.go:944 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-02-11T16:59:14.463Z caller=main.go:947 level=info msg="TSDB started"
ts=2022-02-11T16:59:14.463Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/<path>/prometheus.yml
ts=2022-02-11T16:59:14.474Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/<path>/prometheus.yml totalDuration=10.171861ms db_storage=900ns remote_storage=1.301µs web_handler=400ns query_engine=900ns scrape=8.128208ms scrape_sd=205.705µs notify=31.801µs notify_sd=9.8µs rules=1.15833ms
ts=2022-02-11T16:59:14.474Z caller=main.go:896 level=info msg="Server is ready to receive web requests."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0xef9b78]

goroutine 583 [running]:
github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh.func1(0x0, {{0xc00086e4d0, 0xa6}, {0xc00100ab00, 0x1b}, {0xc00100ab40, 0x1b}, {0xc001f04240, 0x21}, {0xc001420682, ...}, ...})
        /app/discovery/azure/azure.go:364 +0x798
created by github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh
        /app/discovery/azure/azure.go:311 +0xd52
```


Maybe some interesting context:

We are running with multiple subscriptions and have all VMs that we monitor in 1 subscription, and only scalesets in the other, plus one additional VM in both subscriptions where prometheus is hosted on a VM.
Prometheus is working perfectly in the subscription where we don't have any VMs deployed. Just for the fun of it, I deployed a VM there to see if I can provoke the same error in that subscription. And as soon as the deployment was finished, that one went down as well.
I did test the fix, and it works on my test setup:

2.33.1:

```
ts=2022-02-11T18:10:46.261Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2022-02-11T18:10:46.261Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.1, branch=HEAD, revision=4e08110891fd5177f9174c4179bc38d789985a13)"
ts=2022-02-11T18:10:46.261Z caller=main.go:517 level=info build_context="(go=go1.17.6, user=root@37fc1ebac798, date=20220202-15:23:18)"
ts=2022-02-11T18:10:46.261Z caller=main.go:518 level=info host_details="(Linux 5.12.15-roidelapluie #1 SMP Thu Jul 8 22:06:28 CEST 2021 x86_64 hydrogen (none))"
ts=2022-02-11T18:10:46.261Z caller=main.go:519 level=info fd_limits="(soft=1024, hard=524288)"
ts=2022-02-11T18:10:46.261Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-02-11T18:10:46.262Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-02-11T18:10:46.263Z caller=main.go:923 level=info msg="Starting TSDB ..."
ts=2022-02-11T18:10:46.266Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-02-11T18:10:46.269Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-02-11T18:10:46.269Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.819µs
ts=2022-02-11T18:10:46.269Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-02-11T18:10:46.271Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=7
ts=2022-02-11T18:10:46.272Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=7
ts=2022-02-11T18:10:46.272Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=7
ts=2022-02-11T18:10:46.273Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=7
ts=2022-02-11T18:10:46.274Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=7
ts=2022-02-11T18:10:46.275Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=7
ts=2022-02-11T18:10:46.275Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=7
ts=2022-02-11T18:10:46.276Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=7
ts=2022-02-11T18:10:46.276Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=123.759µs wal_replay_duration=6.422316ms total_replay_duration=6.614661ms
ts=2022-02-11T18:10:46.277Z caller=main.go:944 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-02-11T18:10:46.278Z caller=main.go:947 level=info msg="TSDB started"
ts=2022-02-11T18:10:46.278Z caller=main.go:1128 level=info msg="Loading configuration file" filename=prometheus.yml
ts=2022-02-11T18:10:46.279Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=prometheus.yml totalDuration=1.179561ms db_storage=3.283µs remote_storage=4.33µs web_handler=1.886µs query_engine=2.864µs scrape=420.448µs scrape_sd=113.214µs notify=3.283µs notify_sd=4.749µs rules=3.422µs
ts=2022-02-11T18:10:46.279Z caller=main.go:896 level=info msg="Server is ready to receive web requests."
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xef9b81]

goroutine 631 [running]:
github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh.func1(0x1c, {{0xc000b28100, 0x78}, {0xc0001ec408, 0x4}, {0xc00036ee74, 0x4}, {0xc000322690, 0x21}, {0xc0001ec420, ...}, ...})
	/app/discovery/azure/azure.go:365 +0x7a1
created by github.com/prometheus/prometheus/discovery/azure.(*Discovery).refresh
	/app/discovery/azure/azure.go:311 +0xd52
```

2.33.2:

```ts=2022-02-11T18:10:53.329Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d
ts=2022-02-11T18:10:53.329Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.2, branch=HEAD, revision=b4714db0a6108aa7ef9c13daa12ac8eb6b5504a1)"
ts=2022-02-11T18:10:53.329Z caller=main.go:517 level=info build_context="(go=go1.17.6, user=root@841e87177e21, date=20220211-14:52:07)"
ts=2022-02-11T18:10:53.329Z caller=main.go:518 level=info host_details="(Linux 5.12.15-roidelapluie #1 SMP Thu Jul 8 22:06:28 CEST 2021 x86_64 hydrogen (none))"
ts=2022-02-11T18:10:53.329Z caller=main.go:519 level=info fd_limits="(soft=1024, hard=524288)"
ts=2022-02-11T18:10:53.329Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2022-02-11T18:10:53.333Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2022-02-11T18:10:53.334Z caller=main.go:923 level=info msg="Starting TSDB ..."
ts=2022-02-11T18:10:53.335Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/home/roidelapluie/dev/prometheus/data/lock
ts=2022-02-11T18:10:53.335Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false
ts=2022-02-11T18:10:53.342Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2022-02-11T18:10:53.342Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.819µs
ts=2022-02-11T18:10:53.342Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2022-02-11T18:10:53.344Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=8
ts=2022-02-11T18:10:53.344Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=8
ts=2022-02-11T18:10:53.345Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=8
ts=2022-02-11T18:10:53.345Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=8
ts=2022-02-11T18:10:53.346Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=8
ts=2022-02-11T18:10:53.346Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=8
ts=2022-02-11T18:10:53.347Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=8
ts=2022-02-11T18:10:53.347Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=7 maxSegment=8
ts=2022-02-11T18:10:53.347Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=8 maxSegment=8
ts=2022-02-11T18:10:53.348Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=114.68µs wal_replay_duration=5.742265ms total_replay_duration=5.904858ms
ts=2022-02-11T18:10:53.350Z caller=main.go:944 level=info fs_type=EXT4_SUPER_MAGIC
ts=2022-02-11T18:10:53.350Z caller=main.go:947 level=info msg="TSDB started"
ts=2022-02-11T18:10:53.350Z caller=main.go:1128 level=info msg="Loading configuration file" filename=prometheus.yml
ts=2022-02-11T18:10:53.351Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=prometheus.yml totalDuration=1.052099ms db_storage=2.934µs remote_storage=4.261µs web_handler=2.445µs query_engine=2.864µs scrape=372.677µs scrape_sd=105.601µs notify=3.283µs notify_sd=4.261µs rules=3.841µs
ts=2022-02-11T18:10:53.351Z caller=main.go:896 level=info msg="Server is ready to receive web requests."
ts=2022-02-11T18:11:00.832Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..."
ts=2022-02-11T18:11:00.832Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..."
ts=2022-02-11T18:11:00.832Z caller=main.go:812 level=info msg="Stopping notify discovery manager..."
ts=2022-02-11T18:11:00.832Z caller=main.go:834 level=info msg="Stopping scrape manager..."
ts=2022-02-11T18:11:00.832Z caller=main.go:794 level=info msg="Scrape discovery manager stopped"
ts=2022-02-11T18:11:00.832Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..."
ts=2022-02-11T18:11:00.832Z caller=main.go:808 level=info msg="Notify discovery manager stopped"
ts=2022-02-11T18:11:00.832Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped"
ts=2022-02-11T18:11:00.832Z caller=main.go:828 level=info msg="Scrape manager stopped"
ts=2022-02-11T18:11:00.838Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..."
ts=2022-02-11T18:11:00.839Z caller=main.go:1054 level=info msg="Notifier manager stopped"
ts=2022-02-11T18:11:00.839Z caller=main.go:1066 level=info msg="See you next time!"
```
Mmmh the last panic you have sent is about line 364.
Pulled the fix, and still having the error. The line with the error is now `364`. Seems like `ip` itself is `nil`.

https://github.com/prometheus/prometheus/blob/b4714db0a6108aa7ef9c13daa12ac8eb6b5504a1/discovery/azure/azure.go#L363-L366
Interesting. So I guess we need to `nil`-check the `ip` now?

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index bc151740cc6..1750a780f75 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,6 +1,10 @@
+## 2.33.3 / 2022-02-11
+
+* [BUGFIX] Azure SD: Fix a regression when public IP Address isn't set. #10289
+
 ## 2.33.2 / 2022-02-11
 
-* [BUGFIX] Azure SD: Fix panic when IP Address isn't set. #10280
+* [BUGFIX] Azure SD: Fix panic when public IP Address isn't set. #10280
 * [BUGFIX] Remote-write: Fix deadlock when stopping a shard. #10279
 
 ## 2.33.1 / 2022-02-02
diff --git a/VERSION b/VERSION
index 0b82b8d70e2..dc148c42f45 100644
--- a/VERSION
+++ b/VERSION
@@ -1,1 +1,1 @@
-2.33.2
+2.33.3
diff --git a/discovery/azure/azure.go b/discovery/azure/azure.go
index 4a70416bba7..d668a5a2bc5 100644
--- a/discovery/azure/azure.go
+++ b/discovery/azure/azure.go
@@ -361,7 +361,9 @@ func (d *Discovery) refresh(ctx context.Context) ([]*targetgroup.Group, error) {
 
 				if *networkInterface.Primary {
 					for _, ip := range *networkInterface.IPConfigurations {
-						if ip.PublicIPAddress != nil && ip.PublicIPAddress.IPAddress != nil {
+						// IPAddress is a field defined in PublicIPAddressPropertiesFormat,
+						// therefore we need to validate that both are not nil.
+						if ip.PublicIPAddress != nil && ip.PublicIPAddress.PublicIPAddressPropertiesFormat != nil && ip.PublicIPAddress.IPAddress != nil {
 							labels[azureLabelMachinePublicIP] = model.LabelValue(*ip.PublicIPAddress.IPAddress)
 						}
 						if ip.PrivateIPAddress != nil {


Test patch: