Repository: netbirdio/netbird

Issue:
Netbird service 0.36.x and newer  freezes on Windows 
**Describe the problem**

After updating to 0.37.0 (and 0.37.1 as well) this message appeared only once right after connection, but once the connection to the DNS server within netbird universe is estabilished, the message disappeared.

However after putting laptop to sleep and turning on again, this message appears around every 5seconds and the netbird service is fozen thus cannot generate the debug logs.

The service had to be force-killed using ```taskkill /pid PID /f``` (restart using netbird service restart did not finished) so the attached debug log is from the time after service restart.

SelfHosted

**NetBird version**

0.37 client

**NetBird status -dA output:**

could not be generated

**Do you face any (non-mobile) client issues?**

could not be generated

**Screenshots**

![Image](https://github.com/user-attachments/assets/8c36da30-99fc-4f95-a299-65e039f0244b)

![Image](https://github.com/user-attachments/assets/9bb6faaf-65cb-487d-898f-1ced5f3e2c1b)

**Additional context**

Add any other context about the problem here.

Current stat.json content: ```{"dns_state":null,"route_state":{"refCountMap":{},"idMap":{}},"routeselector_state":null}```

[netbird.debug.4066958341.zip](https://github.com/user-attachments/files/18916189/netbird.debug.4066958341.zip)

Discussion:
@Oriann: Facing same problem

[netbird.debug.2399991544.zip](https://github.com/user-attachments/files/18929615/netbird.debug.2399991544.zip)@boopzz: I believe we have the same problem. Previously we had matched domains working, upgraded from v0.36.7 to v0.37.1. At first everything seemed to be working and now the DNS services have an `i/o timeout" error and we are unable to resolve. However we cant route any traffic beyond the peer. We can connect to the peer fine and the peer has all the correct connectivity but the peers wont route traffic for some reason.

Are you able to route any traffic at all?

EDIT: Mine was a separate issue to do with the firewall implementation@tomashora: Reported by multiple users - happens regullary. Attached debug bundle after force-killing the service.
[netbird.debug.2157308570.zip](https://github.com/user-attachments/files/19026938/netbird.debug.2157308570.zip)@iamjustcurious: Unfortunately, we are also facing the problem@cleveHEX: Colleagues that did not update their client do not have this issue. Person on 0.34.x does not face this issue, I face it on 0.36.x and 0.37.x.@tomashora: Still happening also on 0.37.2

[netbird.debug.3307782300.zip](https://github.com/user-attachments/files/19152209/netbird.debug.3307782300.zip)@tomashora: Could this be related? Noticed this after enabling notifications... 

![Image](https://github.com/user-attachments/assets/70b0f456-32c7-4362-b6e8-27d7ba1f4fd9)

[netbird.debug.2640559806.zip](https://github.com/user-attachments/files/19158011/netbird.debug.2640559806.zip)

The client.log says:

```
2025-03-10T08:44:39+01:00 PANC client/internal/connect.go:118: Panic occurred: close of closed channel, stack trace: goroutine 1228322 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.23.6/x64/src/runtime/debug/stack.go:26 +0x5e
github.com/netbirdio/netbird/client/internal.(*ConnectClient).run.func1()
	/home/runner/work/netbird/netbird/client/internal/connect.go:118 +0x7c
panic({0x158ad40?, 0x1943fd0?})
	/opt/hostedtoolcache/go/1.23.6/x64/src/runtime/panic.go:785 +0x132
github.com/netbirdio/netbird/client/firewall/uspfilter/conntrack.(*UDPTracker).Close(0xc001e5dd60)
	/home/runner/work/netbird/netbird/client/firewall/uspfilter/conntrack/udp.go:135 +0x48
github.com/netbirdio/netbird/client/firewall/uspfilter.(*Manager).Close(0xc001010d00, 0x0?)
	/home/runner/work/netbird/netbird/client/firewall/uspfilter/allow_netbird_windows.go:32 +0xed
github.com/netbirdio/netbird/client/internal.(*Engine).close(0xc0007a2488)
	/home/runner/work/netbird/netbird/client/internal/engine.go:1365 +0x223
github.com/netbirdio/netbird/client/internal.(*Engine).Stop(0xc0007a2488)
	/home/runner/work/netbird/netbird/client/internal/engine.go:301 +0x2ea
github.com/netbirdio/netbird/client/internal.(*ConnectClient).run.func3()
	/home/runner/work/netbird/netbird/client/internal/connect.go:295 +0x14ed
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1()
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:18 +0x13
github.com/cenkalti/backoff/v4.doRetryNotify[...](0xc003e3f9a8?, {0x194e398, 0xc001ca2720}, 0x0, {0x0, 0x0?})
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:88 +0x11c
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer(0xc003ab8280?, {0x194e398?, 0xc001ca2720?}, 0x7c1a4b4aea9c106b?, {0x0?, 0x0?})
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:61 +0x56
github.com/cenkalti/backoff/v4.RetryNotify(...)
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:49
github.com/cenkalti/backoff/v4.Retry(...)
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:38
github.com/netbirdio/netbird/client/internal.(*ConnectClient).run(0xc0021b0380, {{0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0}, {0x0, 0x0, 0x0}, ...}, ...)
	/home/runner/work/netbird/netbird/client/internal/connect.go:315 +0x475
github.com/netbirdio/netbird/client/internal.(*ConnectClient).Run(...)
	/home/runner/work/netbird/netbird/client/internal/connect.go:65
github.com/netbirdio/netbird/client/server.(*Server).connectWithRetryRuns.func2()
	/home/runner/work/netbird/netbird/client/server/server.go:196 +0x185
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer.Operation.withEmptyData.func1()
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:18 +0x13
github.com/cenkalti/backoff/v4.doRetryNotify[...](0xc003e3feb0?, {0x299768d51f8, 0xc004480240}, 0x0, {0x0, 0x0?})
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:88 +0x11c
github.com/cenkalti/backoff/v4.RetryNotifyWithTimer(0x1957ae8?, {0x299768d51f8?, 0xc004480240?}, 0x1021145?, {0x0?, 0x0?})
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:61 +0x56
github.com/cenkalti/backoff/v4.RetryNotify(...)
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:49
github.com/cenkalti/backoff/v4.Retry(...)
	/home/runner/go/pkg/mod/github.com/cenkalti/backoff/v4@v4.3.0/retry.go:38
github.com/netbirdio/netbird/client/server.(*Server).connectWithRetryRuns(0xc0004a4488, {0x1957ae8, 0xc00058b440}, 0xc00517c360, 0xc00048c780, 0xc0045a82a0)
	/home/runner/work/netbird/netbird/client/server/server.go:214 +0x194
created by github.com/netbirdio/netbird/client/server.(*Server).Up in goroutine 1228321
	/home/runner/work/netbird/netbird/client/server/server.go:632 +0x6d6

```
@cleveHEX: @pappz I encounter similar behavior on 0.38.0

[netbird.debug.4282123131.zip](https://github.com/user-attachments/files/19181842/netbird.debug.4282123131.zip)@mlsmaycon: > [@pappz](https://github.com/pappz) I encounter similar behavior on 0.38.0
> 
> [netbird.debug.4282123131.zip](https://github.com/user-attachments/files/19181842/netbird.debug.4282123131.zip)

@cleveHEX, can you confirm if you just upgraded the client or if it was a fresh install? Also, what action did you take to restart the client?

Lastly, by any chance, when starting the client, do you click on Networks before Connect?@Oriann: Facing same problem on 0.38.0, everytime I only update the client, restart is made via installer.

[netbird.debug.3569783032.zip](https://github.com/user-attachments/files/19207229/netbird.debug.3569783032.zip)@cleveHEX: @mlsmaycon I have upgraded the client, not a fresh install (the Netbird was not stuck before the upgrade). I kill `netbird.exe`, it restarts automatically and then it starts working, unloading via Windows Services does not work, the service gets stuck in stopping state infinitely.

I don't think so, I don't use this feature at all.@mlsmaycon: @Oriann @cleveHEX are you guys able to reproduce the issue by downgrading it to 0.37.2 and upgrading again? You can find the installers here:

037.2 -> https://github.com/netbirdio/netbird/releases/download/v0.37.2/netbird_installer_0.37.2_windows_amd64.exe
0.38.0 -> https://github.com/netbirdio/netbird/releases/download/v0.38.0/netbird_installer_0.38.0_windows_amd64.exe@Oriann: @mlsmaycon Yes still the same issue even after this test.

Steps I made:
1.Netbird running > ran installer, downgraded to 0.37.2
2. Netbird shows warning as OP described
3. Made debug bundle for 0.37.2
4.Netbird running > ran installer upgraded back to 0.38.0
5.Netbird shows the same warning as on previous version.
6. Made debug bundle for 0.38.0

[netbird.debug.1729080692_0.38.0.zip](https://github.com/user-attachments/files/19207798/netbird.debug.1729080692_0.38.0.zip)
[netbird.debug.4062680248_0.37.2.zip](https://github.com/user-attachments/files/19207800/netbird.debug.4062680248_0.37.2.zip)@mlsmaycon: @Oriann, it's great that you can reproduce it; it will help us understand the issue better. 

Can you enable trace logs before repeating the downgrade/upgrade steps? You can do that with:

```shell
[Environment]::SetEnvironmentVariable("NB_LOG_LEVEL", "debug", "Machine")
```

Then please share the logs again. 

Also, the warning you are referencing to is the Panicking or the DNS?@Oriann: @mlsmaycon ok i have enabled trace logs and attaching debug bundles

I am referencing to DNS issue, I have not succesfully reproduced panicking even when trying hibernate or sleep mode. But I am on PC so things difer from laptop.

[netbird.debug.113234266_0.37.2.zip](https://github.com/user-attachments/files/19210173/netbird.debug.113234266_0.37.2.zip)
[netbird.debug.126377177_0.38.0.zip](https://github.com/user-attachments/files/19210172/netbird.debug.126377177_0.38.0.zip)@mlsmaycon: Thanks @Oriann . Just to confirm, during the upgrade your client received the alert and also freezes the GUI client? @Oriann: @mlsmaycon I only recieve the alert, GUI wont freeze, I have tested it right now again just in case@mlsmaycon: Got it, thanks @Oriann, that is the expected behavior for private DNS since when you start the agent, you won't have connectivity to that particular private server until you connect to the routing peer that routes traffic to it. 

You can also confirm that with the `netbird status -d` command. See output from your peer:
```
Nameservers: 
  [192.168.20.1:53] for [anon-cxxx2.domain, anon-gUfys.domain] is Available
  [10.0.0.1:53] for [.] is Available
  [172.16.3.200:53, 172.16.3.202:53] for [ad.anon-zffBq.domain] is Available
```

To disable the debug logs you can run:
```
[Environment]::SetEnvironmentVariable("NB_LOG_LEVEL", "info", "Machine")
```

@cleveHEX As we couldn't reproduce the issue, it would be great if you could run the tests we described before with the debug logs. 

In the meantime, I will close this issue and we can either open a new one or reopen this one to proceed.@cleveHEX: @mlsmaycon for me it's completely unrelated to updates/upgrades, my Netbird service sometimes dies 2 times a day without any action from my side, maybe the computer puts itself into sleep mode. I have set the debug mode and next time I encounter this I will attach the log.@mlsmaycon: Got it. Then it is better to open a new issue when you reproduce to keep issues in different threads.@tomashora: @mlsmaycon created new issue with debug bundle with log level debug https://github.com/netbirdio/netbird/issues/3503

Solution patch:
diff --git a/client/firewall/uspfilter/conntrack/icmp.go b/client/firewall/uspfilter/conntrack/icmp.go
index 25cd9e87..a8cb0156 100644
--- a/client/firewall/uspfilter/conntrack/icmp.go
+++ b/client/firewall/uspfilter/conntrack/icmp.go
@@ -1,6 +1,7 @@
 package conntrack
 
 import (
+	"context"
 	"net"
 	"sync"
 	"time"
@@ -39,8 +40,8 @@ type ICMPTracker struct {
 	connections   map[ICMPConnKey]*ICMPConnTrack
 	timeout       time.Duration
 	cleanupTicker *time.Ticker
+	tickerCancel  context.CancelFunc
 	mutex         sync.RWMutex
-	done          chan struct{}
 	ipPool        *PreallocatedIPs
 }
 
@@ -50,16 +51,18 @@ func NewICMPTracker(timeout time.Duration, logger *nblog.Logger) *ICMPTracker {
 		timeout = DefaultICMPTimeout
 	}
 
+	ctx, cancel := context.WithCancel(context.Background())
+
 	tracker := &ICMPTracker{
 		logger:        logger,
 		connections:   make(map[ICMPConnKey]*ICMPConnTrack),
 		timeout:       timeout,
 		cleanupTicker: time.NewTicker(ICMPCleanupInterval),
-		done:          make(chan struct{}),
+		tickerCancel:  cancel,
 		ipPool:        NewPreallocatedIPs(),
 	}
 
-	go tracker.cleanupRoutine()
+	go tracker.cleanupRoutine(ctx)
 	return tracker
 }
 
@@ -119,12 +122,14 @@ func (t *ICMPTracker) IsValidInbound(srcIP net.IP, dstIP net.IP, id uint16, seq
 		conn.Sequence == seq
 }
 
-func (t *ICMPTracker) cleanupRoutine() {
+func (t *ICMPTracker) cleanupRoutine(ctx context.Context) {
+	defer t.tickerCancel()
+
 	for {
 		select {
 		case <-t.cleanupTicker.C:
 			t.cleanup()
-		case <-t.done:
+		case <-ctx.Done():
 			return
 		}
 	}
@@ -146,8 +151,7 @@ func (t *ICMPTracker) cleanup() {
 
 // Close stops the cleanup routine and releases resources
 func (t *ICMPTracker) Close() {
-	t.cleanupTicker.Stop()
-	close(t.done)
+	t.tickerCancel()
 
 	t.mutex.Lock()
 	for _, conn := range t.connections {
diff --git a/client/firewall/uspfilter/conntrack/tcp.go b/client/firewall/uspfilter/conntrack/tcp.go
index 7c12e8ad..1b5cbae9 100644
--- a/client/firewall/uspfilter/conntrack/tcp.go
+++ b/client/firewall/uspfilter/conntrack/tcp.go
@@ -3,6 +3,7 @@ package conntrack
 // TODO: Send RST packets for invalid/timed-out connections
 
 import (
+	"context"
 	"net"
 	"sync"
 	"sync/atomic"
@@ -85,23 +86,26 @@ type TCPTracker struct {
 	connections   map[ConnKey]*TCPConnTrack
 	mutex         sync.RWMutex
 	cleanupTicker *time.Ticker
-	done          chan struct{}
+	tickerCancel  context.CancelFunc
 	timeout       time.Duration
 	ipPool        *PreallocatedIPs
 }
 
 // NewTCPTracker creates a new TCP connection tracker
 func NewTCPTracker(timeout time.Duration, logger *nblog.Logger) *TCPTracker {
+
+	ctx, cancel := context.WithCancel(context.Background())
+
 	tracker := &TCPTracker{
 		logger:        logger,
 		connections:   make(map[ConnKey]*TCPConnTrack),
 		cleanupTicker: time.NewTicker(TCPCleanupInterval),
-		done:          make(chan struct{}),
+		tickerCancel:  cancel,
 		timeout:       timeout,
 		ipPool:        NewPreallocatedIPs(),
 	}
 
-	go tracker.cleanupRoutine()
+	go tracker.cleanupRoutine(ctx)
 	return tracker
 }
 
@@ -315,12 +319,14 @@ func (t *TCPTracker) isValidStateForFlags(state TCPState, flags uint8) bool {
 	return false
 }
 
-func (t *TCPTracker) cleanupRoutine() {
+func (t *TCPTracker) cleanupRoutine(ctx context.Context) {
+	defer t.cleanupTicker.Stop()
+
 	for {
 		select {
 		case <-t.cleanupTicker.C:
 			t.cleanup()
-		case <-t.done:
+		case <-ctx.Done():
 			return
 		}
 	}
@@ -355,8 +361,7 @@ func (t *TCPTracker) cleanup() {
 
 // Close stops the cleanup routine and releases resources
 func (t *TCPTracker) Close() {
-	t.cleanupTicker.Stop()
-	close(t.done)
+	t.tickerCancel()
 
 	// Clean up all remaining IPs
 	t.mutex.Lock()
diff --git a/client/firewall/uspfilter/conntrack/udp.go b/client/firewall/uspfilter/conntrack/udp.go
index e73465e3..073eb0fa 100644
--- a/client/firewall/uspfilter/conntrack/udp.go
+++ b/client/firewall/uspfilter/conntrack/udp.go
@@ -1,6 +1,7 @@
 package conntrack
 
 import (
+	"context"
 	"net"
 	"sync"
 	"time"
@@ -26,8 +27,8 @@ type UDPTracker struct {
 	connections   map[ConnKey]*UDPConnTrack
 	timeout       time.Duration
 	cleanupTicker *time.Ticker
+	tickerCancel  context.CancelFunc
 	mutex         sync.RWMutex
-	done          chan struct{}
 	ipPool        *PreallocatedIPs
 }
 
@@ -37,16 +38,18 @@ func NewUDPTracker(timeout time.Duration, logger *nblog.Logger) *UDPTracker {
 		timeout = DefaultUDPTimeout
 	}
 
+	ctx, cancel := context.WithCancel(context.Background())
+
 	tracker := &UDPTracker{
 		logger:        logger,
 		connections:   make(map[ConnKey]*UDPConnTrack),
 		timeout:       timeout,
 		cleanupTicker: time.NewTicker(UDPCleanupInterval),
-		done:          make(chan struct{}),
+		tickerCancel:  cancel,
 		ipPool:        NewPreallocatedIPs(),
 	}
 
-	go tracker.cleanupRoutine()
+	go tracker.cleanupRoutine(ctx)
 	return tracker
 }
 
@@ -103,12 +106,14 @@ func (t *UDPTracker) IsValidInbound(srcIP net.IP, dstIP net.IP, srcPort uint16,
 }
 
 // cleanupRoutine periodically removes stale connections
-func (t *UDPTracker) cleanupRoutine() {
+func (t *UDPTracker) cleanupRoutine(ctx context.Context) {
+	defer t.cleanupTicker.Stop()
+
 	for {
 		select {
 		case <-t.cleanupTicker.C:
 			t.cleanup()
-		case <-t.done:
+		case <-ctx.Done():
 			return
 		}
 	}
@@ -131,8 +136,7 @@ func (t *UDPTracker) cleanup() {
 
 // Close stops the cleanup routine and releases resources
 func (t *UDPTracker) Close() {
-	t.cleanupTicker.Stop()
-	close(t.done)
+	t.tickerCancel()
 
 	t.mutex.Lock()
 	for _, conn := range t.connections {
diff --git a/client/server/panic_windows.go b/client/server/panic_windows.go
index 1d4ba4b7..c5e73be7 100644
--- a/client/server/panic_windows.go
+++ b/client/server/panic_windows.go
@@ -3,7 +3,7 @@ package server
 import (
 	"fmt"
 	"os"
-	"path/filepath"
+	"path"
 	"syscall"
 
 	log "github.com/sirupsen/logrus"
@@ -12,7 +12,6 @@ import (
 )
 
 const (
-	windowsPanicLogEnvVar = "NB_WINDOWS_PANIC_LOG"
 	// STD_ERROR_HANDLE ((DWORD)-12) = 4294967284
 	stdErrorHandle = ^uintptr(11)
 )
@@ -25,13 +24,10 @@ var (
 )
 
 func handlePanicLog() error {
-	logPath := os.Getenv(windowsPanicLogEnvVar)
-	if logPath == "" {
-		return nil
-	}
+	// TODO: move this to a central location
+	logDir := path.Join(os.Getenv("PROGRAMDATA"), "Netbird")
+	logPath := path.Join(logDir, "netbird.err")
 
-	// Ensure the directory exists
-	logDir := filepath.Dir(logPath)
 	if err := os.MkdirAll(logDir, 0750); err != nil {
 		return fmt.Errorf("create panic log directory: %w", err)
 	}
@@ -39,13 +35,11 @@ func handlePanicLog() error {
 		return fmt.Errorf("enforce permission on panic log file: %w", err)
 	}
 
-	// Open log file with append mode
 	f, err := os.OpenFile(logPath, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
 	if err != nil {
 		return fmt.Errorf("open panic log file: %w", err)
 	}
 
-	// Redirect stderr to the file
 	if err = redirectStderr(f); err != nil {
 		if closeErr := f.Close(); closeErr != nil {
 			log.Warnf("failed to close file after redirect error: %v", closeErr)
@@ -59,7 +53,6 @@ func handlePanicLog() error {
 
 // redirectStderr redirects stderr to the provided file
 func redirectStderr(f *os.File) error {
-	// Get the current process's stderr handle
 	if err := setStdHandle(f); err != nil {
 		return fmt.Errorf("failed to set stderr handle: %w", err)
 	}
diff --git a/infrastructure_files/observability/grafana/dashboards/signal.json b/infrastructure_files/observability/grafana/dashboards/signal.json
index 5e36f6ce..0dc1b7aa 100644
--- a/infrastructure_files/observability/grafana/dashboards/signal.json
+++ b/infrastructure_files/observability/grafana/dashboards/signal.json
@@ -757,7 +757,7 @@
         },
         "id": 18,
         "panels": [],
-        "title": "Core metrics / registerations",
+        "title": "Core metrics / registrations",
         "type": "row"
       },
       {
@@ -1874,4 +1874,4 @@
     "uid": "cebyq0fs0m-v001",
     "version": 15,
     "weekStart": ""
-  }
\ No newline at end of file
+  }



Test patch:
diff --git a/client/firewall/uspfilter/conntrack/udp_test.go b/client/firewall/uspfilter/conntrack/udp_test.go
index fa83ee35..40e73cbe 100644
--- a/client/firewall/uspfilter/conntrack/udp_test.go
+++ b/client/firewall/uspfilter/conntrack/udp_test.go
@@ -1,6 +1,7 @@
 package conntrack
 
 import (
+	"context"
 	"net"
 	"testing"
 	"time"
@@ -34,7 +35,7 @@ func TestNewUDPTracker(t *testing.T) {
 			assert.Equal(t, tt.wantTimeout, tracker.timeout)
 			assert.NotNil(t, tracker.connections)
 			assert.NotNil(t, tracker.cleanupTicker)
-			assert.NotNil(t, tracker.done)
+			assert.NotNil(t, tracker.tickerCancel)
 		})
 	}
 }
@@ -154,18 +155,21 @@ func TestUDPTracker_Cleanup(t *testing.T) {
 	timeout := 50 * time.Millisecond
 	cleanupInterval := 25 * time.Millisecond
 
+	ctx, tickerCancel := context.WithCancel(context.Background())
+	defer tickerCancel()
+
 	// Create tracker with custom cleanup interval
 	tracker := &UDPTracker{
 		connections:   make(map[ConnKey]*UDPConnTrack),
 		timeout:       timeout,
 		cleanupTicker: time.NewTicker(cleanupInterval),
-		done:          make(chan struct{}),
+		tickerCancel:  tickerCancel,
 		ipPool:        NewPreallocatedIPs(),
 		logger:        logger,
 	}
 
 	// Start cleanup routine
-	go tracker.cleanupRoutine()
+	go tracker.cleanupRoutine(ctx)
 
 	// Add some connections
 	connections := []struct {