Repository: prometheus-community/windows_exporter

Issue #446: wmi.(*SWbemServices).Query / syscall.Syscall9 sometimes gets stuck indefinitely
Created: 2019-12-10T12:37:43Z
With 0.8.3 we've run into a situation where thousands of goroutines were stuck on:
goroutine 280 [semacquire, 279 minutes]:
sync.runtime_SemacquireMutex(0xc00010fd3c, 0x0)
	C:/go/src/runtime/sema.go:71 +0x44
sync.(*Mutex).Lock(0xc00010fd38)
	C:/go/src/sync/mutex.go:134 +0x110
github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi.(*SWbemServices).Query(0xc00010fd10, 0xc000384060, 0x22, 0x9269a0, 0xc0004ac080, 0x0, 0x0, 0x0, 0x0, 0x0)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi/swbemservices.go:142 +0x52
github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi.Query(0xc000384060, 0x22, 0x9269a0, 0xc0004ac080, 0x0, 0x0, 0x0, 0x3c515e8, 0x1b02)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi/wmi.go:78 +0x90
github.com/martinlindhe/wmi_exporter/collector.(*CSCollector).collect(0xc00010a540, 0xc0001d03c0, 0x4cb895, 0x0, 0xc000026070)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/collector/cs.go:63 +0x91
github.com/martinlindhe/wmi_exporter/collector.(*CSCollector).Collect(0xc00010a540, 0xc0005d61e0, 0xc0001d03c0, 0x50, 0x50)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/collector/cs.go:46 +0x43
main.execute(0xc0000a6008, 0x2, 0xae7ba0, 0xc00010a540, 0xc0005d61e0, 0xc0001d03c0, 0x6a3e83)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/exporter.go:205 +0x8f
main.WmiCollector.Collect.func2(0xc000669bd4, 0xc0005d61e0, 0xc0001d03c0, 0xc000669be0, 0xc000669be8, 0xc0005184e0, 0xc0000a6008, 0x2, 0xae7ba0, 0xc00010a540)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/exporter.go:135 +0x9e
created by main.WmiCollector.Collect
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/exporter.go:133 +0x4db

The offending goroutine which holds the lock is this:
goroutine 280 [semacquire, 279 minutes]:
sync.runtime_SemacquireMutex(0xc00010fd3c, 0x0)
	C:/go/src/runtime/sema.go:71 +0x44
sync.(*Mutex).Lock(0xc00010fd38)
	C:/go/src/sync/mutex.go:134 +0x110
github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi.(*SWbemServices).Query(0xc00010fd10, 0xc000384060, 0x22, 0x9269a0, 0xc0004ac080, 0x0, 0x0, 0x0, 0x0, 0x0)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi/swbemservices.go:142 +0x52
github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi.Query(0xc000384060, 0x22, 0x9269a0, 0xc0004ac080, 0x0, 0x0, 0x0, 0x3c515e8, 0x1b02)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/vendor/github.com/StackExchange/wmi/wmi.go:78 +0x90
github.com/martinlindhe/wmi_exporter/collector.(*CSCollector).collect(0xc00010a540, 0xc0001d03c0, 0x4cb895, 0x0, 0xc000026070)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/collector/cs.go:63 +0x91
github.com/martinlindhe/wmi_exporter/collector.(*CSCollector).Collect(0xc00010a540, 0xc0005d61e0, 0xc0001d03c0, 0x50, 0x50)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/collector/cs.go:46 +0x43
main.execute(0xc0000a6008, 0x2, 0xae7ba0, 0xc00010a540, 0xc0005d61e0, 0xc0001d03c0, 0x6a3e83)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/exporter.go:205 +0x8f
main.WmiCollector.Collect.func2(0xc000669bd4, 0xc0005d61e0, 0xc0001d03c0, 0xc000669be0, 0xc000669be8, 0xc0005184e0, 0xc0000a6008, 0x2, 0xae7ba0, 0xc00010a540)
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/exporter.go:135 +0x9e
created by main.WmiCollector.Collect
	c:/gopath/src/github.com/martinlindhe/wmi_exporter/exporter.go:133 +0x4db

There are two problems here:

goroutines shouldn't indefinitely try to acquire that mutex
syscall.Syscall9 never returns even though it probably should

I don't have much Windows knowledge but this leads to a "memory leak" on wmi_exporter. Not sure how to reproduce this. 0.9.0 still uses wmi for this collector so I guess it should still be reproducible on 0.9.0. Submitting for posterity and I will update if anything comes up.
UPDATE:
What's interesting is that Get-WmiObject -class Win32_ComputerSystem works as expected from the PowerShell. I guess this rules out the WMI interface being unavailable.

Duplicate issue #418: V0.9.0 memory leak
Created: 2019-10-17T07:34:44Z
I am using V0.9.0 as a service, running on default configuration. In a remote server it was using 3GB RAM. Now i tried that version on my local computer it has been running for 2 hours and it is using 61MB RAM. Do we have a solution for this?

Closing comment (2020-01-13T19:29:00Z): Thanks @AntonSmolkov! Could you also upload the goroutine dump? You can find it here: <host>:9182/debug/pprof/goroutine
It looks like it could be related to #446