Repository: elastic/beats (Go)

Issue:
Flaky Test [Build&Test / metricbeat-windows-windows-2019 / TestGetServiceStates – service]
## Flaky Test

* **Test Name:** Build&Test / metricbeat-windows-windows-2019 / TestGetServiceStates – service
* **Artifact Link:** https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats%2FPR-21853/detail/PR-21853/29/


### Error details
```
Failed
```


### Stack Trace
```
    service_status_test.go:29: for testing the flaky test reporting for an existing flaky test
```


Metricbeat windows service metrics stops sending documents when a single service fails
- Version:  8.10.4 
- Operating System:
```
runtime:
    arch: amd64
    os: windows
    osinfo:
        family: windows
        major: 6
        minor: 3
        patch: 0
        type: windows
        version: "6.3"
```

- Steps to Reproduce: 
No clear steps to reproduce, more info on this later.

Multiple instances of elastic-agent installations are failing to send the windows.service metric set for the windows integration. The system integration continues to send data without issues. The problem happens at random and it is resolved by restarting the elastic agent.
~The issue happens in different versions of 8.x for elastic-agent and it hasn't confirmed as occurring on the latest version (as the user who has experienced this has not upgraded to latest version yet).~ The issue so far has only been seen on 8.10.4

The error reported by metricbeat is the following:

```
{"log.level":"error","@timestamp":"2024-07-29T20:49:33.157Z","message":"Error fetching data for metricset windows.service: OpenProcess failed for pid=1724: The parameter is incorrect.","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"windows/metrics-default","type":"windows/metrics"},"log":{"source":"windows/metrics-default"},"log.origin":{"file.line":256,"file.name":"module/wrapper.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

```

So far the error indicates a problem only with one particular windows service, however, all other services being monitored by metricbeat can't continue to be monitored because this particular service getting in an unexpected state causes the entire metricbeat windows service metricset to stop reporting for any service. 

Because this happens at random we are unable to setup debug logging to catch the failure and the logger for this function is not providing any more info. 

We need to address 2 items with this issue:

1. The windows service monitoring stops sending stats for ANY service once a single service gets into a weird state (this fits a bug description).
2. There is no logger that specifies what that weird state was, nor an indication as to why sending service metrics for other services stops working (this fits a feature request that may or may not be necessary to address point 1).






Discussion:
## Flaky Test

* **Test Name:** Build&Test / metricbeat-windows-windows-2019 / TestGetServiceStates – service
* **Artifact Link:** https://beats-ci.elastic.co/blue/organizations/jenkins/it%2Ftest/detail/test/16/


### Error details
```
Failed
```


### Stack Trace
```
    service_status_test.go:29: for testing the flaky test reporting for an existing flaky test
```


## Flaky Test

* **Test Name:** `Build&Test / metricbeat-windows-windows-2019 / TestGetServiceStates – service`
* **Artifact Link:** https://beats-ci.elastic.co/blue/organizations/jenkins/it%2Ftest/detail/test/17/
* **PR:** None
* **Commit:** None


### Error details
```
Failed
```


### Stack Trace
```
    service_status_test.go:29: for testing the flaky test reporting for an existing flaky test
```


## Flaky Test

* **Test Name:** `Build&Test / metricbeat-windows-windows-2019 / TestGetServiceStates – service`
* **Artifact Link:** https://beats-ci.elastic.co/blue/organizations/jenkins/Beats%2Fbeats%2FPR-21853/detail/PR-21853/31/
* **PR:** #21853
* **Commit:** 9c3d14718434fb33f2ceb1caaf567c3bf2652a3e


### Error details
```
Failed
```


### Stack Trace
```
    service_status_test.go:29: for testing the flaky test reporting for an existing flaky test
```


Pinging @elastic/integrations-platforms (Team:Platforms)

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)
@VihasMakwana I think I saw you had root caused the source of the `OpenProcess failed for pid=1724: The parameter is incorrect` error elsewhere? Or am I misremembering?
@cmacknz yes, that's correct. 

On my personal desktop, the metricbeat wasn't able to access following processes, running as root:

- PID 0 and 4 (protected processes, you can never access it.)
- Processes owned by SYSTEM user. (some antivirus processes for eg.)
    - They were accessible, but with limited info.

This was for `system.process` integration though. The above issue is about `windows.service` integration but I believe the root cause is similar.

--- 
@TheRiffRafi do you see any warning related to SeDebugPrivilege at the beginning of logs? 
Something like:
`Metricbeat is running without SeDebugPrivilege, a Windows privilege that allows it to collect metrics...`,
`Failure while attempting to enable SeDebugPrivilege` or `Metricbeat failed to enable the SeDebugPrivilege`?
Can you attach logs from beginning, if possible?

Hello @VihasMakwana! 

Unfortunately I can't help with logs, all the instances I have of the failure have the logs with the problem already started, there is no instance of this where we've caught it in a state where the issue is not occurring and then suddenly starts happening (the systems are going weeks without reporting the service). 

Also, I have to make a correction on the original description, we have only seen this on 8.10.4, we haven't tested on a more recent version as the entire stack for the user is still on 8.10.4, it was a misunderstanding that we had seen this problem on a later version.



Patch:
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
index 6dff17ecc0ed..83a5c19cb5c9 100644
--- a/CHANGELOG.next.asciidoc
+++ b/CHANGELOG.next.asciidoc
@@ -88,6 +88,7 @@ otherwise no tag is added. {issue}42208[42208] {pull}42403[42403]
 - Removed support for the Enterprise Search module {pull}42915[42915]
 - Fix the function to determine CPU cores on windows {issue}42593[42593] {pull}43409[43409]
 - Updated list of supported vSphere versions in the documentation. {pull}43642[43642]
+- Handle permission errors while collecting data from Windows services and don't interrupt the overall collection by skipping affected services {issue}40765[40765] {pull}43665[43665]
 
 *Osquerybeat*
 
diff --git a/metricbeat/module/windows/service/reader.go b/metricbeat/module/windows/service/reader.go
index 379d4cb738f7..961d6789159e 100644
--- a/metricbeat/module/windows/service/reader.go
+++ b/metricbeat/module/windows/service/reader.go
@@ -28,6 +28,7 @@ import (
 
 	"golang.org/x/sys/windows/registry"
 
+	"github.com/elastic/elastic-agent-libs/logp"
 	"github.com/elastic/elastic-agent-libs/mapstr"
 )
 
@@ -48,9 +49,10 @@ type Reader struct {
 	guid              string            // Host's MachineGuid value (a unique ID for the host).
 	ids               map[string]string // Cache of service IDs.
 	protectedServices map[string]struct{}
+	log               *logp.Logger
 }
 
-func NewReader() (*Reader, error) {
+func NewReader(log *logp.Logger) (*Reader, error) {
 	handle, err := openSCManager("", "", ScManagerEnumerateService|ScManagerConnect)
 	if err != nil {
 		return nil, fmt.Errorf("initialization failed: %w", err)
@@ -67,13 +69,14 @@ func NewReader() (*Reader, error) {
 		guid:              guid,
 		ids:               map[string]string{},
 		protectedServices: map[string]struct{}{},
+		log:               log,
 	}
 
 	return r, nil
 }
 
 func (reader *Reader) Read() ([]mapstr.M, error) {
-	services, err := GetServiceStates(reader.handle, reader.state, reader.protectedServices)
+	services, err := GetServiceStates(reader.log, reader.handle, reader.state, reader.protectedServices)
 	if err != nil {
 		return nil, err
 	}
diff --git a/metricbeat/module/windows/service/service.go b/metricbeat/module/windows/service/service.go
index 45e12c6574de..61b950af7454 100644
--- a/metricbeat/module/windows/service/service.go
+++ b/metricbeat/module/windows/service/service.go
@@ -44,7 +44,7 @@ type MetricSet struct {
 // Part of new is also setting up the configuration by processing additional
 // configuration entries if needed.
 func New(base mb.BaseMetricSet) (mb.MetricSet, error) {
-	reader, err := NewReader()
+	reader, err := NewReader(base.Logger())
 	if err != nil {
 		return nil, err
 	}
diff --git a/metricbeat/module/windows/service/service_status.go b/metricbeat/module/windows/service/service_status.go
index 571d324cb599..a4076c6bffe0 100644
--- a/metricbeat/module/windows/service/service_status.go
+++ b/metricbeat/module/windows/service/service_status.go
@@ -131,7 +131,7 @@ func (state ServiceState) String() string {
 	return ""
 }
 
-func GetServiceStates(handle Handle, state ServiceEnumState, protectedServices map[string]struct{}) ([]Status, error) {
+func GetServiceStates(log *logp.Logger, handle Handle, state ServiceEnumState, protectedServices map[string]struct{}) ([]Status, error) {
 	var servicesReturned uint32
 	var servicesBuffer []byte
 
@@ -164,11 +164,18 @@ func GetServiceStates(handle Handle, state ServiceEnumState, protectedServices m
 	var services []Status
 	var sizeStatusProcess = (int)(unsafe.Sizeof(EnumServiceStatusProcess{}))
 	for i := 0; i < int(servicesReturned); i++ {
-		serviceTemp := (*EnumServiceStatusProcess)(unsafe.Pointer(&servicesBuffer[i*sizeStatusProcess]))
+		rawService := (*EnumServiceStatusProcess)(unsafe.Pointer(&servicesBuffer[i*sizeStatusProcess]))
 
-		service, err := getServiceInformation(serviceTemp, servicesBuffer, handle, protectedServices)
+		service, err := getRawServiceStatus(rawService, servicesBuffer)
 		if err != nil {
-			return nil, err
+			log.Errorf("could not parse raw service information for PID %d: %v", rawService.ServiceStatusProcess.DwProcessId, err)
+			continue
+		}
+
+		err = getServiceHandleInformation(log, &service, rawService, handle, protectedServices)
+		if err != nil {
+			log.Errorf("could not get information for the service (name: %s, pid: %d): %v", service.DisplayName, service.PID, err)
+			continue
 		}
 
 		services = append(services, service)
@@ -177,7 +184,7 @@ func GetServiceStates(handle Handle, state ServiceEnumState, protectedServices m
 	return services, nil
 }
 
-func getServiceInformation(rawService *EnumServiceStatusProcess, servicesBuffer []byte, handle Handle, protectedServices map[string]struct{}) (Status, error) {
+func getRawServiceStatus(rawService *EnumServiceStatusProcess, servicesBuffer []byte) (Status, error) {
 	service := Status{
 		PID: rawService.ServiceStatusProcess.DwProcessId,
 	}
@@ -198,6 +205,11 @@ func getServiceInformation(rawService *EnumServiceStatusProcess, servicesBuffer
 	}
 	service.ServiceName = strBuf.String()
 
+	return service, nil
+}
+
+func getServiceHandleInformation(log *logp.Logger, service *Status, rawService *EnumServiceStatusProcess, handle Handle, protectedServices map[string]struct{}) error {
+
 	var state string
 
 	if stat, ok := serviceStates[ServiceState(rawService.ServiceStatusProcess.DwCurrentState)]; ok {
@@ -215,36 +227,38 @@ func getServiceInformation(rawService *EnumServiceStatusProcess, servicesBuffer
 
 	serviceHandle, err := openServiceHandle(handle, service.ServiceName, ServiceQueryConfig)
 	if err != nil {
-		return service, fmt.Errorf("error while opening service %s: %w", service.ServiceName, err)
+		return fmt.Errorf("error while opening service %s: %w", service.ServiceName, err)
 	}
 
 	defer closeHandle(serviceHandle)
 
 	// Get detailed information
-	if err := getAdditionalServiceInfo(serviceHandle, &service); err != nil {
-		return service, err
+	if err := getAdditionalServiceInfo(serviceHandle, service); err != nil {
+		return err
 	}
 
 	// Get optional information
-	if err := getOptionalServiceInfo(serviceHandle, &service); err != nil {
-		return service, err
+	if err := getOptionalServiceInfo(serviceHandle, service); err != nil {
+		return err
 	}
 
 	//Get uptime for service
 	if ServiceState(rawService.ServiceStatusProcess.DwCurrentState) != ServiceStopped {
 		processUpTime, err := getServiceUptime(rawService.ServiceStatusProcess.DwProcessId)
 		if err != nil {
-			if _, ok := protectedServices[service.ServiceName]; errors.Is(err, os.ErrPermission) && !ok {
+			if !errors.Is(err, os.ErrPermission) {
+				// if we have faced any other error, pass it to the caller
+				return err
+			}
+			if _, ok := protectedServices[service.ServiceName]; !ok {
 				protectedServices[service.ServiceName] = struct{}{}
-				logp.Warn("Uptime for service %v is not available because of insufficient rights", service.ServiceName)
-			} else {
-				return service, err
+				log.Warnf("Uptime for service %v is not available because of insufficient rights", service.ServiceName)
 			}
 		}
 		service.Uptime = processUpTime / time.Millisecond
 	}
 
-	return service, nil
+	return nil
 }
 
 func openServiceHandle(handle Handle, serviceName string, desiredAccess ServiceAccessRight) (Handle, error) {


Test patch:
diff --git a/metricbeat/module/windows/service/reader_test.go b/metricbeat/module/windows/service/reader_test.go
index 4a4246a5774a..4e9f47af4fa4 100644
--- a/metricbeat/module/windows/service/reader_test.go
+++ b/metricbeat/module/windows/service/reader_test.go
@@ -23,10 +23,12 @@ import (
 	"testing"
 
 	"github.com/stretchr/testify/assert"
+
+	"github.com/elastic/elastic-agent-libs/logp"
 )
 
 func TestNewReader(t *testing.T) {
-	reader, err := NewReader()
+	reader, err := NewReader(logp.NewTestingLogger(t, ""))
 	assert.NoError(t, err)
 	assert.NotNil(t, reader)
 	defer reader.Close()
@@ -57,7 +59,7 @@ func TestGetMachineGUID(t *testing.T) {
 func TestRead(t *testing.T) {
 	t.Skip("Flaky test: https://github.com/elastic/beats/issues/22171")
 
-	reader, err := NewReader()
+	reader, err := NewReader(logp.NewTestingLogger(t, ""))
 	assert.NoError(t, err)
 	result, err := reader.Read()
 	assert.NoError(t, err)
diff --git a/metricbeat/module/windows/service/service_status_test.go b/metricbeat/module/windows/service/service_status_test.go
index 3b81399d2cb3..f39594ac873b 100644
--- a/metricbeat/module/windows/service/service_status_test.go
+++ b/metricbeat/module/windows/service/service_status_test.go
@@ -23,15 +23,15 @@ import (
 	"testing"
 
 	"github.com/stretchr/testify/assert"
+
+	"github.com/elastic/elastic-agent-libs/logp"
 )
 
 func TestGetServiceStates(t *testing.T) {
-	t.Skip("Flaky test: https://github.com/elastic/beats/issues/22172")
-
 	handle, err := openSCManager("", "", ScManagerEnumerateService|ScManagerConnect)
 	assert.NoError(t, err)
 	assert.NotEqual(t, handle, InvalidDatabaseHandle)
-	services, err := GetServiceStates(handle, ServiceStateAll, map[string]struct{}{})
+	services, err := GetServiceStates(logp.NewTestingLogger(t, ""), handle, ServiceStateAll, map[string]struct{}{})
 	assert.NoError(t, err)
 	assert.True(t, len(services) > 0)
 	closeHandle(handle)