Repository: open-telemetry/opentelemetry-collector-contrib
Pull request #39529: [chore][stanza/namedpipe] Refactor the namedpipe input to remove the fsnotify
Author: login: namco1992
Labels: name: pkg/stanza
name: receiver/namedpipe
Created: 2025-04-22T08:49:28Z | Merged: 2025-04-30T03:05:54Z

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

The current logic is that the input starts 2 goroutines, one for watching for write events, and another one listens from `watcher.C` and kick off the process. In other words, it relies on a write event to start to read from the pipe.

When an active writer is writing to the pipe, and the collector exits for whatever reason (or simply can't consume fast enough), the pipe buffer can be filled and no write can come in. When no write happens, the watcher doesn't receive any write event, and won't kick off the process to read the pipe. This is effectively a deadlock, the writer can't write to the pipe because the pipe buffer is full, and the collector won't read from the pipe because no write event triggered.

Refactor the namedpipe input to get rid of the watcher. The input will now keep reading from pipe. If no data or no external writer, it blocks at the `scan.Scan()` wait for new data or an error when pipe is closed.

#### How to reproduce the bug
A simple python script to write to a named pipe periodically:

```python
import os
import time

PIPE_PATH = "/tmp/my_named_pipe"

try:
    fd = os.open(PIPE_PATH, os.O_RDWR | os.O_APPEND | os.O_NONBLOCK)
except OSError as e:
    print(f"Failed to open pipe: {e}")
    exit(1)

msg = b"x" * 1024 * 10 + b"\n" # 10 KB message with a newline at the end

while True:
    try:
        os.write(fd, msg)
        print("write ok")
    except BlockingIOError:
        print("PIPE FULL - write blocked")
    time.sleep(1)
```

The minimal Otel collector config:
```yaml
receivers:
  namedpipe:
    path: /tmp/my_named_pipe
    mode: 0600

exporters:
  debug:
    verbosity: basic

service:
  pipelines:
    logs:
      receivers: [namedpipe]
      processors: []
      exporters: [debug]
```

1. Start the python script to write the pipe to full (when seeing "PIPE FULL - write blocked"). It's usually the 7th write if the pipe buffer is 64K (default value).
2. Start the otel collector and see it doesn't process anything from the pipe.

<!--Describe what testing was performed and which tests were added.-->
#### Testing
Passed all current unit tests. Also tested locally with the steps above to confirm it reads from the pipe from the start.



Diff:
diff --git a/pkg/stanza/go.mod b/pkg/stanza/go.mod
index 4b913e93b207f..b067723c389ce 100644
--- a/pkg/stanza/go.mod
+++ b/pkg/stanza/go.mod
@@ -6,7 +6,6 @@ require (
 	github.com/bmatcuk/doublestar/v4 v4.8.1
 	github.com/cespare/xxhash/v2 v2.3.0
 	github.com/expr-lang/expr v1.17.2
-	github.com/fsnotify/fsnotify v1.9.0
 	github.com/goccy/go-json v0.10.5
 	github.com/google/uuid v1.6.0
 	github.com/jonboulle/clockwork v0.5.0
@@ -46,6 +45,7 @@ require (
 	github.com/cenkalti/backoff/v4 v4.3.0 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/elastic/lunes v0.1.0 // indirect
+	github.com/fsnotify/fsnotify v1.9.0 // indirect
 	github.com/go-logr/logr v1.4.2 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
diff --git a/pkg/stanza/operator/input/namedpipe/input.go b/pkg/stanza/operator/input/namedpipe/input.go
index 8dc89150d1f97..c6afd4d801fa1 100644
--- a/pkg/stanza/operator/input/namedpipe/input.go
+++ b/pkg/stanza/operator/input/namedpipe/input.go
@@ -11,6 +11,7 @@ import (
 	"fmt"
 	"os"
 	"sync"
+	"time"
 
 	"go.uber.org/zap"
 	"golang.org/x/sys/unix"
@@ -20,6 +21,8 @@ import (
 	"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/trim"
 )
 
+const ReadTimeout = 2 * time.Second
+
 type Input struct {
 	helper.InputOperator
 
@@ -55,59 +58,61 @@ func (i *Input) Start(_ operator.Persister) error {
 		return fmt.Errorf("failed to chmod named pipe: %w", chmodErr)
 	}
 
-	watcher, err := NewWatcher(i.path)
-	if err != nil {
-		return fmt.Errorf("failed to create watcher: %w", err)
-	}
-
+	// Open the pipe with O_RDWR so it won't block on opening the pipe when there is no writer.
+	// The current process is both a writer and reader, which prevents the read from receiving
+	// EOF because there is always a writer (the process itself) connects to the pipe.
 	pipe, err := os.OpenFile(i.path, os.O_RDWR, os.ModeNamedPipe)
 	if err != nil {
 		return fmt.Errorf("failed to open named pipe: %w", err)
 	}
-
 	i.pipe = pipe
 
 	ctx, cancel := context.WithCancel(context.Background())
 	i.cancel = cancel
 
-	i.wg.Add(2)
-	go func() {
-		defer i.wg.Done()
-		if err := watcher.Watch(ctx); err != nil {
-			i.Logger().Error("failed to watch named pipe", zap.Error(err))
-		}
-	}()
-
-	go func() {
-		defer i.wg.Done()
-		for {
-			select {
-			case <-watcher.C:
-				if err := i.process(ctx, pipe); err != nil {
-					i.Logger().Error("failed to process named pipe", zap.Error(err))
-				}
-			case <-ctx.Done():
-				return
-			}
-		}
-	}()
+	i.wg.Add(1)
+	go i.readLoop(ctx)
 
 	return nil
 }
 
 func (i *Input) Stop() error {
-	if i.pipe != nil {
-		i.pipe.Close()
-	}
-
 	if i.cancel != nil {
 		i.cancel()
 	}
 
+	if i.pipe != nil {
+		i.pipe.Close()
+	}
+
 	i.wg.Wait()
 	return nil
 }
 
+func (i *Input) readLoop(ctx context.Context) {
+	defer i.wg.Done()
+	pipe := i.pipe
+
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		default:
+			if err := i.process(ctx, pipe); err != nil {
+				i.Logger().Error("error processing named pipe", zap.Error(err))
+			}
+
+			// The process exits due to whatever reason, wait for ReadTimeout and try again.
+			select {
+			case <-ctx.Done():
+				return
+			case <-time.After(ReadTimeout):
+				i.Logger().Warn("processing named pipe is interrupted, retrying the process now", zap.String("path", i.path))
+			}
+		}
+	}
+}
+
 func (i *Input) process(ctx context.Context, pipe *os.File) error {
 	scan := bufio.NewScanner(pipe)
 	scan.Split(i.splitFunc)
diff --git a/pkg/stanza/operator/input/namedpipe/input_test.go b/pkg/stanza/operator/input/namedpipe/input_test.go
index 97d07ef0e7100..6716a9002f6f3 100644
--- a/pkg/stanza/operator/input/namedpipe/input_test.go
+++ b/pkg/stanza/operator/input/namedpipe/input_test.go
@@ -150,3 +150,60 @@ func TestPipeWrites(t *testing.T) {
 		}
 	}
 }
+
+// TestPipeHasDataAtStartup will test if the receiver can consume from a named
+// pipe that has buffered data before startup.
+func TestPipeHasDataAtStartup(t *testing.T) {
+	fake := testutil.NewFakeOutput(t)
+
+	conf := NewConfig()
+	conf.Path = filename(t)
+	conf.Permissions = 0o666
+	conf.OutputIDs = []string{fake.ID()}
+
+	set := componenttest.NewNopTelemetrySettings()
+	op, err := conf.Build(set)
+	require.NoError(t, err)
+	ops := []operator.Operator{op, fake}
+
+	// create pipe
+	require.NoError(t, unix.Mkfifo(conf.Path, conf.Permissions))
+
+	pipe, err := os.OpenFile(conf.Path, os.O_RDWR|os.O_APPEND, os.ModeNamedPipe)
+	require.NoError(t, err)
+	defer func() {
+		require.NoError(t, pipe.Close())
+	}()
+
+	logs := []string{"log1\n", "log2\n"}
+
+	for _, log := range logs {
+		_, err = pipe.WriteString(log)
+		require.NoError(t, err)
+	}
+
+	p, err := pipeline.NewDirectedPipeline(ops)
+	require.NoError(t, err)
+
+	// start receiver
+	require.NoError(t, p.Start(testutil.NewUnscopedMockPersister()))
+	defer func() {
+		require.NoError(t, p.Stop())
+	}()
+
+	for _, log := range logs {
+		expect := &entry.Entry{
+			Body: strings.TrimSpace(log),
+		}
+
+		select {
+		case e := <-fake.Received:
+			obs := time.Now()
+			expect.ObservedTimestamp = obs
+			e.ObservedTimestamp = obs
+			require.Equal(t, expect, e)
+		case <-time.After(time.Second):
+			t.Fatal("timed-out waiting for log entry")
+		}
+	}
+}
diff --git a/pkg/stanza/operator/input/namedpipe/watcher.go b/pkg/stanza/operator/input/namedpipe/watcher.go
deleted file mode 100644
index 62c721f0f442c..0000000000000
--- a/pkg/stanza/operator/input/namedpipe/watcher.go
+++ /dev/null
@@ -1,70 +0,0 @@
-// Copyright The OpenTelemetry Authors
-// SPDX-License-Identifier: Apache-2.0
-
-//go:build linux
-
-package namedpipe // import "github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/namedpipe"
-
-import (
-	"context"
-	"fmt"
-
-	"github.com/fsnotify/fsnotify"
-)
-
-// Watcher watches a file for writes, notifying via `C` when one is observed.
-type Watcher struct {
-	C       chan struct{}
-	watcher *fsnotify.Watcher
-}
-
-// NewWatcher creates a new watcher for the given path.
-func NewWatcher(path string) (*Watcher, error) {
-	watcher, err := newWatcher(path)
-	if err != nil {
-		return nil, err
-	}
-
-	return &Watcher{
-		C:       make(chan struct{}),
-		watcher: watcher,
-	}, nil
-}
-
-// Watch starts the watcher, sending a message to `C` when a write is observed.
-func (w *Watcher) Watch(ctx context.Context) error {
-	defer func() {
-		w.watcher.Close()
-		close(w.C)
-	}()
-
-	for {
-		select {
-		case <-ctx.Done():
-			return nil
-		case event := <-w.watcher.Events:
-			if event.Op&fsnotify.Write == fsnotify.Write {
-				select {
-				case w.C <- struct{}{}:
-				case <-ctx.Done():
-					return nil
-				}
-			}
-		case err := <-w.watcher.Errors:
-			return fmt.Errorf("watcher error: %w", err)
-		}
-	}
-}
-
-func newWatcher(path string) (*fsnotify.Watcher, error) {
-	watcher, err := fsnotify.NewWatcher()
-	if err != nil {
-		return nil, fmt.Errorf("failed to create watcher: %w", err)
-	}
-
-	if err := watcher.Add(path); err != nil {
-		return nil, fmt.Errorf("failed to add path to watcher: %w", err)
-	}
-
-	return watcher, nil
-}
diff --git a/receiver/namedpipereceiver/go.mod b/receiver/namedpipereceiver/go.mod
index eb2407f05fa94..355f4e2bf0a1b 100644
--- a/receiver/namedpipereceiver/go.mod
+++ b/receiver/namedpipereceiver/go.mod
@@ -31,7 +31,6 @@ require (
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/elastic/lunes v0.1.0 // indirect
 	github.com/expr-lang/expr v1.17.2 // indirect
-	github.com/fsnotify/fsnotify v1.9.0 // indirect
 	github.com/go-logr/logr v1.4.2 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-viper/mapstructure/v2 v2.2.1 // indirect
diff --git a/receiver/namedpipereceiver/go.sum b/receiver/namedpipereceiver/go.sum
index ff6f43a086761..aa0a894762fd0 100644
--- a/receiver/namedpipereceiver/go.sum
+++ b/receiver/namedpipereceiver/go.sum
@@ -9,8 +9,6 @@ github.com/elastic/lunes v0.1.0 h1:amRtLPjwkWtzDF/RKzcEPMvSsSseLDLW+bnhfNSLRe4=
 github.com/elastic/lunes v0.1.0/go.mod h1:xGphYIt3XdZRtyWosHQTErsQTd4OP1p9wsbVoHelrd4=
 github.com/expr-lang/expr v1.17.2 h1:o0A99O/Px+/DTjEnQiodAgOIK9PPxL8DtXhBRKC+Iso=
 github.com/expr-lang/expr v1.17.2/go.mod h1:8/vRC7+7HBzESEqt5kKpYXxrxkr31SaO8r40VO/1IT4=
-github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
-github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
 github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
 github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
 github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=


Reviews:
user: login: djaglowski
body: Makes sense to me. Any thoughts on this @sinkingpoint?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-24T17:36:53Z
user: login: sinkingpoint
body: This looks good to me. Only one minor comment. I do think at some point (and this is more of a note to self) that we should probably hoist the whole `Scanner` into the Input rather than constructing it on every `process` call, but that doesn't belong here, and it probably doesn't have too much of an impact considering we're already reusing the buffer
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T14:50:06Z
user: login: namco1992
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-30T02:44:43Z
user: login: djaglowski
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-30T03:05:48Z

Review comments:
diff_hunk: @@ -55,59 +58,61 @@ func (i *Input) Start(_ operator.Persister) error {
 		return fmt.Errorf("failed to chmod named pipe: %w", chmodErr)
 	}
 
-	watcher, err := NewWatcher(i.path)
-	if err != nil {
-		return fmt.Errorf("failed to create watcher: %w", err)
-	}
-
+	// Open the pipe with O_RDWR so it won't block on opening the pipe when there is no writer.
+	// The current process is both a writer and reader, which prevents the read from receiving
+	// EOF because there is always a writer (the process itself) connects to the pipe.
 	pipe, err := os.OpenFile(i.path, os.O_RDWR, os.ModeNamedPipe)
 	if err != nil {
 		return fmt.Errorf("failed to open named pipe: %w", err)
 	}
-
 	i.pipe = pipe
 
 	ctx, cancel := context.WithCancel(context.Background())
 	i.cancel = cancel
 
-	i.wg.Add(2)
-	go func() {
-		defer i.wg.Done()
-		if err := watcher.Watch(ctx); err != nil {
-			i.Logger().Error("failed to watch named pipe", zap.Error(err))
-		}
-	}()
-
-	go func() {
-		defer i.wg.Done()
-		for {
-			select {
-			case <-watcher.C:
-				if err := i.process(ctx, pipe); err != nil {
-					i.Logger().Error("failed to process named pipe", zap.Error(err))
-				}
-			case <-ctx.Done():
-				return
-			}
-		}
-	}()
+	i.wg.Add(1)
+	go i.readLoop(ctx)
 
 	return nil
 }
 
 func (i *Input) Stop() error {
-	if i.pipe != nil {
-		i.pipe.Close()
-	}
-
 	if i.cancel != nil {
 		i.cancel()
 	}
 
+	if i.pipe != nil {
+		i.pipe.Close()
+	}
+
 	i.wg.Wait()
 	return nil
 }
 
+func (i *Input) readLoop(ctx context.Context) {
+	defer i.wg.Done()
+	pipe := i.pipe
+
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		default:
+			err := i.process(ctx, pipe)
+			if err != nil {
+				i.Logger().Error("error processing named pipe", zap.Error(err))
+			}
+
+			// The process exits due to whatever reason, wait for ReadTimeout and try again.
+			select {
+			case <-ctx.Done():
+				return
+			case <-time.After(ReadTimeout):
path: pkg/stanza/operator/input/namedpipe/input.go
user: login: sinkingpoint
body: Might be worth a log here so we can keep track of when this happens? It probably indicates wider issues
created_at: 2025-04-29T14:45:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -55,59 +58,61 @@ func (i *Input) Start(_ operator.Persister) error {
 		return fmt.Errorf("failed to chmod named pipe: %w", chmodErr)
 	}
 
-	watcher, err := NewWatcher(i.path)
-	if err != nil {
-		return fmt.Errorf("failed to create watcher: %w", err)
-	}
-
+	// Open the pipe with O_RDWR so it won't block on opening the pipe when there is no writer.
+	// The current process is both a writer and reader, which prevents the read from receiving
+	// EOF because there is always a writer (the process itself) connects to the pipe.
 	pipe, err := os.OpenFile(i.path, os.O_RDWR, os.ModeNamedPipe)
 	if err != nil {
 		return fmt.Errorf("failed to open named pipe: %w", err)
 	}
-
 	i.pipe = pipe
 
 	ctx, cancel := context.WithCancel(context.Background())
 	i.cancel = cancel
 
-	i.wg.Add(2)
-	go func() {
-		defer i.wg.Done()
-		if err := watcher.Watch(ctx); err != nil {
-			i.Logger().Error("failed to watch named pipe", zap.Error(err))
-		}
-	}()
-
-	go func() {
-		defer i.wg.Done()
-		for {
-			select {
-			case <-watcher.C:
-				if err := i.process(ctx, pipe); err != nil {
-					i.Logger().Error("failed to process named pipe", zap.Error(err))
-				}
-			case <-ctx.Done():
-				return
-			}
-		}
-	}()
+	i.wg.Add(1)
+	go i.readLoop(ctx)
 
 	return nil
 }
 
 func (i *Input) Stop() error {
-	if i.pipe != nil {
-		i.pipe.Close()
-	}
-
 	if i.cancel != nil {
 		i.cancel()
 	}
 
+	if i.pipe != nil {
+		i.pipe.Close()
+	}
+
 	i.wg.Wait()
 	return nil
 }
 
+func (i *Input) readLoop(ctx context.Context) {
+	defer i.wg.Done()
+	pipe := i.pipe
+
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		default:
+			err := i.process(ctx, pipe)
+			if err != nil {
+				i.Logger().Error("error processing named pipe", zap.Error(err))
+			}
+
+			// The process exits due to whatever reason, wait for ReadTimeout and try again.
+			select {
+			case <-ctx.Done():
+				return
+			case <-time.After(ReadTimeout):
path: pkg/stanza/operator/input/namedpipe/input.go
user: login: namco1992
body: We logged the error from the process call, but no harm adding a log here too.
created_at: 2025-04-30T02:44:43Z
author_association: CONTRIBUTOR

PR comments:
user: login: namco1992
created_at: 2025-04-23T06:17:55Z
body: The failed check doesn't seem to be related to my change. @djaglowski would u mind taking a look? Thanks!
author_association: CONTRIBUTOR
user: login: sinkingpoint
created_at: 2025-04-29T07:54:48Z
body: Apologies for the delay - I've been offline for a bit. Will look at this today 
author_association: CONTRIBUTOR
user: login: namco1992
created_at: 2025-04-30T03:02:57Z
body: > This looks good to me. Only one minor comment. I do think at some point (and this is more of a note to self) that we should probably hoist the whole `Scanner` into the Input rather than constructing it on every `process` call, but that doesn't belong here, and it probably doesn't have too much of an impact considering we're already reusing the buffer

Sure, I can do it in a separated PR. This should be less of a concern since it won't exit the process in normal circumstances, and as you mentioned, we are alrd reusing the buffer.
author_association: CONTRIBUTOR