Repository: cilium/ebpf
Pull request #755 (2022-08-03 04:00:34)

link: add maxactive for kretprobe
In some case we want to missed as little as possible and
it is helpful to have this configuration option.

alahaiyo: > Thank you for your PR!
> 
> Can you explain your use case for maxactive and why implementing your own sampling in BPF isn't sufficient?
> 
> As it stands, the PR is too special cased for me:
> 
> * doesn't work with pmu probes
> * only supported for retprobes (?)
> 
> Kprobes have a large API, and supporting all of it is not possible. So we need a very compelling reason to add something like your proposed change.


alahaiyo: > Thank you for your PR!
> 
> Can you explain your use case for maxactive and why implementing your own sampling in BPF isn't sufficient?
> 
> As it stands, the PR is too special cased for me:
> 
> * doesn't work with pmu probes
> * only supported for retprobes (?)
> 
> Kprobes have a large API, and supporting all of it is not possible. So we need a very compelling reason to add something like your proposed change.

If the traced function can sleep, then when the number of processes exceeds the number of CPUs, the kretprobe will not be triggered. Because the kernel defaults maxactive to NR_CPU. When we trace some kernel network functions (such as inet_accept), when the network connection pressure is very high, there will be a lot of kretprobe not triggered. 

When writing kprobe_event through tracefs, you can set maxactive to a maximum of 2048, which is enough. The implementation of pmu probe in the kernel will call create_local_trace_kprobe, in which maxactive is hardcoded to 0.
lmb: Thanks for the background, that's interesting.

So what is the behaviour of a pmu retprobe in the same scenario? It drops retprobe calls, without being able to work around the limitation? Asked another way, what does it mean that maxactive is 0 in the pmu case?

What problem does maxactive guard against in the first place? Maybe we could just hardcode it to 2048 in the tracefs case?

Cc @mmat11 
mmat11: > It drops retprobe calls, without being able to work around the limitation? Asked another way, what does it mean that maxactive is 0 in the pmu case?

I believe this is the case

> What problem does maxactive guard against in the first place?

I found an explanation here https://github.com/iovisor/bcc/issues/1072#issue-217189257

> Maybe we could just hardcode it to 2048 in the tracefs case?

From this commit https://github.com/torvalds/linux/commit/696ced4fb1d76802f864d8848aa4716633f83c17 it seems the maximum is 4096. I guess there's some overhead (see https://elixir.bootlin.com/linux/v5.19/source/kernel/kprobes.c#L2202) in having a big number, else it could have been hardcoded in the kernel itself (?).

I wonder why this parameter isn't configurable via `perf_event_open`.
IMO we can support the feature but we should raise an error when it's not possible to do so (kernel < 4.12 or maxactive > 0 on non-return probes) and clearly document that setting this option will force the probe to be installed via tracefs
nashuiliang: hi, @mmat11 

> I wonder why this parameter isn't configurable via `perf_event_open`. 

Tow related commits(not merged):

* https://lore.kernel.org/all/20191007223111.1142454-1-songliubraving@fb.com/

```text
Enable specifying maxactive for fd based kretprobe. This will be useful
for tracing tools like bcc and bpftrace. [1] discussed the need of this
in bpftrace. Use highest highest 12 bit (bit 52-63) to allow maximal
maxactive of 4095.

[1] https://github.com/iovisor/bpftrace/issues/835
```

* https://lore.kernel.org/bpf/20220714193403.13211-1-9erthalion6@gmail.com/

```text
From: Song Liu <songliubraving@fb.com>

Enable specifying maxactive for fd based kretprobe. This will be useful
for tracing tools like bcc and bpftrace (see for example discussion [1]).
Use highest 4 bit (bit 59-63) to allow specifying maxactive by log2.

The original patch [2] seems to be fallen through the cracks and wasn't
applied. I've merely rebased the work done by Song Liu, verififed it
still works, and modified to allow specifying maxactive by log2 per
suggestion from the discussion thread.

Note that changes in rethook implementation may render maxactive
obsolete.

[1]: https://github.com/iovisor/bpftrace/issues/835
[2]: https://lore.kernel.org/all/20191007223111.1142454-1-songliubraving@fb.com/
```

It seems to be discussing a more elegant way to implement.

> IMO we can support the feature but we should raise an error when it's not possible to do so (kernel < 4.12 or maxactive > 0 on non-return probes) and clearly document that setting this option will force the probe to be installed via tracefs

Agree, I also expect this feature.

alahaiyo: Thank you for your reply！
> > It drops retprobe calls, without being able to work around the limitation? Asked another way, what does it mean that maxactive is 0 in the pmu case?
> 
> I believe this is the case

Agree
 
> > What problem does maxactive guard against in the first place?
> 
> I found an explanation here [iovisor/bcc#1072 (comment)](https://github.com/iovisor/bcc/issues/1072#issue-217189257)
> 
> > Maybe we could just hardcode it to 2048 in the tracefs case?
> 
> From this commit [torvalds/linux@696ced4](https://github.com/torvalds/linux/commit/696ced4fb1d76802f864d8848aa4716633f83c17) it seems the maximum is 4096. 

Oh my mistake, the maximum is 4096.

>I guess there's some overhead (see https://elixir.bootlin.com/linux/v5.19/source/kernel/kprobes.c#L2202) in having a big number, else it could have been hardcoded in the kernel itself (?).
> 
> I wonder why this parameter isn't configurable via `perf_event_open`. 

As @nashuiliang said, the feature is under discussion now.

> IMO we can support the feature but we should raise an error when it's not possible to do so (kernel < 4.12 or maxactive > 0 on non-return probes) and clearly document that setting this option will force the probe to be installed via tracefs

Yes, I agree. I will add this and commit again.

lmb: @alahaiyo I'm not sure yet what the best approach is here, so there is a risk that your work might be wasted. I'll read through all the info here and try to reach an opinion today or tomorrow.
alahaiyo: > @alahaiyo I'm not sure yet what the best approach is here, so there is a risk that your work might be wasted. I'll read through all the info here and try to reach an opinion today or tomorrow.

I'm really looking forward to your opinion.
lmb: From the kernel side we have two separate APIs to create a kretprobe:

1. via `tracefs`: supports setting `maxactive` and missed events are reported via
[`/sys/kernel/debug/tracing/kprobe_profile`](https://docs.kernel.org/trace/kprobetrace.html#event-profiling).

2. via `perf_event_open`: supports neither `maxactive` nor missed events. Peter Zijlstra wants to [avoid adding maxactive to p_e_o if possible](https://lore.kernel.org/bpf/YtgMKDgNLnMIkHLI@worktop.programming.kicks-ass.net/). A patch to [expose missed events has gotten stuck, but the maintainer seems open to it](https://lore.kernel.org/bpf/20191028091229.GJ4131@hirez.programming.kicks-ass.net/).

On the user space side, maxactive is exposed in [`bcc`](https://github.com/iovisor/bcc/pull/2224) and [`gobpf`](https://github.com/iovisor/gobpf/pull/39). `bpftrace` [hardcodes maxactive to `0`](https://github.com/iovisor/bpftrace/blob/dcc105da3ff024f981143a8408b6d3956bfd73b6/src/attached_probe.cpp#L901) and would like to [print the number of missed events](https://github.com/iovisor/bpftrace/issues/835#issuecomment-515166844) instead.

Let's consider the implications of the `maxactive` API as proposed in the PR:

- On pre 4.12 kernels the user will receive an error from the kernel, since the feature is not supported. So users have to avoid setting maxactive on old kernels. This requires checking kernel version or similar, or feature probing from the library to avoid setting maxactive.
- On >= 4.12 kernels, we will force use of `tracefs` if maxactive!=0 even if `pmu` is available. `tracefs` behaves poorly when the tracer crashes.
- If `maxactive` becomes available on `pmu` after all, we have to do feature detection to
  fall back to `tracefs` for kernels that have `pmu` but don't have the maxactive API.
- If `maxactive` never becomes available on `pmu` we are forever locked into `tracefs` for kretprobes, even if regular pmu kprobes don't require maxactive anymore.
- It's completely unclear what users should set `maxactive` to, since it needs to account for all callers of a function on the whole system. So being able to set maxactive still doesn't guarantee that all retprobes fire.

All in all, `maxactive` seems like a bad API. I think `bpftrace` has it right with exposing the number of missed events instead. If we manage to resurrect the fd-based missed events patch we could provide a somewhat unified API across tracefs and pmu.

@alahaiyo in your PR description you say that you want to "miss as little as possible". Would it be acceptable to you if you instead had a way to see how many misses you had, and scaling your result accordingly?

If not, what value would you set `maxactive` to? How do you figure out whether the kretprobe needs `maxactive` in the first place? How do you plan on dealing with kernels < 4.12?

If just misses is not enough, we could add a higher level option like `FunctionMaySleep bool` or similar instead of `Maxactive int`. Behind the scenes the lib would try to figure out the best strategy, which for now would be setting `maxactive=4096` and forcing `tracefs`. This is better than `maxactive` since it avoids exposing the low level implementation detail and gives us flexibility in the future. The downside is that it requires a lot more feature probing and magic in the library itself.
lmb: @alahaiyo do you notice the missed events on specific machines, or is your goal to reduce misses outright? If there is a problem with a specific machine, is CONFIG_PREEMPT enabled? What is the value of `/sys/devices/system/cpu/possible`?
lmb: From https://github.com/iovisor/bcc/pull/2224#issue-412958330:

> For kernels without maxactive supports in debugfs, the error resolution is not straightforward. The kernel will still install a probe, but under a different name.

The issue is that < 4.12 kernels will treat the `r50:...` written to tracefs as the name of the probe. This will need to be handled if we end up adding maxactive after all.
alahaiyo: > @alahaiyo do you notice the missed events on specific machines, or is your goal to reduce misses outright? If there is a problem with a specific machine, is CONFIG_PREEMPT enabled? What is the value of `/sys/devices/system/cpu/possible`?

It's not on specific machines. CONFIG_PREEMPT is not set and the value of `/sys/devices/system/cpu/possible` is `0-95`.
alahaiyo: > From [iovisor/bcc#2224 (comment)](https://github.com/iovisor/bcc/pull/2224#issue-412958330):
> 
> > For kernels without maxactive supports in debugfs, the error resolution is not straightforward. The kernel will still install a probe, but under a different name.
> 
> The issue is that < 4.12 kernels will treat the `r50:...` written to tracefs as the name of the probe. This will need to be handled if we end up adding maxactive after all.

Yes, that is a good reference.
alahaiyo: >On pre 4.12 kernels the user will receive an error from the kernel, since the feature is not supported. So users have to avoid setting maxactive on old kernels. This requires checking kernel version or similar, or feature probing from the library to avoid setting maxactive.

We can handle with that.

>On >= 4.12 kernels, we will force use of tracefs if maxactive!=0 even if pmu is available. tracefs behaves poorly when the tracer crashes.

In the interface of `link.Kprobe`, users can ignore maxactive, so the default is 0 and pmu will be used. If the user does not want to missed events, then they can set maxactive by themselves, especially when the misses is unbearable.

> If maxactive becomes available on pmu after all, we have to do feature detection to
fall back to tracefs for kernels that have pmu but don't have the maxactive API.
> If maxactive never becomes available on pmu we are forever locked into tracefs for kretprobes, even if regular pmu kprobes don't require maxactive anymore.
> It's completely unclear what users should set maxactive to, since it needs to account for all callers of a function on the whole system. So being able to set maxactive still doesn't guarantee that all retprobes fire.

Even setting `maxactive` to 4096 does not guarantee that events will not be missed, and if so we have no other way. But I think the user needs to be given the ability to adjust the possibility of event missed.

>If not, what value would you set maxactive to? How do you figure out whether the kretprobe needs maxactive in the first place? How do you plan on dealing with kernels < 4.12?

In fact, I don't know how to set maxactive. In our case it was only found that there were kretprobe events missing and this was not acceptable. The solution to this problem is to increase `maxactive` to a large enough value.

>we could add a higher level option like FunctionMaySleep bool or similar instead of Maxactive int

But I still think exposing `maxactive` is better than making the user think about whether the function might sleep. Maxactive is considered as an attribute of kretprobe, which can make it easier for users to find out whether the function of trace will have the problem of loss of kretprobe events.

alahaiyo: I'm glad that this PR can continue to push forward. I have made changes to the code based on the comments. @lmb 
lmb: @alahaiyo I think the issue I pointed out in https://github.com/cilium/ebpf/pull/755#issuecomment-1211758219 is not addressed yet?
alahaiyo: I found that it already has a query for the event ID after the event is created. So I just need to determine if I need to recreate it. My new commit is pushed. @lmb 
alahaiyo: > I like how you integrated the check for unsupported maxactive! In the interests of getting this merged soon I'll push a couple of commits that contain changes I would've asked you to make. There is only one functional change: I'm dropping the fallback to disable maxactive.
> 
> Please take a look at my commits and let me know if you are OK with the modifications. Please also update the tests as mentioned.

I total OK with that. And I push my commits which deal with tests and a little bugfix.
lmb: > deal with tests and a little bugfix.

A good catch. I've opted to return `os.ErrExist` from `create...` instead, that way we don't have to change the tests. I also included the symbol offset in the even name when removing, since that may be != 0.

Patch:
diff --git a/link/kprobe.go b/link/kprobe.go
index 99715f6..9ce7eb4 100644
--- a/link/kprobe.go
+++ b/link/kprobe.go
@@ -25,7 +25,7 @@ type probeType uint8
 type probeArgs struct {
 	symbol, group, path          string
 	offset, refCtrOffset, cookie uint64
-	pid                          int
+	pid, retprobeMaxActive       int
 	ret                          bool
 }
 
@@ -41,6 +41,12 @@ type KprobeOptions struct {
 	// Can be used to insert kprobes at arbitrary offsets in kernel functions,
 	// e.g. in places where functions have been inlined.
 	Offset uint64
+	// Increase the maximum number of concurrent invocations of a kretprobe.
+	// Required when tracing some long running functions in the kernel.
+	//
+	// Deprecated: this setting forces the use of an outdated kernel API and is not portable
+	// across kernel versions.
+	RetprobeMaxActive int
 }
 
 const (
@@ -176,6 +182,7 @@ func kprobe(symbol string, prog *ebpf.Program, opts *KprobeOptions, ret bool) (*
 	}
 
 	if opts != nil {
+		args.retprobeMaxActive = opts.RetprobeMaxActive
 		args.cookie = opts.Cookie
 		args.offset = opts.Offset
 	}
@@ -231,6 +238,11 @@ func pmuProbe(typ probeType, args probeArgs) (*perfEvent, error) {
 		return nil, err
 	}
 
+	// Use tracefs if we want to set kretprobe's retprobeMaxActive.
+	if args.retprobeMaxActive != 0 {
+		return nil, fmt.Errorf("pmu probe: non-zero retprobeMaxActive: %w", ErrNotSupported)
+	}
+
 	var config uint64
 	if args.ret {
 		bit, err := readUint64FromFileOnce("config:%d\n", "/sys/bus/event_source/devices", typ.String(), "/format/retprobe")
@@ -347,7 +359,7 @@ func tracefsKprobe(args probeArgs) (*perfEvent, error) {
 // Path and offset are only set in the case of uprobe(s) and are used to set
 // the executable/library path on the filesystem and the offset where the probe is inserted.
 // A perf event is then opened on the newly-created trace event and returned to the caller.
-func tracefsProbe(typ probeType, args probeArgs) (_ *perfEvent, err error) {
+func tracefsProbe(typ probeType, args probeArgs) (*perfEvent, error) {
 	// Generate a random string for each trace event we attempt to create.
 	// This value is used as the 'group' token in tracefs to allow creating
 	// multiple kprobe trace events with the same name.
@@ -357,41 +369,20 @@ func tracefsProbe(typ probeType, args probeArgs) (_ *perfEvent, err error) {
 	}
 	args.group = group
 
-	// Before attempting to create a trace event through tracefs,
-	// check if an event with the same group and name already exists.
-	// Kernels 4.x and earlier don't return os.ErrExist on writing a duplicate
-	// entry, so we need to rely on reads for detecting uniqueness.
-	_, err = getTraceEventID(group, args.symbol)
-	if err == nil {
-		return nil, fmt.Errorf("trace event already exists: %s/%s", group, args.symbol)
-	}
-	if err != nil && !errors.Is(err, os.ErrNotExist) {
-		return nil, fmt.Errorf("checking trace event %s/%s: %w", group, args.symbol, err)
-	}
-
 	// Create the [k,u]probe trace event using tracefs.
-	if err := createTraceFSProbeEvent(typ, args); err != nil {
-		return nil, fmt.Errorf("creating probe entry on tracefs: %w", err)
-	}
-	defer func() {
-		if err != nil {
-			// Make sure we clean up the created tracefs event when we return error.
-			// If a livepatch handler is already active on the symbol, the write to
-			// tracefs will succeed, a trace event will show up, but creating the
-			// perf event will fail with EBUSY.
-			_ = closeTraceFSProbeEvent(typ, args.group, args.symbol)
-		}
-	}()
-
-	// Get the newly-created trace event's id.
-	tid, err := getTraceEventID(group, args.symbol)
+	tid, err := createTraceFSProbeEvent(typ, args)
 	if err != nil {
-		return nil, fmt.Errorf("getting trace event id: %w", err)
+		return nil, fmt.Errorf("creating probe entry on tracefs: %w", err)
 	}
 
 	// Kprobes are ephemeral tracepoints and share the same perf event type.
 	fd, err := openTracepointPerfEvent(tid, args.pid)
 	if err != nil {
+		// Make sure we clean up the created tracefs event when we return error.
+		// If a livepatch handler is already active on the symbol, the write to
+		// tracefs will succeed, a trace event will show up, but creating the
+		// perf event will fail with EBUSY.
+		_ = closeTraceFSProbeEvent(typ, args.group, args.symbol)
 		return nil, err
 	}
 
@@ -405,15 +396,32 @@ func tracefsProbe(typ probeType, args probeArgs) (_ *perfEvent, err error) {
 	}, nil
 }
 
-// createTraceFSProbeEvent creates a new ephemeral trace event by writing to
-// <tracefs>/[k,u]probe_events. Returns os.ErrNotExist if symbol is not a valid
+var errInvalidMaxActive = errors.New("can only set maxactive on kretprobes")
+
+// createTraceFSProbeEvent creates a new ephemeral trace event.
+//
+// Returns os.ErrNotExist if symbol is not a valid
 // kernel symbol, or if it is not traceable with kprobes. Returns os.ErrExist
-// if a probe with the same group and symbol already exists.
-func createTraceFSProbeEvent(typ probeType, args probeArgs) error {
+// if a probe with the same group and symbol already exists. Returns an error if
+// args.retprobeMaxActive is used on non kprobe types. Returns ErrNotSupported if
+// the kernel is too old to support kretprobe maxactive.
+func createTraceFSProbeEvent(typ probeType, args probeArgs) (uint64, error) {
+	// Before attempting to create a trace event through tracefs,
+	// check if an event with the same group and name already exists.
+	// Kernels 4.x and earlier don't return os.ErrExist on writing a duplicate
+	// entry, so we need to rely on reads for detecting uniqueness.
+	_, err := getTraceEventID(args.group, args.symbol)
+	if err == nil {
+		return 0, fmt.Errorf("trace event %s/%s: %w", args.group, args.symbol, os.ErrExist)
+	}
+	if err != nil && !errors.Is(err, os.ErrNotExist) {
+		return 0, fmt.Errorf("checking trace event %s/%s: %w", args.group, args.symbol, err)
+	}
+
 	// Open the kprobe_events file in tracefs.
 	f, err := os.OpenFile(typ.EventsPath(), os.O_APPEND|os.O_WRONLY, 0666)
 	if err != nil {
-		return fmt.Errorf("error opening '%s': %w", typ.EventsPath(), err)
+		return 0, fmt.Errorf("error opening '%s': %w", typ.EventsPath(), err)
 	}
 	defer f.Close()
 
@@ -434,8 +442,11 @@ func createTraceFSProbeEvent(typ probeType, args probeArgs) error {
 		// subsampling or rate limiting logic can be more accurately implemented in
 		// the eBPF program itself.
 		// See Documentation/kprobes.txt for more details.
+		if args.retprobeMaxActive != 0 && !args.ret {
+			return 0, errInvalidMaxActive
+		}
 		token = kprobeToken(args)
-		pe = fmt.Sprintf("%s:%s/%s %s", probePrefix(args.ret), args.group, sanitizeSymbol(args.symbol), token)
+		pe = fmt.Sprintf("%s:%s/%s %s", probePrefix(args.ret, args.retprobeMaxActive), args.group, sanitizeSymbol(args.symbol), token)
 	case uprobeType:
 		// The uprobe_events syntax is as follows:
 		// p[:[GRP/]EVENT] PATH:OFFSET [FETCHARGS] : Set a probe
@@ -447,38 +458,63 @@ func createTraceFSProbeEvent(typ probeType, args probeArgs) error {
 		// p:ebpf_5678/main_mySymbol /bin/mybin:0x12345(0x123)
 		//
 		// See Documentation/trace/uprobetracer.txt for more details.
+		if args.retprobeMaxActive != 0 {
+			return 0, errInvalidMaxActive
+		}
 		token = uprobeToken(args)
-		pe = fmt.Sprintf("%s:%s/%s %s", probePrefix(args.ret), args.group, args.symbol, token)
+		pe = fmt.Sprintf("%s:%s/%s %s", probePrefix(args.ret, 0), args.group, args.symbol, token)
 	}
 	_, err = f.WriteString(pe)
 
 	// Since commit 97c753e62e6c, ENOENT is correctly returned instead of EINVAL
 	// when trying to create a retprobe for a missing symbol.
 	if errors.Is(err, os.ErrNotExist) {
-		return fmt.Errorf("token %s: not found: %w", token, err)
+		return 0, fmt.Errorf("token %s: not found: %w", token, err)
 	}
 	// Since commit ab105a4fb894, EILSEQ is returned when a kprobe sym+offset is resolved
 	// to an invalid insn boundary. The exact conditions that trigger this error are
 	// arch specific however.
 	if errors.Is(err, syscall.EILSEQ) {
-		return fmt.Errorf("token %s: bad insn boundary: %w", token, os.ErrNotExist)
+		return 0, fmt.Errorf("token %s: bad insn boundary: %w", token, os.ErrNotExist)
 	}
 	// ERANGE is returned when the `SYM[+offs]` token is too big and cannot
 	// be resolved.
 	if errors.Is(err, syscall.ERANGE) {
-		return fmt.Errorf("token %s: offset too big: %w", token, os.ErrNotExist)
+		return 0, fmt.Errorf("token %s: offset too big: %w", token, os.ErrNotExist)
 	}
 
 	if err != nil {
-		return fmt.Errorf("token %s: writing '%s': %w", token, pe, err)
+		return 0, fmt.Errorf("token %s: writing '%s': %w", token, pe, err)
 	}
 
-	return nil
+	// Get the newly-created trace event's id.
+	tid, err := getTraceEventID(args.group, args.symbol)
+	if args.retprobeMaxActive != 0 && errors.Is(err, os.ErrNotExist) {
+		// Kernels < 4.12 don't support maxactive and therefore auto generate
+		// group and event names from the symbol and offset. The symbol is used
+		// without any sanitization.
+		// See https://elixir.bootlin.com/linux/v4.10/source/kernel/trace/trace_kprobe.c#L712
+		event := fmt.Sprintf("kprobes/r_%s_%d", args.symbol, args.offset)
+		if err := removeTraceFSProbeEvent(typ, event); err != nil {
+			return 0, fmt.Errorf("failed to remove spurious maxactive event: %s", err)
+		}
+		return 0, fmt.Errorf("create trace event with non-default maxactive: %w", ErrNotSupported)
+	}
+	if err != nil {
+		return 0, fmt.Errorf("get trace event id: %w", err)
+	}
+
+	return tid, nil
 }
 
 // closeTraceFSProbeEvent removes the [k,u]probe with the given type, group and symbol
 // from <tracefs>/[k,u]probe_events.
 func closeTraceFSProbeEvent(typ probeType, group, symbol string) error {
+	pe := fmt.Sprintf("%s/%s", group, sanitizeSymbol(symbol))
+	return removeTraceFSProbeEvent(typ, pe)
+}
+
+func removeTraceFSProbeEvent(typ probeType, pe string) error {
 	f, err := os.OpenFile(typ.EventsPath(), os.O_APPEND|os.O_WRONLY, 0666)
 	if err != nil {
 		return fmt.Errorf("error opening %s: %w", typ.EventsPath(), err)
@@ -487,9 +523,8 @@ func closeTraceFSProbeEvent(typ probeType, group, symbol string) error {
 
 	// See [k,u]probe_events syntax above. The probe type does not need to be specified
 	// for removals.
-	pe := fmt.Sprintf("-:%s/%s", group, sanitizeSymbol(symbol))
-	if _, err = f.WriteString(pe); err != nil {
-		return fmt.Errorf("writing '%s' to '%s': %w", pe, typ.EventsPath(), err)
+	if _, err = f.WriteString("-:" + pe); err != nil {
+		return fmt.Errorf("remove event %q from %s: %w", pe, typ.EventsPath(), err)
 	}
 
 	return nil
@@ -517,8 +552,11 @@ func randomGroup(prefix string) (string, error) {
 	return group, nil
 }
 
-func probePrefix(ret bool) string {
+func probePrefix(ret bool, maxActive int) string {
 	if ret {
+		if maxActive > 0 {
+			return fmt.Sprintf("r%d", maxActive)
+		}
 		return "r"
 	}
 	return "p"
diff --git a/link/perf_event.go b/link/perf_event.go
index ab2f049..61f8062 100644
--- a/link/perf_event.go
+++ b/link/perf_event.go
@@ -276,7 +276,7 @@ func getTraceEventID(group, name string) (uint64, error) {
 	}
 	tid, err := readUint64FromFile("%d\n", path)
 	if errors.Is(err, os.ErrNotExist) {
-		return 0, fmt.Errorf("trace event %s/%s: %w", group, name, os.ErrNotExist)
+		return 0, err
 	}
 	if err != nil {
 		return 0, fmt.Errorf("reading trace event ID of %s/%s: %w", group, name, err)


Test patch:
diff --git a/link/kprobe_test.go b/link/kprobe_test.go
index e9ba12a..9c01e43 100644
--- a/link/kprobe_test.go
+++ b/link/kprobe_test.go
@@ -10,6 +10,7 @@ import (
 
 	"github.com/cilium/ebpf"
 	"github.com/cilium/ebpf/asm"
+	"github.com/cilium/ebpf/internal"
 	"github.com/cilium/ebpf/internal/testutils"
 	"github.com/cilium/ebpf/internal/unix"
 )
@@ -70,6 +71,30 @@ func TestKprobeOffset(t *testing.T) {
 	t.Fatal("Can't attach with non-zero offset")
 }
 
+func TestKretprobeMaxActive(t *testing.T) {
+	prog := mustLoadProgram(t, ebpf.Kprobe, 0, "")
+
+	k, err := Kprobe("do_sys_open", prog, &KprobeOptions{RetprobeMaxActive: 4096})
+	if !errors.Is(err, errInvalidMaxActive) {
+		t.Fatal("Expected errInvalidMaxActive, got", err)
+	}
+	if k != nil {
+		k.Close()
+	}
+
+	k, err = Kretprobe("__put_task_struct", prog, &KprobeOptions{RetprobeMaxActive: 4096})
+	if testutils.MustKernelVersion().Less(internal.Version{4, 12, 0}) {
+		if !errors.Is(err, ErrNotSupported) {
+			t.Fatal("Expected ErrNotSupported, got", err)
+		}
+	} else if err != nil {
+		t.Fatal("Kretprobe with maxactive returned an error:", err)
+	}
+	if k != nil {
+		k.Close()
+	}
+}
+
 func TestKretprobe(t *testing.T) {
 	prog := mustLoadProgram(t, ebpf.Kprobe, 0, "")
 
@@ -210,13 +235,13 @@ func TestKprobeTraceFS(t *testing.T) {
 	args := probeArgs{group: "testgroup", symbol: "symbol"}
 
 	// Write a k(ret)probe event for a non-existing symbol.
-	err = createTraceFSProbeEvent(kprobeType, args)
+	_, err = createTraceFSProbeEvent(kprobeType, args)
 	c.Assert(errors.Is(err, os.ErrNotExist), qt.IsTrue, qt.Commentf("got error: %s", err))
 
 	// A kernel bug was fixed in 97c753e62e6c where EINVAL was returned instead
 	// of ENOENT, but only for kretprobes.
 	args.ret = true
-	err = createTraceFSProbeEvent(kprobeType, args)
+	_, err = createTraceFSProbeEvent(kprobeType, args)
 	if !(errors.Is(err, os.ErrNotExist) || errors.Is(err, unix.EINVAL)) {
 		t.Fatal(err)
 	}
@@ -238,12 +263,7 @@ func BenchmarkKprobeCreateTraceFS(b *testing.B) {
 }
 
 // Test k(ret)probe creation writing directly to <tracefs>/kprobe_events.
-// Only runs on 5.0 and over. Earlier versions ignored writes of duplicate
-// events, while 5.0 started returning -EEXIST when a kprobe event already
-// exists.
 func TestKprobeCreateTraceFS(t *testing.T) {
-	testutils.SkipOnOldKernel(t, "5.0", "<tracefs>/kprobe_events doesn't reject duplicate events")
-
 	c := qt.New(t)
 
 	pg, _ := randomGroup("ebpftest")
@@ -259,12 +279,12 @@ func TestKprobeCreateTraceFS(t *testing.T) {
 	args := probeArgs{group: pg, symbol: ksym}
 
 	// Create a kprobe.
-	err := createTraceFSProbeEvent(kprobeType, args)
+	_, err := createTraceFSProbeEvent(kprobeType, args)
 	c.Assert(err, qt.IsNil)
 
 	// Attempt to create an identical kprobe using tracefs,
 	// expect it to fail with os.ErrExist.
-	err = createTraceFSProbeEvent(kprobeType, args)
+	_, err = createTraceFSProbeEvent(kprobeType, args)
 	c.Assert(errors.Is(err, os.ErrExist), qt.IsTrue,
 		qt.Commentf("expected consecutive kprobe creation to contain os.ErrExist, got: %v", err))
 
@@ -275,10 +295,10 @@ func TestKprobeCreateTraceFS(t *testing.T) {
 	args.ret = true
 
 	// Same test for a kretprobe.
-	err = createTraceFSProbeEvent(kprobeType, args)
+	_, err = createTraceFSProbeEvent(kprobeType, args)
 	c.Assert(err, qt.IsNil)
 
-	err = createTraceFSProbeEvent(kprobeType, args)
+	_, err = createTraceFSProbeEvent(kprobeType, args)
 	c.Assert(os.IsExist(err), qt.IsFalse,
 		qt.Commentf("expected consecutive kretprobe creation to contain os.ErrExist, got: %v", err))
 
diff --git a/link/uprobe_test.go b/link/uprobe_test.go
index f0ef040..3403e7c 100644
--- a/link/uprobe_test.go
+++ b/link/uprobe_test.go
@@ -245,12 +245,7 @@ func TestUprobeTraceFS(t *testing.T) {
 }
 
 // Test u(ret)probe creation writing directly to <tracefs>/uprobe_events.
-// Only runs on 5.0 and over. Earlier versions ignored writes of duplicate
-// events, while 5.0 started returning -EEXIST when a uprobe event already
-// exists.
 func TestUprobeCreateTraceFS(t *testing.T) {
-	testutils.SkipOnOldKernel(t, "5.0", "<tracefs>/uprobe_events doesn't reject duplicate events")
-
 	c := qt.New(t)
 
 	// Fetch the offset from the /bin/bash Executable already defined.
@@ -278,26 +273,26 @@ func TestUprobeCreateTraceFS(t *testing.T) {
 	}
 
 	// Create a uprobe.
-	err = createTraceFSProbeEvent(uprobeType, args)
+	_, err = createTraceFSProbeEvent(uprobeType, args)
 	c.Assert(err, qt.IsNil)
 
 	// Attempt to create an identical uprobe using tracefs,
 	// expect it to fail with os.ErrExist.
-	err = createTraceFSProbeEvent(uprobeType, args)
+	_, err = createTraceFSProbeEvent(uprobeType, args)
 	c.Assert(errors.Is(err, os.ErrExist), qt.IsTrue,
 		qt.Commentf("expected consecutive uprobe creation to contain os.ErrExist, got: %v", err))
 
-	// Expect a successful close of the kprobe.
+	// Expect a successful close of the uprobe.
 	c.Assert(closeTraceFSProbeEvent(uprobeType, pg, ssym), qt.IsNil)
 
 	args.group = rg
 	args.ret = true
 
 	// Same test for a kretprobe.
-	err = createTraceFSProbeEvent(uprobeType, args)
+	_, err = createTraceFSProbeEvent(uprobeType, args)
 	c.Assert(err, qt.IsNil)
 
-	err = createTraceFSProbeEvent(uprobeType, args)
+	_, err = createTraceFSProbeEvent(uprobeType, args)
 	c.Assert(os.IsExist(err), qt.IsFalse,
 		qt.Commentf("expected consecutive uretprobe creation to contain os.ErrExist, got: %v", err))