Repository: cilium/ebpf
Pull request #265 (2021-03-25 17:18:42)

K(ret)probe and Tracepoint support
This is a continuation of https://github.com/cilium/ebpf/pull/186. Finally got a Round Tuit!

I believe this addresses one of the bigger shortcomings of the lib (as demonstrated by the verbose tracepoint/kprobe examples that were merged recently), and has been written about in more than a few blog posts over the years. (recent example: https://est357.github.io/posts/cilium_iovisor) There was an issue in the old newtools repo for this that was lost in the migration, so it kind of fell off my radar.

After reaching out to the more prominent/active users of the go-perf package (https://github.com/acln0/perf/issues/30), we've concluded that having this in the lib might be preferable, especially given the tiny subset of go-perf functionality we actually need for ebpf. Additionally, go-perf still doesn't provide the more thorny bits around handling tracefs kprobe_events, so we'd have to implement that here anyway, all while still relying on an external dependency. (whether it eventually makes it into the stdlib or not)

Putting this out here for a first review round as the first 90% is done, but I still need to convert the examples in order to evaluate if the package is fit for external consumption. Having some early feedback would be great. Getting the terminology right was the hardest part, I've found the body of existing libraries and documentation to be inconsistent.

@ungureanuvladvictor uprobe support should fit into this structure nicely, and will be added in a follow-up.

## To Do
- [x] Convert `examples/`
- [ ] Add Go `Example*`s in tests

cc fyi @lmb @mmat11 @florianl @buglloc 

florianl: The problem I see at the moment with the proposed approach is, that it limits the power of eBPF on k(ret)probe and tracepoints. E.g. it can be useful to enable these eBPF programs on perf event just on certain CPU cores or for particular processes (e.g. a syscall monitor for a particular process). Or in case of a sampling approach to disable and enable it again.
As pointed out by @hodgesds in https://github.com/acln0/perf/issues/30 the perf system can become complex (and time consuming to maintain) quite fast. And so far I don't know of a high level API in Go that covers 90% of use cases. 
Where do you see the limits that should be supported by this proposal and use cases that should use `golang.org/x/sys/unix` directly? I think it is important to document these differences and make the user aware of it.
ti-mo: Hi @florianl, thanks for taking a look! This is exactly the kind of feedback I'm looking for. The use cases you've given here are still incredibly narrow compared to the perf API surface. Could you think of any use cases that this implementation completely lacks?

> enable these eBPF programs on perf event just on certain CPU cores

Okay, this convinced me to add a `*LinkOpts` argument to `Tracepoint()`, `Kprobe()` etc. to allow the users to turn a few knobs. What you're describing is common in uprobes, correct? Could you give a more practical example where this is used?

> a sampling approach to disable and enable it again

This could be addressed by expanding the `Link` interface to have an `Enable()` and `Disable()` and implementing them on `perfEvent`. WDYT @lmb, would this make sense for other `Link` types?

> Where do you see the limits that should be supported by this proposal and use cases that should use golang.org/x/sys/unix directly?

I feel like users that know they need something radically different would've reached for `x/sys/unix` long ago. The perf_kprobe PMU stuff is done nearly identically in libbpf, and tracefs usage I've seen floating around is all 99% similar to what's in the proposal. (looking at the kernel code, the intersection of bpf+perf is but a tiny fraction of the perf API as a whole)

I think the limits of the package are immediately clear by looking at the API surface: create, attach and close tracepoints, k(ret)probes and uprobes (soon). As the community discovers more use cases, they will be exposed in the package and added to `examples/`. (as well as being presented in a few blog posts on ebpf.io)

>  As pointed out (...) the perf system can become complex (and time consuming to maintain) quite fast. And so far I don't know of a high level API in Go that covers 90% of use cases.

We advertise this as an eBPF API, not a perf API. :slightly_smiling_face: As I saw the same 2 syscalls and 3 ioctls enabling most use cases, I felt it would make sense to maintain them as a coherent set. I think this proposal walks the line between BCC's "load && attach all the things with a single call!" and users having to reach for `runtime.KeepAlive` and `unsafe` for simply attaching a kprobe. It hides the thorny bits and does the heavy lifting while still providing the freedom to choose when/how things are hooked together. It doesn't feel like magic, which is pretty much in line with the rest of the library.
hodgesds: This [patch](https://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git/commit/?id=7fac83aaf2eecc9e) might be interesting in this space as well.
ti-mo: I think this is ready for wider review. Updated examples are here https://github.com/ti-mo/ebpf/commit/7deb85b2c66a00abdef30f4874b060e80c5f42b9 and will follow after this has been merged. (go.mod constraints)

@kkourt @jrfastab What do you think about the API? Anything that should be added?
lmb: > > the perf system can become complex (and time consuming to maintain) quite fast. And so far I don't know of a high level API in Go that covers 90% of use cases.
>  
>     * Do we need support to run on a subset of CPU? I'd say no: the BPF can call `bpf_get_smp_processor_id` and do it's own filtering.
> 
>     * Do we need to support turning it on and off again? I'd say no: you can update a map to bail out of your program early.
> 
>     * Do we need to support a subset of processes? Maybe: BPF can call bpf_get_current_pid_tgid to do it's own filtering. Are there are permission implications?

Maybe this comment got lost, what are we going to do about these?
lmb: Another thing that occurred to me too late:

Current API:
```
tp, _ := Tracepoint(...)
link, _ := tp.Attach(prog)
```

What if we did this instead:
```
link, _ := Tracepoint(..., prog)
```

Pros:
* Shorter in the common case
* Works like the other links
* TraceEvent isn't exported
* "Simpler"

Cons:
* Can't attach multiple programs via a single TraceEvent. Does that ever happen though?

This might also make implementing `perfEvent.Update()` in the future simpler:
```
func (pe) Update(*Program) error {
    // create new TraceEvent in the background
    // attach program
    // close old TraceEvent, which removes the old bpf program (???)
}
```
ti-mo: `Tracepoint()`, `Kprobe()` and `Kretprobe()` now take group/name/symbol and `*ebpf.Program`, no longer burdening the caller with keeping track of `TraceEvent` objects to correctly manage their lifecycle. Closing a kprobe `TraceEvent` early on kernels pre-4.17 could've been a source of subtle bugs. The new API design maps better onto PMU perf events, which is the future anyway.

Patch:
diff --git a/internal/unix/types_linux.go b/internal/unix/types_linux.go
index 2530e60..f0a9672 100644
--- a/internal/unix/types_linux.go
+++ b/internal/unix/types_linux.go
@@ -40,7 +40,11 @@ const (
 	PROT_WRITE               = linux.PROT_WRITE
 	MAP_SHARED               = linux.MAP_SHARED
 	PERF_TYPE_SOFTWARE       = linux.PERF_TYPE_SOFTWARE
+	PERF_TYPE_TRACEPOINT     = linux.PERF_TYPE_TRACEPOINT
 	PERF_COUNT_SW_BPF_OUTPUT = linux.PERF_COUNT_SW_BPF_OUTPUT
+	PERF_EVENT_IOC_DISABLE   = linux.PERF_EVENT_IOC_DISABLE
+	PERF_EVENT_IOC_ENABLE    = linux.PERF_EVENT_IOC_ENABLE
+	PERF_EVENT_IOC_SET_BPF   = linux.PERF_EVENT_IOC_SET_BPF
 	PerfBitWatermark         = linux.PerfBitWatermark
 	PERF_SAMPLE_RAW          = linux.PERF_SAMPLE_RAW
 	PERF_FLAG_FD_CLOEXEC     = linux.PERF_FLAG_FD_CLOEXEC
@@ -72,6 +76,11 @@ func FcntlInt(fd uintptr, cmd, arg int) (int, error) {
 	return linux.FcntlInt(fd, cmd, arg)
 }
 
+// IoctlSetInt is a wrapper
+func IoctlSetInt(fd int, req uint, value int) error {
+	return linux.IoctlSetInt(fd, req, value)
+}
+
 // Statfs is a wrapper
 func Statfs(path string, buf *Statfs_t) (err error) {
 	return linux.Statfs(path, buf)
diff --git a/internal/unix/types_other.go b/internal/unix/types_other.go
index 250127a..9ee460e 100644
--- a/internal/unix/types_other.go
+++ b/internal/unix/types_other.go
@@ -41,7 +41,11 @@ const (
 	PROT_WRITE               = 0x2
 	MAP_SHARED               = 0x1
 	PERF_TYPE_SOFTWARE       = 0x1
+	PERF_TYPE_TRACEPOINT     = 0
 	PERF_COUNT_SW_BPF_OUTPUT = 0xa
+	PERF_EVENT_IOC_DISABLE   = 0
+	PERF_EVENT_IOC_ENABLE    = 0
+	PERF_EVENT_IOC_SET_BPF   = 0
 	PerfBitWatermark         = 0x4000
 	PERF_SAMPLE_RAW          = 0x400
 	PERF_FLAG_FD_CLOEXEC     = 0x8
@@ -89,6 +93,11 @@ func FcntlInt(fd uintptr, cmd, arg int) (int, error) {
 	return -1, errNonLinux
 }
 
+// IoctlSetInt is a wrapper
+func IoctlSetInt(fd int, req uint, value int) error {
+	return errNonLinux
+}
+
 // Statfs is a wrapper
 func Statfs(path string, buf *Statfs_t) error {
 	return errNonLinux
diff --git a/link/kprobe.go b/link/kprobe.go
new file mode 100644
index 0000000..07a0dd0
--- /dev/null
+++ b/link/kprobe.go
@@ -0,0 +1,295 @@
+package link
+
+import (
+	"crypto/rand"
+	"errors"
+	"fmt"
+	"os"
+	"path/filepath"
+	"runtime"
+
+	"github.com/cilium/ebpf"
+	"github.com/cilium/ebpf/internal"
+	"github.com/cilium/ebpf/internal/unix"
+)
+
+var (
+	kprobeEventsPath = filepath.Join(tracefsPath, "kprobe_events")
+)
+
+// Kprobe attaches the given eBPF program to a perf event that fires when the
+// given kernel symbol starts executing. See /proc/kallsyms for available
+// symbols. For example, printk():
+//
+//	Kprobe("printk")
+//
+// The resulting Link must be Closed during program shutdown to avoid leaking
+// system resources.
+func Kprobe(symbol string, prog *ebpf.Program) (Link, error) {
+	k, err := kprobe(symbol, prog, false)
+	if err != nil {
+		return nil, err
+	}
+
+	err = k.attach(prog)
+	if err != nil {
+		k.Close()
+		return nil, err
+	}
+
+	return k, nil
+}
+
+// Kretprobe attaches the given eBPF program to a perf event that fires right
+// before the given kernel symbol exits, with the function stack left intact.
+// See /proc/kallsyms for available symbols. For example, printk():
+//
+//	Kretprobe("printk")
+//
+// The resulting Link must be Closed during program shutdown to avoid leaking
+// system resources.
+func Kretprobe(symbol string, prog *ebpf.Program) (Link, error) {
+	k, err := kprobe(symbol, prog, true)
+	if err != nil {
+		return nil, err
+	}
+
+	err = k.attach(prog)
+	if err != nil {
+		k.Close()
+		return nil, err
+	}
+
+	return k, nil
+}
+
+// kprobe opens a perf event on the given symbol and attaches prog to it.
+// If ret is true, create a kretprobe.
+func kprobe(symbol string, prog *ebpf.Program, ret bool) (*perfEvent, error) {
+	if symbol == "" {
+		return nil, fmt.Errorf("symbol name cannot be empty: %w", errInvalidInput)
+	}
+	if prog == nil {
+		return nil, fmt.Errorf("prog cannot be nil: %w", errInvalidInput)
+	}
+	if !rgxTraceEvent.MatchString(symbol) {
+		return nil, fmt.Errorf("symbol '%s' must be alphanumeric or underscore: %w", symbol, errInvalidInput)
+	}
+	if prog.Type() != ebpf.Kprobe {
+		return nil, fmt.Errorf("eBPF program type %s is not a Kprobe: %w", prog.Type(), errInvalidInput)
+	}
+
+	// Use kprobe PMU if the kernel has it available.
+	tp, err := pmuKprobe(symbol, ret)
+	if err == nil {
+		return tp, nil
+	}
+	if err != nil && !errors.Is(err, ErrNotSupported) {
+		return nil, fmt.Errorf("creating perf_kprobe PMU: %w", err)
+	}
+
+	// Use tracefs if kprobe PMU is missing.
+	tp, err = tracefsKprobe(symbol, ret)
+	if err != nil {
+		return nil, fmt.Errorf("creating trace event '%s' in tracefs: %w", symbol, err)
+	}
+
+	return tp, nil
+}
+
+// pmuKprobe opens a perf event based on a Performance Monitoring Unit.
+// Requires at least 4.17 (e12f03d7031a "perf/core: Implement the
+// 'perf_kprobe' PMU").
+// Returns ErrNotSupported if the kernel doesn't support perf_kprobe PMU,
+// or os.ErrNotExist if the given symbol does not exist in the kernel.
+func pmuKprobe(symbol string, ret bool) (*perfEvent, error) {
+
+	// Getting the PMU type will fail if the kernel doesn't support
+	// the perf_kprobe PMU.
+	et, err := getPMUEventType("kprobe")
+	if err != nil {
+		return nil, err
+	}
+
+	// Create a pointer to a NUL-terminated string for the kernel.
+	sp, err := unsafeStringPtr(symbol)
+	if err != nil {
+		return nil, err
+	}
+
+	// TODO: Parse the position of the bit from /sys/bus/event_source/devices/%s/format/retprobe.
+	config := 0
+	if ret {
+		config = 1
+	}
+
+	attr := unix.PerfEventAttr{
+		Type:   uint32(et),          // PMU event type read from sysfs
+		Ext1:   uint64(uintptr(sp)), // Kernel symbol to trace
+		Config: uint64(config),      // perf_kprobe PMU treats config as flags
+	}
+
+	fd, err := unix.PerfEventOpen(&attr, perfAllThreads, 0, -1, unix.PERF_FLAG_FD_CLOEXEC)
+
+	// Since commit 97c753e62e6c, ENOENT is correctly returned instead of EINVAL
+	// when trying to create a kretprobe for a missing symbol. Make sure ENOENT
+	// is returned to the caller.
+	if errors.Is(err, os.ErrNotExist) || errors.Is(err, unix.EINVAL) {
+		return nil, fmt.Errorf("symbol '%s' not found: %w", symbol, os.ErrNotExist)
+	}
+	if err != nil {
+		return nil, fmt.Errorf("opening perf event: %w", err)
+	}
+
+	// Ensure the string pointer is not collected before PerfEventOpen returns.
+	runtime.KeepAlive(sp)
+
+	// Kernel has perf_kprobe PMU available, initialize perf event.
+	return &perfEvent{
+		fd:       internal.NewFD(uint32(fd)),
+		pmuID:    et,
+		name:     symbol,
+		ret:      ret,
+		progType: ebpf.Kprobe,
+	}, nil
+}
+
+// tracefsKprobe creates a trace event by writing an entry to <tracefs>/kprobe_events.
+// A new trace event group name is generated on every call to support creating
+// multiple trace events for the same kernel symbol. A perf event is then opened
+// on the newly-created trace event and returned to the caller.
+func tracefsKprobe(symbol string, ret bool) (*perfEvent, error) {
+
+	// Generate a random string for each trace event we attempt to create.
+	// This value is used as the 'group' token in tracefs to allow creating
+	// multiple kprobe trace events with the same name.
+	group, err := randomGroup("ebpf")
+	if err != nil {
+		return nil, fmt.Errorf("randomizing group name: %w", err)
+	}
+
+	// Before attempting to create a trace event through tracefs,
+	// check if an event with the same group and name already exists.
+	// Kernels 4.x and earlier don't return os.ErrExist on writing a duplicate
+	// entry, so we need to rely on reads for detecting uniqueness.
+	_, err = getTraceEventID(group, symbol)
+	if err == nil {
+		return nil, fmt.Errorf("trace event already exists: %s/%s", group, symbol)
+	}
+	// The read is expected to fail with ErrNotSupported due to a non-existing event.
+	if err != nil && !errors.Is(err, ErrNotSupported) {
+		return nil, fmt.Errorf("checking trace event %s/%s: %w", group, symbol, err)
+	}
+
+	// Create the kprobe trace event using tracefs.
+	if err := createTraceFSKprobeEvent(group, symbol, ret); err != nil {
+		return nil, fmt.Errorf("creating kprobe event on tracefs: %w", err)
+	}
+
+	// Get the newly-created trace event's id.
+	tid, err := getTraceEventID(group, symbol)
+	if err != nil {
+		return nil, fmt.Errorf("getting trace event id: %w", err)
+	}
+
+	// Kprobes are ephemeral tracepoints and share the same perf event type.
+	fd, err := openTracepointPerfEvent(tid)
+	if err != nil {
+		return nil, err
+	}
+
+	return &perfEvent{
+		fd:        fd,
+		group:     group,
+		name:      symbol,
+		ret:       ret,
+		tracefsID: tid,
+		progType:  ebpf.Kprobe, // kernel only allows attaching kprobe programs to kprobe events
+	}, nil
+}
+
+// createTraceFSKprobeEvent creates a new ephemeral trace event by writing to
+// <tracefs>/kprobe_events. Returns ErrNotSupported if symbol is not a valid
+// kernel symbol, or if it is not traceable with kprobes.
+func createTraceFSKprobeEvent(group, symbol string, ret bool) error {
+	// Open the kprobe_events file in tracefs.
+	f, err := os.OpenFile(kprobeEventsPath, os.O_APPEND|os.O_WRONLY, 0666)
+	if err != nil {
+		return fmt.Errorf("error opening kprobe_events: %w", err)
+	}
+	defer f.Close()
+
+	// The kprobe_events syntax is as follows (see Documentation/trace/kprobetrace.txt):
+	// p[:[GRP/]EVENT] [MOD:]SYM[+offs]|MEMADDR [FETCHARGS] : Set a probe
+	// r[MAXACTIVE][:[GRP/]EVENT] [MOD:]SYM[+0] [FETCHARGS] : Set a return probe
+	// -:[GRP/]EVENT                                        : Clear a probe
+	//
+	// Some examples:
+	// r:ebpf_1234/r_my_kretprobe nf_conntrack_destroy
+	// p:ebpf_5678/p_my_kprobe __x64_sys_execve
+	//
+	// Leaving the kretprobe's MAXACTIVE set to 0 (or absent) will make the
+	// kernel default to NR_CPUS. This is desired in most eBPF cases since
+	// subsampling or rate limiting logic can be more accurately implemented in
+	// the eBPF program itself. See Documentation/kprobes.txt for more details.
+	pe := fmt.Sprintf("%s:%s/%s %s", kprobePrefix(ret), group, symbol, symbol)
+	_, err = f.WriteString(pe)
+	// Writing to <tracefs>/kprobe_events will return ENOENT when the tracee
+	// kernel symbol does not exist (yet) in the kernel.
+	if os.IsNotExist(err) {
+		return fmt.Errorf("kernel symbol %s not found: %w", symbol, ErrNotSupported)
+	}
+	if err != nil {
+		return fmt.Errorf("writing '%s' to kprobe_events: %w", pe, err)
+	}
+
+	return nil
+}
+
+// closeTraceFSKprobeEvent removes the kprobe with the given group, symbol and kind
+// from <tracefs>/kprobe_events.
+func closeTraceFSKprobeEvent(group, symbol string) error {
+	f, err := os.OpenFile(kprobeEventsPath, os.O_APPEND|os.O_WRONLY, 0666)
+	if err != nil {
+		return fmt.Errorf("error opening kprobe_events: %w", err)
+	}
+	defer f.Close()
+
+	// See kprobe_events syntax above. Kprobe type does not need to be specified
+	// for removals.
+	pe := fmt.Sprintf("-:%s/%s", group, symbol)
+	if _, err = f.WriteString(pe); err != nil {
+		return fmt.Errorf("writing '%s' to kprobe_events: %w", pe, err)
+	}
+
+	return nil
+}
+
+// randomGroup generates a pseudorandom string for use as a tracefs group name.
+// Returns an error when the output string would exceed 63 characters (kernel
+// limitation), when rand.Read() fails or when prefix contains characters not
+// allowed by rgxTraceEvent.
+func randomGroup(prefix string) (string, error) {
+	if !rgxTraceEvent.MatchString(prefix) {
+		return "", fmt.Errorf("prefix '%s' must be alphanumeric or underscore: %w", prefix, errInvalidInput)
+	}
+
+	b := make([]byte, 8)
+	if _, err := rand.Read(b); err != nil {
+		return "", fmt.Errorf("reading random bytes: %w", err)
+	}
+
+	group := fmt.Sprintf("%s_%x", prefix, b)
+	if len(group) > 63 {
+		return "", fmt.Errorf("group name '%s' cannot be longer than 63 characters: %w", group, errInvalidInput)
+	}
+
+	return group, nil
+}
+
+func kprobePrefix(ret bool) string {
+	if ret {
+		return "r"
+	}
+	return "p"
+}
diff --git a/link/perf_event.go b/link/perf_event.go
new file mode 100644
index 0000000..b27c183
--- /dev/null
+++ b/link/perf_event.go
@@ -0,0 +1,246 @@
+package link
+
+import (
+	"bytes"
+	"errors"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"regexp"
+	"strconv"
+	"strings"
+	"unsafe"
+
+	"github.com/cilium/ebpf"
+	"github.com/cilium/ebpf/internal"
+	"github.com/cilium/ebpf/internal/unix"
+)
+
+// Getting the terminology right is usually the hardest part. For posterity and
+// for staying sane during implementation:
+//
+// - trace event: Representation of a kernel runtime hook. Filesystem entries
+//   under <tracefs>/events. Can be tracepoints (static), kprobes or uprobes.
+//   Can be instantiated into perf events (see below).
+// - tracepoint: A predetermined hook point in the kernel. Exposed as trace
+//   events in (sub)directories under <tracefs>/events. Cannot be closed or
+//   removed, they are static.
+// - k(ret)probe: Ephemeral trace events based on entry or exit points of
+//   exported kernel symbols. kprobe-based (tracefs) trace events can be
+//   created system-wide by writing to the <tracefs>/kprobe_events file, or
+//   they can be scoped to the current process by creating PMU perf events.
+// - perf event: An object instantiated based on an existing trace event or
+//   kernel symbol. Referred to by fd in userspace.
+//   Exactly one eBPF program can be attached to a perf event. Multiple perf
+//   events can be created from a single trace event. Closing a perf event
+//   stops any further invocations of the attached eBPF program.
+
+var (
+	tracefsPath = "/sys/kernel/debug/tracing"
+
+	// Trace event groups, names and kernel symbols must adhere to this set
+	// of characters. Non-empty, first character must not be a number, all
+	// characters must be alphanumeric or underscore.
+	rgxTraceEvent = regexp.MustCompile("^[a-zA-Z_][0-9a-zA-Z_]*$")
+
+	errInvalidInput = errors.New("invalid input")
+)
+
+const (
+	perfAllThreads = -1
+)
+
+// A perfEvent represents a perf event kernel object. Exactly one eBPF program
+// can be attached to it. It is created based on a tracefs trace event or a
+// Performance Monitoring Unit (PMU).
+type perfEvent struct {
+
+	// Group and name of the tracepoint/kprobe/uprobe.
+	group string
+	name  string
+
+	// PMU event ID read from sysfs. Valid IDs are non-zero.
+	pmuID uint64
+	// ID of the trace event read from tracefs. Valid IDs are non-zero.
+	tracefsID uint64
+
+	// True for kretprobes/uretprobes.
+	ret bool
+
+	fd       *internal.FD
+	progType ebpf.ProgramType
+}
+
+func (pe *perfEvent) isLink() {}
+
+func (pe *perfEvent) Pin(string) error {
+	return fmt.Errorf("pin perf event: %w", ErrNotSupported)
+}
+
+// Since 4.15 (e87c6bc3852b "bpf: permit multiple bpf attachments for a single perf event"),
+// calling PERF_EVENT_IOC_SET_BPF appends the given program to a prog_array
+// owned by the perf event, which means multiple programs can be attached
+// simultaneously.
+//
+// Before 4.15, calling PERF_EVENT_IOC_SET_BPF more than once on a perf event
+// returns EEXIST.
+//
+// Detaching a program from a perf event is currently not possible, so a
+// program replacement mechanism cannot be implemented for perf events.
+func (pe *perfEvent) Update(prog *ebpf.Program) error {
+	return fmt.Errorf("can't replace eBPF program in perf event: %w", ErrNotSupported)
+}
+
+func (pe *perfEvent) Close() error {
+	if pe.fd == nil {
+		return nil
+	}
+
+	pfd, err := pe.fd.Value()
+	if err != nil {
+		return fmt.Errorf("getting perf event fd: %w", err)
+	}
+
+	err = unix.IoctlSetInt(int(pfd), unix.PERF_EVENT_IOC_DISABLE, 0)
+	if err != nil {
+		return fmt.Errorf("disabling perf event: %w", err)
+	}
+
+	err = pe.fd.Close()
+	if err != nil {
+		return fmt.Errorf("closing perf event fd: %w", err)
+	}
+
+	switch t := pe.progType; t {
+	case ebpf.Kprobe:
+		// For kprobes created using tracefs, clean up the <tracefs>/kprobe_events entry.
+		if pe.tracefsID != 0 {
+			return closeTraceFSKprobeEvent(pe.group, pe.name)
+		}
+	case ebpf.TracePoint:
+		// Tracepoint trace events don't hold any extra resources.
+		return nil
+	}
+
+	return nil
+}
+
+// attach the given eBPF prog to the perf event stored in pe.
+// pe must contain a valid perf event fd.
+// prog's type must match the program type stored in pe.
+func (pe *perfEvent) attach(prog *ebpf.Program) error {
+	if prog == nil {
+		return errors.New("cannot attach a nil program")
+	}
+	if pe.fd == nil {
+		return errors.New("cannot attach to nil perf event")
+	}
+	if t := prog.Type(); t != pe.progType {
+		return fmt.Errorf("invalid program type (expected %s): %s", pe.progType, t)
+	}
+	if prog.FD() < 0 {
+		return fmt.Errorf("invalid program: %w", internal.ErrClosedFd)
+	}
+
+	// The ioctl below will fail when the fd is invalid.
+	kfd, _ := pe.fd.Value()
+
+	// Assign the eBPF program to the perf event.
+	err := unix.IoctlSetInt(int(kfd), unix.PERF_EVENT_IOC_SET_BPF, prog.FD())
+	if err != nil {
+		return fmt.Errorf("setting perf event bpf program: %w", err)
+	}
+
+	// PERF_EVENT_IOC_ENABLE and _DISABLE ignore their given values.
+	if err := unix.IoctlSetInt(int(kfd), unix.PERF_EVENT_IOC_ENABLE, 0); err != nil {
+		return fmt.Errorf("enable perf event: %s", err)
+	}
+
+	return nil
+}
+
+// unsafeStringPtr returns an unsafe.Pointer to a NUL-terminated copy of str.
+func unsafeStringPtr(str string) (unsafe.Pointer, error) {
+	p, err := unix.BytePtrFromString(str)
+	if err != nil {
+		return nil, err
+	}
+	return unsafe.Pointer(p), nil
+}
+
+// getTraceEventID reads a trace event's ID from tracefs given its group and name.
+// group and name must be alphanumeric or underscore, as required by the kernel.
+func getTraceEventID(group, name string) (uint64, error) {
+	tid, err := uint64FromFile(tracefsPath, "events", group, name, "id")
+	if errors.Is(err, ErrNotSupported) {
+		return 0, fmt.Errorf("trace event %s/%s: %w", group, name, ErrNotSupported)
+	}
+	if err != nil {
+		return 0, fmt.Errorf("reading trace event ID of %s/%s: %w", group, name, err)
+	}
+
+	return tid, nil
+}
+
+// getPMUEventType reads a Performance Monitoring Unit's type (numeric identifier)
+// from /sys/bus/event_source/devices/<pmu>/type.
+func getPMUEventType(pmu string) (uint64, error) {
+	et, err := uint64FromFile("/sys/bus/event_source/devices", pmu, "type")
+	if errors.Is(err, ErrNotSupported) {
+		return 0, fmt.Errorf("pmu type %s: %w", pmu, ErrNotSupported)
+	}
+	if err != nil {
+		return 0, fmt.Errorf("reading pmu type %s: %w", pmu, err)
+	}
+
+	return et, nil
+}
+
+// openTracepointPerfEvent opens a tracepoint-type perf event. System-wide
+// kprobes created by writing to <tracefs>/kprobe_events are tracepoints
+// behind the scenes, and can be attached to using these perf events.
+func openTracepointPerfEvent(tid uint64) (*internal.FD, error) {
+	attr := unix.PerfEventAttr{
+		Type:        unix.PERF_TYPE_TRACEPOINT,
+		Config:      tid,
+		Sample_type: unix.PERF_SAMPLE_RAW,
+		Sample:      1,
+		Wakeup:      1,
+	}
+
+	fd, err := unix.PerfEventOpen(&attr, perfAllThreads, 0, -1, unix.PERF_FLAG_FD_CLOEXEC)
+	if err != nil {
+		return nil, fmt.Errorf("opening tracepoint perf event: %w", err)
+	}
+
+	return internal.NewFD(uint32(fd)), nil
+}
+
+// uint64FromFile reads a uint64 from a file. All elements of path are sanitized
+// and joined onto base. Returns error if base no longer prefixes the path after
+// joining all components.
+func uint64FromFile(base string, path ...string) (uint64, error) {
+
+	// Resolve leaf path separately for error feedback. Makes the join onto
+	// base more readable (can't mix with variadic args).
+	l := filepath.Join(path...)
+
+	p := filepath.Join(base, l)
+	if !strings.HasPrefix(p, base) {
+		return 0, fmt.Errorf("path '%s' attempts to escape base path '%s': %w", l, base, errInvalidInput)
+	}
+
+	data, err := ioutil.ReadFile(p)
+	if os.IsNotExist(err) {
+		// Only echo leaf path, the base path can be prepended at the call site
+		// if more verbosity is required.
+		return 0, fmt.Errorf("symbol %s: %w", l, ErrNotSupported)
+	}
+	if err != nil {
+		return 0, fmt.Errorf("reading file %s: %w", p, err)
+	}
+
+	et := bytes.TrimSpace(data)
+	return strconv.ParseUint(string(et), 10, 64)
+}
diff --git a/link/tracepoint.go b/link/tracepoint.go
new file mode 100644
index 0000000..fe6af7f
--- /dev/null
+++ b/link/tracepoint.go
@@ -0,0 +1,55 @@
+package link
+
+import (
+	"fmt"
+
+	"github.com/cilium/ebpf"
+)
+
+// Tracepoint attaches the given eBPF program to the tracepoint with the given
+// group and name. See /sys/kernel/debug/tracing/events to find available
+// tracepoints. The top-level directory is the group, the event's subdirectory
+// is the name. Example:
+//
+//	Tracepoint("syscalls", "sys_enter_fork")
+//
+// Note that attaching eBPF programs to syscalls (sys_enter_*/sys_exit_*) is
+// only possible as of kernel 4.14 (commit cf5f5ce).
+func Tracepoint(group, name string, prog *ebpf.Program) (Link, error) {
+	if group == "" || name == "" {
+		return nil, fmt.Errorf("group and name cannot be empty: %w", errInvalidInput)
+	}
+	if prog == nil {
+		return nil, fmt.Errorf("prog cannot be nil: %w", errInvalidInput)
+	}
+	if !rgxTraceEvent.MatchString(group) || !rgxTraceEvent.MatchString(name) {
+		return nil, fmt.Errorf("group and name '%s/%s' must be alphanumeric or underscore: %w", group, name, errInvalidInput)
+	}
+	if prog.Type() != ebpf.TracePoint {
+		return nil, fmt.Errorf("eBPF program type %s is not a Tracepoint: %w", prog.Type(), errInvalidInput)
+	}
+
+	tid, err := getTraceEventID(group, name)
+	if err != nil {
+		return nil, err
+	}
+
+	fd, err := openTracepointPerfEvent(tid)
+	if err != nil {
+		return nil, err
+	}
+
+	pe := &perfEvent{
+		fd:        fd,
+		tracefsID: tid,
+		group:     group,
+		name:      name,
+		progType:  ebpf.TracePoint,
+	}
+
+	if err := pe.attach(prog); err != nil {
+		return nil, err
+	}
+
+	return pe, nil
+}


Test patch:
diff --git a/link/kprobe_test.go b/link/kprobe_test.go
new file mode 100644
index 0000000..0015a5e
--- /dev/null
+++ b/link/kprobe_test.go
@@ -0,0 +1,274 @@
+package link
+
+import (
+	"errors"
+	"os"
+	"testing"
+
+	qt "github.com/frankban/quicktest"
+
+	"github.com/cilium/ebpf"
+	"github.com/cilium/ebpf/asm"
+	"github.com/cilium/ebpf/internal/testutils"
+)
+
+var (
+	kprobeSpec = ebpf.ProgramSpec{
+		Type:    ebpf.Kprobe,
+		License: "MIT",
+		Instructions: asm.Instructions{
+			// set exit code to 0
+			asm.Mov.Imm(asm.R0, 0),
+			asm.Return(),
+		},
+	}
+)
+
+func TestKprobe(t *testing.T) {
+
+	//TODO: implement kprobe program version rewriting for pre-5.0 kernels.
+	// Requires at least 5.0 (6c4fc209fcf9 "bpf: remove useless version check for prog load")
+	testutils.SkipOnOldKernel(t, "5.0", "lifted version check for kprobes")
+
+	c := qt.New(t)
+
+	prog, err := ebpf.NewProgram(&kprobeSpec)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer prog.Close()
+
+	k, err := Kprobe("printk", prog)
+	c.Assert(err, qt.IsNil)
+	defer k.Close()
+
+	testLink(t, k, testLinkOptions{
+		prog: prog,
+	})
+
+	k, err = Kprobe("bogus", prog)
+	c.Assert(errors.Is(err, os.ErrNotExist), qt.IsTrue)
+	if k != nil {
+		k.Close()
+	}
+}
+
+func TestKretprobe(t *testing.T) {
+
+	//TODO: implement kprobe program version rewriting for pre-5.0 kernels.
+	// Requires at least 5.0 (6c4fc209fcf9 "bpf: remove useless version check for prog load")
+	testutils.SkipOnOldKernel(t, "5.0", "lifted version check for kprobes")
+
+	c := qt.New(t)
+
+	prog, err := ebpf.NewProgram(&kprobeSpec)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer prog.Close()
+
+	k, err := Kretprobe("printk", prog)
+	c.Assert(err, qt.IsNil)
+	defer k.Close()
+
+	testLink(t, k, testLinkOptions{
+		prog: prog,
+	})
+
+	k, err = Kretprobe("bogus", prog)
+	c.Assert(errors.Is(err, os.ErrNotExist), qt.IsTrue, qt.Commentf("got error: %s", err))
+	if k != nil {
+		k.Close()
+	}
+}
+
+func TestKprobeErrors(t *testing.T) {
+	c := qt.New(t)
+
+	// Invalid Kprobe incantations. Kretprobe uses the same code paths
+	// with a different ret flag.
+	_, err := Kprobe("", nil) // empty symbol
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+
+	_, err = Kprobe("_", nil) // empty prog
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+
+	_, err = Kprobe(".", &ebpf.Program{}) // illegal chars in symbol
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+
+	_, err = Kprobe("foo", &ebpf.Program{}) // wrong prog type
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+}
+
+// Test k(ret)probe creation using perf_kprobe PMU.
+func TestKprobeCreatePMU(t *testing.T) {
+
+	// Requires at least 4.17 (e12f03d7031a "perf/core: Implement the 'perf_kprobe' PMU")
+	testutils.SkipOnOldKernel(t, "4.17", "perf_kprobe PMU")
+
+	c := qt.New(t)
+
+	// kprobe happy path. printk is always present.
+	pk, err := pmuKprobe("printk", false)
+	c.Assert(err, qt.IsNil)
+	defer pk.Close()
+
+	c.Assert(pk.progType, qt.Equals, ebpf.Kprobe)
+
+	// kretprobe happy path.
+	pr, err := pmuKprobe("printk", true)
+	c.Assert(err, qt.IsNil)
+	defer pr.Close()
+
+	c.Assert(pr.progType, qt.Equals, ebpf.Kprobe)
+
+	// Expect os.ErrNotExist when specifying a non-existent kernel symbol
+	// on kernels 4.17 and up.
+	_, err = pmuKprobe("bogus", false)
+	c.Assert(errors.Is(err, os.ErrNotExist), qt.IsTrue, qt.Commentf("got error: %s", err))
+
+	// A kernel bug was fixed in 97c753e62e6c where EINVAL was returned instead
+	// of ENOENT, but only for kretprobes.
+	_, err = pmuKprobe("bogus", true)
+	c.Assert(errors.Is(err, os.ErrNotExist), qt.IsTrue, qt.Commentf("got error: %s", err))
+}
+
+// Test fallback behaviour on kernels without perf_kprobe PMU available.
+func TestKprobePMUUnavailable(t *testing.T) {
+	c := qt.New(t)
+
+	pk, err := pmuKprobe("printk", false)
+	if err == nil {
+		pk.Close()
+		t.Skipf("Kernel supports perf_kprobe PMU, not asserting error.")
+	}
+
+	// Only allow a PMU creation with a valid kernel symbol to fail with ErrNotSupported.
+	c.Assert(errors.Is(err, ErrNotSupported), qt.IsTrue, qt.Commentf("got error: %s", err))
+}
+
+func BenchmarkKprobeCreatePMU(b *testing.B) {
+	for n := 0; n < b.N; n++ {
+		pr, err := pmuKprobe("printk", false)
+		if err != nil {
+			b.Error("error creating perf_kprobe PMU:", err)
+		}
+
+		if err := pr.Close(); err != nil {
+			b.Error("error closing perf_kprobe PMU:", err)
+		}
+	}
+}
+
+// Test tracefs k(ret)probe creation on all kernel versions.
+// Create two trace events on the same symbol and ensure their IDs differ.
+func TestKprobeTraceFS(t *testing.T) {
+	c := qt.New(t)
+
+	symbol := "printk"
+
+	// Open and close tracefs kprobe, checking all errors.
+	te, err := tracefsKprobe(symbol, false)
+	c.Assert(err, qt.IsNil)
+	c.Assert(te.Close(), qt.IsNil)
+
+	// Create similar trace events, ensure their IDs differ.
+	te1, err := tracefsKprobe(symbol, false)
+	c.Assert(err, qt.IsNil)
+	defer te1.Close()
+	c.Assert(te1.progType, qt.Equals, ebpf.Kprobe)
+	c.Assert(te1.tracefsID, qt.Not(qt.Equals), 0)
+
+	te2, err := tracefsKprobe(symbol, false)
+	c.Assert(err, qt.IsNil)
+	defer te2.Close()
+	c.Assert(te2.progType, qt.Equals, ebpf.Kprobe)
+	c.Assert(te2.tracefsID, qt.Not(qt.Equals), 0)
+
+	c.Assert(te1.tracefsID, qt.Not(qt.CmpEquals()), te2.tracefsID)
+
+	// Write a kprobe event for a non-existing symbol.
+	err = createTraceFSKprobeEvent("syscalls", "bogus", false)
+	c.Assert(errors.Is(err, ErrNotSupported), qt.IsTrue)
+}
+
+func BenchmarkKprobeCreateTraceFS(b *testing.B) {
+	for n := 0; n < b.N; n++ {
+		// Include <tracefs>/kprobe_events operations in the benchmark loop
+		// because we create one per perf event.
+		pr, err := tracefsKprobe("printk", false)
+		if err != nil {
+			b.Error("error creating tracefs perf event:", err)
+		}
+
+		if err := pr.Close(); err != nil {
+			b.Error("error closing tracefs perf event:", err)
+		}
+	}
+}
+
+// Test k(ret)probe creation writing directly to <tracefs>/kprobe_events.
+// Only runs on 5.0 and over. Earlier versions ignored writes of duplicate
+// events, while 5.0 started returning -EEXIST when a kprobe event already
+// exists.
+func TestKprobeCreateTraceFS(t *testing.T) {
+	testutils.SkipOnOldKernel(t, "5.0", "<tracefs>/kprobe_events doesn't reject duplicate events")
+
+	c := qt.New(t)
+
+	symbol := "printk"
+	pg, _ := randomGroup("ebpftest")
+	rg, _ := randomGroup("ebpftest")
+
+	// Tee up cleanups in case any of the Asserts abort the function.
+	defer func() {
+		_ = closeTraceFSKprobeEvent(pg, symbol)
+		_ = closeTraceFSKprobeEvent(rg, symbol)
+	}()
+
+	// Create a kprobe.
+	err := createTraceFSKprobeEvent(pg, symbol, false)
+	c.Assert(err, qt.IsNil)
+
+	// Attempt to create an identical kprobe using tracefs,
+	// expect it to fail with os.ErrExist.
+	err = createTraceFSKprobeEvent(pg, symbol, false)
+	c.Assert(errors.Is(err, os.ErrExist), qt.IsTrue,
+		qt.Commentf("expected consecutive kprobe creation to contain os.ErrExist, got: %v", err))
+
+	// Expect a successful close of the kprobe.
+	c.Assert(closeTraceFSKprobeEvent(pg, symbol), qt.IsNil)
+
+	// Same test for a kretprobe.
+	err = createTraceFSKprobeEvent(rg, symbol, true)
+	c.Assert(err, qt.IsNil)
+
+	err = createTraceFSKprobeEvent(rg, symbol, true)
+	c.Assert(os.IsExist(err), qt.IsFalse,
+		qt.Commentf("expected consecutive kretprobe creation to contain os.ErrExist, got: %v", err))
+
+	// Expect a successful close of the kretprobe.
+	c.Assert(closeTraceFSKprobeEvent(rg, symbol), qt.IsNil)
+
+}
+
+func TestKprobeTraceFSGroup(t *testing.T) {
+	c := qt.New(t)
+
+	// Expect <prefix>_<16 random hex chars>.
+	g, err := randomGroup("ebpftest")
+	c.Assert(err, qt.IsNil)
+	c.Assert(g, qt.Matches, `ebpftest_[a-f0-9]{16}`)
+
+	// Expect error when the generator's output exceeds 63 characters.
+	p := make([]byte, 47) // 63 - 17 (length of the random suffix and underscore) + 1
+	for i := range p {
+		p[i] = byte('a')
+	}
+	_, err = randomGroup(string(p))
+	c.Assert(err, qt.Not(qt.IsNil))
+
+	// Reject non-alphanumeric characters.
+	_, err = randomGroup("/")
+	c.Assert(err, qt.Not(qt.IsNil))
+}
diff --git a/link/perf_event_test.go b/link/perf_event_test.go
new file mode 100644
index 0000000..3f7d252
--- /dev/null
+++ b/link/perf_event_test.go
@@ -0,0 +1,69 @@
+package link
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/cilium/ebpf/internal/testutils"
+	qt "github.com/frankban/quicktest"
+)
+
+func TestTraceEventTypePMU(t *testing.T) {
+	// Requires at least 4.17 (e12f03d7031a "perf/core: Implement the 'perf_kprobe' PMU")
+	testutils.SkipOnOldKernel(t, "4.17", "perf_kprobe PMU")
+
+	c := qt.New(t)
+
+	et, err := getPMUEventType("kprobe")
+	c.Assert(err, qt.IsNil)
+	c.Assert(et, qt.Not(qt.Equals), 0)
+}
+
+func TestTraceEventID(t *testing.T) {
+	c := qt.New(t)
+
+	eid, err := getTraceEventID("syscalls", "sys_enter_fork")
+	c.Assert(err, qt.IsNil)
+	c.Assert(eid, qt.Not(qt.Equals), 0)
+}
+
+func TestTraceReadID(t *testing.T) {
+	_, err := uint64FromFile("/base/path/", "../escaped")
+	if !errors.Is(err, errInvalidInput) {
+		t.Errorf("expected error %s, got: %s", errInvalidInput, err)
+	}
+
+	_, err = uint64FromFile("/base/path/not", "../not/escaped")
+	if !errors.Is(err, ErrNotSupported) {
+		t.Errorf("expected error %s, got: %s", ErrNotSupported, err)
+	}
+}
+
+func TestTraceEventRegex(t *testing.T) {
+	var tests = []struct {
+		name string
+		in   string
+		fail bool
+	}{
+		{"empty string", "", true},
+		{"leading number", "1test", true},
+		{"underscore first", "__x64_syscall", false},
+		{"contains number", "bpf_trace_run1", false},
+		{"underscore", "_", false},
+		{"contains dash", "-EINVAL", true},
+		{"contains number", "all0wed", false},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			exp := "pass"
+			if tt.fail {
+				exp = "fail"
+			}
+
+			if rgxTraceEvent.MatchString(tt.in) == tt.fail {
+				t.Errorf("expected string '%s' to %s regex match", tt.in, exp)
+			}
+		})
+	}
+}
diff --git a/link/tracepoint_test.go b/link/tracepoint_test.go
new file mode 100644
index 0000000..736f765
--- /dev/null
+++ b/link/tracepoint_test.go
@@ -0,0 +1,73 @@
+package link
+
+import (
+	"errors"
+	"testing"
+
+	"github.com/cilium/ebpf"
+	"github.com/cilium/ebpf/asm"
+	"github.com/cilium/ebpf/internal"
+	"github.com/cilium/ebpf/internal/testutils"
+
+	qt "github.com/frankban/quicktest"
+)
+
+func TestTracepoint(t *testing.T) {
+
+	// Requires at least 4.7 (98b5c2c65c29 "perf, bpf: allow bpf programs attach to tracepoints")
+	testutils.SkipOnOldKernel(t, "4.7", "tracepoint support")
+
+	prog, err := ebpf.NewProgram(&ebpf.ProgramSpec{
+		Type:    ebpf.TracePoint,
+		License: "MIT",
+		Instructions: asm.Instructions{
+			// set exit code to 0
+			asm.Mov.Imm(asm.R0, 0),
+			asm.Return(),
+		},
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer prog.Close()
+
+	// printk is guaranteed to be present.
+	// Kernels before 4.14 don't support attaching to syscall tracepoints.
+	tp, err := Tracepoint("printk", "console", prog)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if err := tp.Close(); err != nil {
+		t.Error("closing tracepoint:", err)
+	}
+}
+
+func TestTracepointErrors(t *testing.T) {
+	c := qt.New(t)
+
+	// Invalid Tracepoint incantations.
+	_, err := Tracepoint("", "", nil) // empty names
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+
+	_, err = Tracepoint("_", "_", nil) // empty prog
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+
+	_, err = Tracepoint(".", "+", &ebpf.Program{}) // illegal chars in group/name
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+
+	_, err = Tracepoint("foo", "bar", &ebpf.Program{}) // wrong prog type
+	c.Assert(errors.Is(err, errInvalidInput), qt.IsTrue)
+}
+
+func TestTraceGetEventID(t *testing.T) {
+	_, err := getTraceEventID("syscalls", "sys_enter_open")
+	if err != nil {
+		t.Fatal("Can't read trace event ID:", err)
+	}
+
+	_, err = getTraceEventID("totally", "bogus")
+	if !errors.Is(err, internal.ErrNotSupported) {
+		t.Fatal("Doesn't return ErrNotSupported")
+	}
+}
diff --git a/run-tests.sh b/run-tests.sh
index 647a61a..ed8679f 100755
--- a/run-tests.sh
+++ b/run-tests.sh
@@ -9,6 +9,7 @@ if [[ "${1:-}" = "--in-vm" ]]; then
   shift
 
   mount -t bpf bpf /sys/fs/bpf
+  mount -t tracefs tracefs /sys/kernel/debug/tracing
   export CGO_ENABLED=0
   export GOFLAGS=-mod=readonly
   export GOPATH=/run/go-path