neulab/tldr huggingface dataset:
$source_tree See the section on /etc/dkms/framework.conf for more information. This variable represents the location where DKMS keeps source on the local system. By default this is /usr/src , but this value should not be hard-coded into a dkms.conf in the event that the local user has changed it on their system.
$kernel_source_dir This variable holds the value of the location of your kernel source directory. Usually, this will be /usr/lib/modules/$kernelver/build , unless otherwise specified with the --kernelsourcedir option.
DKMS.CONF OVERRIDES You can override the module-provided dkms.conf files. Every time after a dkms.conf file is read, dkms will look for and read the following files in order:
/etc/dkms/<module>.conf /etc/dkms/<module>-<module-version>.conf /etc/dkms/<module>-<module-version>-<kernel>.conf /etc/dkms/<module>-<module-version>-<kernel>-<arch>.conf
You can use these files to override settings in the module-provided dkms.conf files.
/etc/dkms/framework.conf This configuration file controls how the overall DKMS framework handles. It is sourced in every time the dkms command is run. Mainly it can currently be used to set different default values for the variables.
$dkms_tree, $source_tree, $install_tree control where DKMS looks for its framework.
$symlink_modules controls whether binary modules are copied to /usr/lib/modules or if only symlinks are created there. Note that these variables can also be manipulated on the command line with --dkmstree, --sourcetree, --installtree and --symlink-modules options.
$sign_tool Script to be run at build for signing modules.  Two arguments will be passed to the script. The first argument is the target kernel version , the second is the module file path.  If the script exits with a non-zero value, the build will be aborted.
$autoinstall_all_kernels used by the common postinst for DKMS modules. It controls if the build should be done for all installed kernels or only for the current and latest installed kernel. It has no command line equivalent.
dkms_autoinstaller This boot-time service automatically installs any module which has AUTOINSTALL="yes" set in its dkms.conf file. The service works quite simply and if multiple versions of a module are in your system's DKMS tree, it will not do anything and instead explain that manual intervention is required.
Gary Lerhaupt
https://github.com/dell/dkms
MAILING-LIST dkms-devel@dell.com http://lists.us.dell.com/mailman/listinfo/dkms-devel
GIT-PR(1)                          Git Extras                          GIT-PR(1)
git-pr - Checks out a pull request locally
git-pr <number> [<remote>] git-pr <url> git-pr clean
Creates a local branch based on a GitHub pull request number or URL, and switch to that branch afterwards.
<remote>
The name of the remote to fetch from. Defaults to origin.
<url>
GitHub pull request URL in the format https://github.com/tj/git-extras/pull/453.
This checks out the pull request 226 from origin:
$ git pr 226
remote: Counting objects: 12, done. remote: Compressing objects: 100% (9/9), done. remote: Total 12 (delta 3), reused 9 (delta 3) Unpacking objects: 100% (12/12), done. From https://github.com/tj/git-extras * [new ref]         refs/pull/226/head -> pr/226
Switched to branch ´pr/226´
This pulls from a different remote:
$ git pr 226 upstream
You can also checkout a pull request based on a GitHub URL:
$ git pr https://github.com/tj/git-extras/pull/453
From https://github.com/tj/git-extras * [new ref]         refs/pull/453/head -> pr/453 Switched to branch ´pr/453´
To clean up old branches:
$ git pr clean
Deleted branch pr/226 (was b96a8c2). Deleted branch pr/220 (was d34dc0f).
Originally from https://gist.github.com/gnarf/5406589
IPERF3(1)                         User Manuals                         IPERF3(1)
iperf3 - perform network throughput tests
iperf3 -s [ options ] iperf3 -c server [ options ]
iperf3 is a tool for performing network throughput measurements.  It can test TCP, UDP, or SCTP throughput.  To perform an iperf3 test the user must establish both a server and a client.
The iperf3 executable contains both client and server functionality.  An iperf3 server can be started using either of the -s or --server command- line parameters, for example:
iperf3 -s
iperf3 --server
Note that many iperf3 parameters have both short (-s) and long (--server) forms.  In this section we will generally use the short form of command- line flags, unless only the long form of a flag is available.
By default, the iperf3 server listens on TCP port 5201 for connections from an iperf3 client.  A custom port can be specified by using the -p flag, for example:
iperf3 -s -p 5002
After the server is started, it will listen for connections from iperf3 clients (in other words, the iperf3 program run in client mode).  The client mode can be started using the -c command-line option, which also requires a host to which iperf3 should connect.  The host can by specified by hostname, IPv4 literal, or IPv6 literal:
iperf3 -c iperf3.example.com
iperf3 -c 192.0.2.1
iperf3 -c 2001:db8::1
If the iperf3 server is running on a non-default TCP port, that port number needs to be specified on the client as well:
iperf3 -c iperf3.example.com -p 5002
The initial TCP connection is used to exchange test parameters, control the start and end of the test, and to exchange test results.  This is sometimes referred to as the "control connection".  The actual test data is sent over a separate TCP connection, as a separate flow of UDP packets, or as an independent SCTP connection, depending on what protocol was specified by the client.
Normally, the test data is sent from the client to the server, and measures the upload speed of the client.  Measuring the download speed from the server can be done by specifying the -R flag on the client. This causes data to be sent from the server to the client.
iperf3 -c iperf3.example.com -p 5202 -R
Results are displayed on both the client and server.  There will be at least one line of output per measurement interval (by default a measurement interval lasts for one second, but this can be changed by the -i option).  Each line of output includes (at least) the time since the start of the test, amount of data transferred during the interval, and the average bitrate over that interval.  Note that the values for each measurement interval are taken from the point of view of the endpoint process emitting that output (in other words, the output on the client shows the measurement interval data for the client.
At the end of the test is a set of statistics that shows (at least as much as possible) a summary of the test as seen by both the sender and the receiver, with lines tagged accordingly.  Recall that by default the client is the sender and the server is the receiver, although as indicated above, use of the -R flag will reverse these roles.
The client can be made to retrieve the server-side output for a given test by specifying the --get-server-output flag.
Either the client or the server can produce its output in a JSON structure, useful for integration with other programs, by passing it the -J flag.  Because the contents of the JSON structure are only competely known after the test has finished, no JSON output will be emitted until the end of the test.
iperf3 has a (overly) large set of command-line options that can be used to set the parameters of a test.  They are given in the "GENERAL OPTIONS" section of the manual page below, as well as summarized in iperf3's help output, which can be viewed by running iperf3 with the -h flag.
-p, --port n set server port to listen on/connect to to n (default 5201)
-f, --format [kmgtKMGT]   format to report: Kbits/Mbits/Gbits/Tbits
-i, --interval n pause n seconds between periodic throughput reports; default is 1, use 0 to disable
-I, --pidfile file write a file with the process ID, most useful when running as a daemon.
-F, --file name Use a file as the source (on the sender) or sink (on the receiver) of data, rather than just generating random data or throwing it away.  This feature is used for finding whether or not the storage subsystem is the bottleneck for file transfers.  It does not turn iperf3 into a file transfer tool.  The length, attributes, and in some cases contents of the received file may not match those of the original file.
-A, --affinity n/n,m Set the CPU affinity, if possible (Linux, FreeBSD, and Windows only).  On both the client and server you can set the local affinity by using the n form of this argument (where n is a CPU number).  In addition, on the client side you can override the server's affinity for just that one test, using the n,m form of argument.  Note that when using this feature, a process will only be bound to a single CPU (as opposed to a set containing potentialy multiple CPUs).
-B, --bind host bind to the specific interface associated with address host. --bind-dev dev.ft R bind to the specified network interface.  This option uses SO_BINDTODEVICE, and may require root permissions. (Available on Linux and possibly other systems.)
-V, --verbose give more detailed output
-J, --json output in JSON format
--logfile file send output to a log file.
--forceflush force flushing output at every interval.  Used to avoid buffering when sending output to pipe.
--timestamps[=format] prepend a timestamp at the start of each output line.  By default, timestamps have the format emitted by ctime(1).  Optionally, = followed by a format specification can be passed to customize the timestamps, see strftime(3).  If this optional format is given, the = must immediately follow the --timestamps option with no whitespace intervening.
--rcv-timeout # set idle timeout for receiving data during active tests. The receiver will halt a test if no data is received from the sender for this number of ms (default to 12000 ms, or 2 minutes).
-d, --debug emit debugging output.  Primarily (perhaps exclusively) of use to developers.
-v, --version show version information and quit
-h, --help show a help synopsis
-D, --daemon run the server in background as a daemon
-1, --one-off handle one client connection, then exit.
--server-bitrate-limit n[KMGT] set a limit on the server side, which will cause a test to abort if the client specifies a test of more than n bits per second, or if the average data sent or received by the client (including all data streams) is greater than n bits per second.  The default limit is zero, which implies no limit.  The interval over which to average the data rate is 5 seconds by default, but can be specified by adding a '/' and a number to the bitrate specifier.
--rsa-private-key-path file path to the RSA private key (not password-protected) used to decrypt authentication credentials from the client (if built with OpenSSL support).
--authorized-users-path file path to the configuration file containing authorized users credentials to run iperf tests (if built with OpenSSL support). The file is a comma separated list of usernames and password hashes; more information on the structure of the file can be found in the EXAMPLES section.
--time-skew-thresholdsecond seconds time skew threshold (in seconds) between the server and client during the authentication process.
-c, --client host run in client mode, connecting to the specified server.  By default, a test consists of sending data from the client to the server, unless the -R flag is specified.
--sctp use SCTP rather than TCP (FreeBSD and Linux)
--connect-timeout n set timeout for establishing the initial control connection to the server, in milliseconds.  The default behavior is the operating system's timeout for TCP connection establishment.  Providing a shorter value may speed up detection of a down iperf3 server.
-b, --bitrate n[KMGT] set target bitrate to n bits/sec (default 1 Mbit/sec for UDP, unlimited for TCP/SCTP).  If there are multiple streams (-P flag), the throughput limit is applied separately to each stream.  You can also add a '/' and a number to the bitrate specifier.  This is called "burst mode".  It will send the given number of packets without pausing, even if that temporarily exceeds the specified throughput limit.  Setting the target bitrate to 0 will disable bitrate limits (particularly useful for UDP tests).  This throughput limit is implemented internally inside iperf3, and is available on all platforms.  Compare with the --fq-rate flag. This option replaces the --bandwidth flag, which is now deprecated but (at least for now) still accepted.
--pacing-timer n[KMGT] set pacing timer interval in microseconds (default 1000 microseconds, or 1 ms).  This controls iperf3's internal pacing timer for the -b/--bitrate option.  The timer fires at the interval set by this parameter.  Smaller values of the pacing timer parameter smooth out the traffic emitted by iperf3, but potentially at the cost of performance due to more frequent timer processing.
--fq-rate n[KMGT] Set a rate to be used with fair-queueing based socket-level pacing, in bits per second.  This pacing (if specified) will be in addition to any pacing due to iperf3's internal throughput pacing (-b/--bitrate flag), and both can be specified for the same test. Only available on platforms supporting the SO_MAX_PACING_RATE socket option (currently only Linux).  The default is no fair- queueing based pacing.
--no-fq-socket-pacing This option is deprecated and will be removed.  It is equivalent to specifying --fq-rate=0.
-t, --time n time in seconds to transmit for (default 10 secs)
-n, --bytes n[KMGT] number of bytes to transmit (instead of -t)
-k, --blockcount n[KMGT] number of blocks (packets) to transmit (instead of -t or -n)
-l, --length n[KMGT] length of buffer to read or write.  For TCP tests, the default value is 128KB.  In the case of UDP, iperf3 tries to dynamically determine a reasonable sending size based on the path MTU; if that cannot be determined it uses 1460 bytes as a sending size.  For SCTP tests, the default size is 64KB.
--cport port bind data streams to a specific client port (for TCP and UDP only, default is to use an ephemeral port)
-P, --parallel n number of parallel client streams to run. Note that iperf3 is single threaded, so if you are CPU bound, this will not yield higher throughput.
-R, --reverse reverse the direction of a test, so that the server sends data to the client
--bidir test in both directions (normal and reverse), with both the client and server sending and receiving data simultaneously
-w, --window n[KMGT] window size / socket buffer size (this gets sent to the server and used on that side too)
-M, --set-mss n set TCP/SCTP maximum segment size (MTU - 40 bytes)
-N, --no-delay set TCP/SCTP no delay, disabling Nagle's Algorithm
-4, --version4 only use IPv4
-6, --version6 only use IPv6
-S, --tos n set the IP type of service. The usual prefixes for octal and hex can be used, i.e. 52, 064 and 0x34 all specify the same value.
--dscp dscp set the IP DSCP bits.  Both numeric and symbolic values are accepted. Numeric values can be specified in decimal, octal and hex (see --tos above).
-L, --flowlabel n set the IPv6 flow label (currently only supported on Linux)
-X, --xbind name Bind SCTP associations to a specific subset of links using sctp_bindx(3).  The --B flag will be ignored if this flag is specified.  Normally SCTP will include the protocol addresses of all active links on the local host when setting up an association. Specifying at least one --X name will disable this behaviour. This flag must be specified for each link to be included in the association, and is supported for both iperf servers and clients (the latter are supported by passing the first --X argument to bind(2)).  Hostnames are accepted as arguments and are resolved using getaddrinfo(3).  If the --4 or --6 flags are specified, names which do not resolve to addresses within the specified protocol family will be ignored.
--nstreams n Set number of SCTP streams.
-Z, --zerocopy Use a "zero copy" method of sending data, such as sendfile(2), instead of the usual write(2).
-O, --omit n Omit the first n seconds of the test, to skip past the TCP slow- start period.
-T, --title str Prefix every output line with this string.
--extra-data str Specify an extra data string field to be included in JSON output.
-C, --congestion algo Set the congestion control algorithm (Linux and FreeBSD only).  An older --linux-congestion synonym for this flag is accepted but is deprecated.
--get-server-output Get the output from the server.  The output format is determined by the server (in particular, if the server was invoked with the --json flag, the output will be in JSON format, otherwise it will be in human-readable format).  If the client is run with --json, the server output is included in a JSON object; otherwise it is appended at the bottom of the human-readable output.
--udp-counters-64bit Use 64-bit counters in UDP test packets.  The use of this option can help prevent counter overflows during long or high-bitrate UDP tests.  Both client and server need to be running at least version 3.1 for this option to work.  It may become the default behavior at some point in the future.
--repeating-payload Use repeating pattern in payload, instead of random bytes.  The same payload is used in iperf2 (ASCII '0..9' repeating).  It might help to test and reveal problems in networking gear with hardware compression (including some WiFi access points), where iperf2 and iperf3 perform differently, just based on payload entropy.
--dont-fragment Set the IPv4 Don't Fragment (DF) bit on outgoing packets.  Only applicable to tests doing UDP over IPv4.
--username username username to use for authentication to the iperf server (if built with OpenSSL support).  The password will be prompted for interactively when the test is run.  Note, the password to use can also be specified via the IPERF3_PASSWORD environment variable. If this variable is present, the password prompt will be skipped.
--rsa-public-key-path file path to the RSA public key used to encrypt authentication credentials (if built with OpenSSL support)
Authentication - RSA Keypair The authentication feature of iperf3 requires an RSA public keypair.  The public key is used to encrypt the authentication token containing the user credentials, while the private key is used to decrypt the authentication token.  The private key must be in PEM format and additionally must not have a password set.  The public key must be in PEM format and use SubjectPrefixKeyInfo encoding.  An example of a set of UNIX/Linux commands using OpenSSL to generate a correctly-formed keypair follows:
> openssl genrsa -des3 -out private.pem 2048 > openssl rsa -in private.pem -outform PEM -pubout -out public.pem > openssl rsa -in private.pem -out private_not_protected.pem -outform PEM
After these commands, the public key will be contained in the file public.pem and the private key will be contained in the file private_not_protected.pem.
Authentication - Authorized users configuration file A simple plaintext file must be provided to the iperf3 server in order to specify the authorized user credentials.  The file is a simple list of comma-separated pairs of a username and a corresponding password hash. The password hash is a SHA256 hash of the string "{$user}$password".  The file can also contain commented lines (starting with the # character). An example of commands to generate the password hash on a UNIX/Linux system is given below:
> S_USER=mario S_PASSWD=rossi > echo -n "{$S_USER}$S_PASSWD" | sha256sum | awk '{ print $1 }'
An example of a password file (with an entry corresponding to the above username and password) is given below: > cat credentials.csv # file format: username,sha256 mario,bf7a49a846d44b454a5d11e7acfaf13d138bbe0b7483aa3e050879700572709b
A list of the contributors to iperf3 can be found within the documentation located at https://software.es.net/iperf/dev.html#authors.
libiperf(3), https://software.es.net/iperf
BPFTRACE(8)                                                          BPFTRACE(8)
bpftrace - a high-level tracing language
bpftrace [OPTIONS] FILENAME bpftrace [OPTIONS] -e 'program code'
bpftrace is a high-level tracing language and runtime for Linux based on BPF. It supports static and dynamic tracing for both the kernel and user-space.
When FILENAME is "-", read from stdin.
List all probes with "sleep" in their name
# bpftrace -l '*sleep*'
Trace processes calling sleep
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }'
Trace processes calling sleep while spawning sleep 5 as a child process
# bpftrace -e 'kprobe:do_nanosleep { printf("%d sleeping\n", pid); }' -c 'sleep 5'
x86_64, arm64 and s390x
Output format -B MODE, Set the buffer mode for stdout. Valid values are none No buffering. Each I/O is written as soon as possible line Data is written on the first newline or when the buffer is full. This is the default mode. full Data is written once the buffer is full.
-f FORMAT, Set the output format. Valid values are json text
-o FILENAME Write bpftrace tracing output to FILENAME instead of stdout. This doesn’t include child process (-c option) output. Errors are still written to stderr.
--no-warnings Suppress all warning messages created by bpftrace.
Tracing -e PROGRAM Execute PROGRAM instead of reading the program from a file
-I DIR Add the directory DIR to the search path for C headers. This option can be used multiple times.
--include FILENAME Add FILENAME as an include for the pre-processor. This is equal to adding '#include FILENAME' to the start bpftrace program. This option can be used multiple times.
-l [SEARCH] List all probes that match the SEARCH pattern. If the pattern is omitted all probes will be listed. This pattern supports wildcards in the same way that probes do. E.g. '-l kprobe:*file*' to list all 'kprobes' with 'file' in the name. For more details see the LISTING PROBES section.
--unsafe Some calls, like 'system', are marked as unsafe as they can have dangerous side effects ('system("rm -rf")') and are disabled by default. This flag allows their use.
-k Errors from bpf-helpers(7) are silently ignored by default which can lead to strange results. This flag enables the detection of errors (except for errors from 'probe_read_*'). When errors occurs bpftrace will log an error containing the source location and the error code:
stdin:48-57: WARNING: Failed to probe_read_user_str: Bad address (-14) u:lib.so:"fn(char const*)" { printf("arg0:%s\n", str(arg0));} ~~~~~~~~~
-kk Same as '-k' but also includes the errors from 'probe_read_*' helpers.
Process management -p PID Attach to the process with PID. If the process terminates, bpftrace will also terminate. When using USDT probes they will be attached to only this process.
-c COMMAND Run COMMAND as a child process. When the child terminates bpftrace stops as well, as if 'exit()' has been called. If bpftrace terminates before the child process does the child process will be terminated with a SIGTERM. If used, 'USDT' probes these will only be attached to the child process. To avoid a race condition when using 'USDTs' the child is stopped after 'execve' using 'ptrace(2)' and continued when all 'USDT' probes are attached. The child PID is available to programs as the 'cpid' builtin. The child process runs with the same privileges as bpftrace itself (usually root).
--usdt-file-activation activate usdt semaphores based on file path
Miscellaneous --info Print detailed information about features supported by the kernel and the bpftrace build.
-h, --help Print the help summary
-V, --version Print bpftrace version information
-v verbose messages
-d debug mode
-dd verbose debug mode
Some behavior can only be controlled through environment variables. This section lists all those variables.
BPFTRACE_STRLEN Default: 64
Number of bytes allocated on the BPF stack for the string returned by str().
Make this larger if you wish to read bigger strings with str().
Beware that the BPF stack is small (512 bytes).
Support for even larger strings is [being discussed](https://github.com/iovisor/bpftrace/issues/305).
BPFTRACE_NO_CPP_DEMANGLE Default: 0
C++ symbol demangling in user space stack traces is enabled by default.
This feature can be turned off by setting the value of this environment variable to 1.
BPFTRACE_MAP_KEYS_MAX Default: 4096
This is the maximum number of keys that can be stored in a map. Increasing the value will consume more memory and increase startup times. There are some cases where you will want to: for example, sampling stack traces, recording timestamps for each page, etc.
BPFTRACE_MAX_PROBES Default: 512
This is the maximum number of probes that bpftrace can attach to. Increasing the value will consume more memory, increase startup times and can incur high performance overhead or even freeze or crash the system.
BPFTRACE_CACHE_USER_SYMBOLS Default: 0 if ASLR is enabled on system and -c option is not given; otherwise 1
By default, bpftrace caches the results of symbols resolutions only when ASLR (Address Space Layout Randomization) is disabled. This is because the symbol addresses change with each execution with ASLR. However, disabling caching may incur some performance penalty. Set this env variable to 1 to force bpftrace to cache.
BPFTRACE_VMLINUX Default: None
This specifies the vmlinux path used for kernel symbol resolution when attaching kprobe to offset. If this value is not given, bpftrace searches vmlinux from pre defined locations. See src/attached_probe.cpp:find_vmlinux() for details.
BPFTRACE_BTF Default: None
The path to a BTF file. By default, bpftrace searches several locations to find a BTF file. See src/btf.cpp for the details.
BPFTRACE_PERF_RB_PAGES Default: 64
Number of pages to allocate per CPU for perf ring buffer. The value must be a power of 2.
If you’re getting a lot of dropped events bpftrace may not be processing events in the ring buffer fast enough. It may be useful to bump the value higher so more events can be queued up. The tradeoff is that bpftrace will use more memory.
Overview The bpftrace (bt) language is inspired by the D language used by dtrace and uses the same program structure. Each script consists of an preamble and one or more action blocks.
actionblock1 actionblock2
Preprocessor and type definitions take place in the preamble:
#include <linux/socket.h> #define RED "\033[31m"
struct S { int x; }
Each action block consists of three parts:
probe[,probe] /predicate/ { action }
Probes A probe specifies the event and event type to attach too.
Predicate The predicate is optional condition that must be met for the action to be executed.
Action Actions are the programs that run when an event fires (and the predicate is met). An action is a semicolon (;) separated list of statements and always enclosed by brackets {}
A basic script that traces the open(2) and openat(2) system calls can be written as follows:
BEGIN { printf("Tracing open syscalls... Hit Ctrl-C to end.\n"); }
tracepoint:syscalls:sys_enter_open, tracepoint:syscalls:sys_enter_openat { printf("%-6d %-16s %s\n", pid, comm, str(args->filename)); }
This script has two action blocks and a total of 3 probes. The first action block uses the special BEGIN probe, which fires once during bpftrace startup. This probe is used to print a header, indicating that the tracing has started.
The second action block uses two probes, one for open and one for openat, and defines an action that prints the file being open ed as well as the pid and comm of the process that execute the syscall. See the PROBES section for details on the available probe types.
Identifiers Identifiers must match the following regular expression: [_a-zA-Z][_a-zA-Z0-9]*
Comments Both single line and multi line comments are supported.
// A single line comment i:s:1 { // can also be used to comment inline /* a multi line comment
*/ print(/* inline comment block */ 1); }
Data Types The following fundamental integer types are provided by the language.
┌───────┬─────────────────────────┐ │       │                         │ │Type   │ Description             │ ├───────┼─────────────────────────┤ │       │                         │ │uint8  │ Unsigned 8 bit integer  │ ├───────┼─────────────────────────┤ │       │                         │ │int8   │ Signed 8 bit integer    │ ├───────┼─────────────────────────┤ │       │                         │ │uint16 │ Unsigned 16 bit integer │ ├───────┼─────────────────────────┤ │       │                         │ │int16  │ Signed 16 bit integer   │ ├───────┼─────────────────────────┤ │       │                         │ │uint32 │ Unsigned 32 bit integer │ ├───────┼─────────────────────────┤ │       │                         │ │int32  │ Signed 32 bit integer   │ ├───────┼─────────────────────────┤ │       │                         │ │uint64 │ Unsigned 64 bit integer │ ├───────┼─────────────────────────┤ │       │                         │ │int64  │ Signed 64 bit integer   │ └───────┴─────────────────────────┘
Floating-point Floating-point numbers are not supported by BPF and therefore not by bpftrace.
Constants Integers constants can be defined in the following formats:
•   decimal (base 10)
•   octal (base 8)
•   hexadecimal (base 16)
•   scientific (base 10)
Octal constants have to be prefixed with a 0, e.g. 0123. Hexadecimal constants start with either 0x or 0X, e.g. 0x10. Scientific are written in the <m>e<n> format which is a shorthand for m*10^n, e.g. $i = 2e3;. Note that scientific literals are integer only due to the lack of floating point support, 1e-3 is not valid.
To improve the readability of big literals a underscore _ can be used as field separator, e.g. 1_000_123_000.
Integer suffixes as found in the C language are parsed by bpftrace to ensure compatibility with C headers/definitions but they’re not used as size specifiers. 123UL, 123U and 123LL all result in the same integer type with a value of 123.
Character constants can be defined by enclosing the character in single quotes, e.g. $c = 'c';.
String constants can be defined by enclosing the character string in double quotes, e.g. $str = "Hello world";.
Characters and strings support the following escape sequences:
┌─────┬──────────────────────┐ │     │                      │ │\n   │ Newline              │ ├─────┼──────────────────────┤ │     │                      │ │\t   │ Tab                  │ ├─────┼──────────────────────┤ │     │                      │ │\0nn │ Octal value nn       │ ├─────┼──────────────────────┤ │     │                      │ │\xnn │ Hexadecimal value nn │ └─────┴──────────────────────┘
Type conversion Integer and pointer types can be converted using explicit type conversion with an expression like:
$y = (uint32) $z; $py = (int16 *) $pz;
Integer casts to a higher rank are sign extended. Conversion to a lower rank is done by zeroing leading bits.
Operators and Expressions Arithmetic Operators The following operators are available for integer arithmetic:
┌──┬────────────────────────┐ │  │                        │ │+ │ integer addition       │ ├──┼────────────────────────┤ │  │                        │ │- │ integer subtraction    │ ├──┼────────────────────────┤ │  │                        │ │* │ integer multiplication │ ├──┼────────────────────────┤ │  │                        │ │/ │ integer division       │ ├──┼────────────────────────┤ │  │                        │ │% │ integer modulo         │ └──┴────────────────────────┘
Logical Operators ┌───┬─────────────┐ │   │             │ │&& │ Logical AND │ ├───┼─────────────┤ │   │             │ │|| │ Logical OR  │ ├───┼─────────────┤ │   │             │ │!  │ Logical NOT │ └───┴─────────────┘
Bitwise Operators ┌───┬─────────────────────────────┐ │   │                             │ │&  │ AND                         │ ├───┼─────────────────────────────┤ │   │                             │ │|  │ OR                          │ ├───┼─────────────────────────────┤ │   │                             │ │^  │ XOR                         │ ├───┼─────────────────────────────┤ │   │                             │ │<< │ Left shift the left-hand    │ │   │ operand by the number of    │ │   │ bits specified by the       │ │   │ right-hand expression value │ ├───┼─────────────────────────────┤ │   │                             │ │>> │ Right shift the left-hand   │ │   │ operand by the number of    │ │   │ bits specified by the       │ │   │ right-hand expression value │ └───┴─────────────────────────────┘
Relational Operators The following relational operators are defined for integers and pointers.
┌───┬────────────────────────────┐ │   │                            │ │<  │ left-hand expression is    │ │   │ less than right-hand       │ ├───┼────────────────────────────┤ │   │                            │ │<= │ left-hand expression is    │ │   │ less than or equal to      │ │   │ right-hand                 │ ├───┼────────────────────────────┤ │   │                            │ │>  │ left-hand expression is    │ │   │ bigger than right-hand     │ ├───┼────────────────────────────┤ │   │                            │ │>= │ left-hand expression is    │ │   │ bigger or equal to than    │ │   │ right-hand                 │ ├───┼────────────────────────────┤ │   │                            │ │== │ left-hand expression equal │ │   │ to right-hand              │ ├───┼────────────────────────────┤ │   │                            │ │!= │ left-hand expression not   │ │   │ equal to right-hand        │ └───┴────────────────────────────┘
The following relation operators are available for comparing strings.
┌───┬────────────────────────────┐ │   │                            │ │== │ left-hand string equal to  │ │   │ right-hand                 │ ├───┼────────────────────────────┤ │   │                            │ │!= │ left-hand string not equal │ │   │ to right-hand              │ └───┴────────────────────────────┘
Assignment Operators The following assignment operators can be used on both map and scratch variables:
┌────┬─────────────────────────────┐ │    │                             │ │=   │ Assignment, assign the      │ │    │ right-hand expression to    │ │    │ the left-hand variable      │ ├────┼─────────────────────────────┤ │    │                             │ │<<= │ Update the variable with    │ │    │ its value left shifted by   │ │    │ the number of bits          │ │    │ specified by the right-hand │ │    │ expression value            │ ├────┼─────────────────────────────┤ │    │                             │ │>>= │ Update the variable with    │ │    │ its value right shifted by  │ │    │ the number of bits          │ │    │ specified by the right-hand │ │    │ expression value            │ ├────┼─────────────────────────────┤ │    │                             │ │+=  │ Increment the variable by   │ │    │ the right-hand expression   │ │    │ value                       │ ├────┼─────────────────────────────┤ │    │                             │ │-=  │ Decrement the variable by   │ │    │ the right-hand expression   │ │    │ value                       │ ├────┼─────────────────────────────┤ │    │                             │ │*=  │ Multiple the variable by    │ │    │ the right-hand expression   │ │    │ value                       │ ├────┼─────────────────────────────┤ │    │                             │ │/=  │ Divide the variable by the  │ │    │ right-hand expression value │ ├────┼─────────────────────────────┤ │    │                             │ │%=  │ Modulo the variable by the  │ │    │ right-hand expression value │ ├────┼─────────────────────────────┤ │    │                             │ │&=  │ Bitwise AND the variable by │ │    │ the right-hand expression   │ │    │ value                       │ ├────┼─────────────────────────────┤ │    │                             │ │|=  │ Bitwise OR the variable by  │ │    │ the right-hand expression   │ │    │ value                       │ ├────┼─────────────────────────────┤ │    │                             │ │^=  │ Bitwise XOR the variable by │ │    │ the right-hand expression   │ │    │ value                       │ └────┴─────────────────────────────┘
All these operators are syntactic sugar for combining assignment with the specified operator. @ -= 5 is equal to @ = @ - 5.
Increment and Decrement Operators The increment (++) and decrement (--) operators can be used on integer and pointer variables to increment their value by one. They can only be used on variables and can either be applied as prefix or suffix. The difference is that the expression x++ returns the original value of x, before it got incremented while ++x returns the value of x post increment. E.g.
$x = 10; $y = $x--; // y = 10; x = 9 $a = 10; $b = --$a; // a = 9; b = 9
Note that maps will be implicitly declared and initialized to 0 if not already declared or defined. Scratch variables must be initialized before using these operators.
Variables and Maps bpftrace knows two types of variables, scratch and map.
'scratch' variables are kept on the BPF stack and only exists during the execution of the action block and cannot be accessed outside of the program. Scratch variable names always start with a $, e.g. $myvar.
'map' variables use BPF 'maps'. These exist for the lifetime of bpftrace itself and can be accessed from all action blocks and user-space. Map names always start with a @, e.g. @mymap.
All valid identifiers can be used as name.
The data type of a variable is automatically determined during first assignment and cannot be changed afterwards.
Associative Arrays Associative arrays are a collection of elements indexed by a key, similar to the hash tables found in languages like C++ (std::map) and Python (dict). They’re a variant of 'map' variables.
@name[key] = expression @name[key1,key2] = expression
Just like with any variable the type is determined on first use and cannot be modified afterwards. This applies to both the key(s) and the value type.
The following snippet creates a map with key signature [int64, string[16]] and a value type of int64:
@[pid, comm]++
Variable scoping Pointers Pointers in bpftrace are similar to those found in C.
Tuples bpftrace has support for immutable N-tuples (n > 1). A tuple is a sequence type (like an array) where, unlike an array, every element can have a different type.
Tuples are a comma separated list of expressions, enclosed in brackets, (1,2) Individual fields can be accessed with the . operator. Tuples are zero indexed like arrays are.
i:s:1 { $a = (1,2); $b = (3,4, $a); print($a); print($b); print($b.0); }
Prints:
(1, 2) (3, 4, (1, 2)) 3
Arrays bpftrace supports accessing one-dimensional arrays like those found in C.
Constructing arrays from scratch, like int a[] = {1,2,3} in C, is not supported. They can only be read into a variable from a pointer.
The [] operator is used to access elements.
struct MyStruct { int y[4]; }
kprobe:dummy { $s = (struct MyStruct *) arg0; print($s->y[0]); }
Structs C like structs are supported by bpftrace. Fields are accessed with the . operator. Fields of a pointer to a struct can be accessed with the -> operator.
Custom struct can be defined in the preamble
Constructing structs from scratch, like struct X var = {.f1 = 1} in C, is not supported. They can only be read into a variable from a pointer.
struct MyStruct { int a; }
kprobe:dummy { $ptr = (struct MyStruct *) arg0; $st = *$ptr; print($st.a); print($ptr->a); }
Conditionals Conditional expressions are supported in the form of if/else statements and the ternary operator.
The ternary operator consists of three operands: a condition followed by a ?, the expression to execute when the condition is true followed by a : and the expression to execute if the condition is false.
condition ? ifTrue : ifFalse
Both the ifTrue and ifFalse expressions must be of the same type, mixing types is not allowed.
The ternary operator can be used as part of an assignment.
$a == 1 ? print("true") : print("false"); $b = $a > 0 ? $a : -1;
If/else statements, like the one in C, are supported.
if (condition) { ifblock } else if (condition) { if2block } else { elseblock }
Loops Since kernel 5.3 BPF supports loops as long as the verifier can prove they’re bounded and fit within the instruction limit.
In bpftrace loops are available through the while statement.
while (condition) { block; }
Within a while-loop the following control flow statements can be used:
┌─────────┬─────────────────────────────┐ │         │                             │ │continue │ skip processing of the rest │ │         │ of the block and jump back  │ │         │ to the evaluation of the    │ │         │ conditional                 │ ├─────────┼─────────────────────────────┤ │         │                             │ │break    │ Terminate the loop          │ └─────────┴─────────────────────────────┘
i:s:1 { $i = 0; while ($i <= 100) { printf("%d ", $i); if ($i > 5) { break; } $i++ } printf("\n"); }
Loop unrolling is also supported with the unroll statement.
unroll(n) { block; }
The compiler will evaluate the block n times and generate the BPF code for the block n times. As this happens at compile time n must be a constant greater than 0 (n > 0).
The following two probes compile into the same code:
i:s:1 { unroll(3) { print("Unrolled") } }
i:s:1 { print("Unrolled") print("Unrolled") print("Unrolled") }
While BPF in the kernel can do a lot there are still things that can only be done from user space, like the outputting (printing) of data. The way bpftrace handles this is by sending events from the BPF program which user-space will pick up some time in the future (usually in milliseconds). Operations that happen in the kernel are 'synchronous' ('sync') and those that are handled in user space are 'asynchronous' ('async')
The async behaviour can lead to some unexpected behavior as updates can happen before user space had time to process the event. One example is updating a map value in a tight loop:
BEGIN { @=0; unroll(10) { print(@); @++; } exit() }
Maps are printed by reference not by value and as the value gets updated right after the print user-space will likely only see the final value once it processes the event:
@: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10 @: 10
ADDRESS-SPACES Kernel and user pointers live in different address spaces which, depending on the CPU architecture, might overlap. Trying to read a pointer that is in the wrong address space results in a runtime error. This error is hidden by default but can be enabled with the -kk flag:
stdin:1:9-12: WARNING: Failed to probe_read_user: Bad address (-14) BEGIN { @=*uptr(kaddr("do_poweroff")) } ~~~
bpftrace tries to automatically set the correct address space for a pointer based on the probe type, but might fail in cases where it is unclear. The address space can be changed with the kptr() and uptr() functions.
Builtins are special variables built into the language. Unlike the scratch and map variable they don’t need a $ or @ as prefix (except for the positional parameters).
Map functions are built-in functions who’s return value can only be assigned to maps. The data type associated with these functions are only for internal use and are not compatible with the (integer) operators.
Functions that are marked async are asynchronous which can lead to unexpected behavior, see the SYNC AND ASYNC section for more information.
avg variants
•   avg(int64 n)
Calculate the running average of n between consecutive calls.
i:s:1 { @x++; @y = avg(@x); print(@x); print(@y); }
Internally this keeps two values in the map: value count and running total. The average is computed in user-space when printing by dividing the total by the count.
clear variants
•   clear(map m)
Clear all keys/values from map m.
i:ms:100 { @[rand % 10] = count(); }
i:s:10 { print(@); clear(@); }
count variants
•   count()
Count how often this function is called.
Using @=count() is conceptually similar to @++. The difference is that the count() function uses a map type optimized for this (PER_CPU), increasing performance. Due to this the map cannot be accessed as a regular integer.
i:ms:100 { @ = count(); }
delete variants
•   delete(mapkey k)
Delete a single key from a map. For a single value map this deletes the only element. For an associative-array the key to delete has to be specified.
k:dummy { @scalar = 1; @associative[1,2] = 1; delete(@scalar); delete(@associative[1,2]);
delete(@associative); // error }
hist variants
•   hist(int64 n)
Create a log2 histogram of n.
kretprobe:vfs_read { @bytes = hist(retval); }
Results in:
@: [1M, 2M)               3 |                                                    | [2M, 4M)               2 |                                                    | [4M, 8M)               2 |                                                    | [8M, 16M)              6 |                                                    | [16M, 32M)            16 |                                                    | [32M, 64M)            27 |                                                    | [64M, 128M)           48 |@                                                   | [128M, 256M)          98 |@@@                                                 | [256M, 512M)         191 |@@@@@@                                              | [512M, 1G)           394 |@@@@@@@@@@@@@                                       | [1G, 2G)             820 |@@@@@@@@@@@@@@@@@@@@@@@@@@@                         |
lhist variants
•   lhist(int64 n, int64 min, int64 max, int64 step)
Create a linear histogram of n. lhist creates M ((max - min) / step) buckets in the range [min,max) where each bucket is step in size. Values in the range (-inf, min) and (max, inf) get their get their own bucket too, bringing the total amount of buckets created to M+2.
i:ms:1 { @ = lhist(rand %10, 0, 10, 1); }
i:s:5 { exit(); }
@: [0, 1)               306 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@         | [1, 2)               284 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@            | [2, 3)               294 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@          | [3, 4)               318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@       | [4, 5)               311 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@        | [5, 6)               362 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [6, 7)               336 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    | [7, 8)               326 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@      | [8, 9)               328 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@     | [9, 10)              318 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@       |
max variants
•   max(int64 n)
Update the map with n if n is bigger than the current value held.
min variants
•   min(int64 n)
Update the map with n if n is smaller than the current value held.
stats variants
•   stats(int64 n)
stats combines the count, avg and sum calls into one.
kprobe:vfs_read { @bytes[comm] = stats(arg2); }
@bytes[bash]: count 7, average 1, total 7 @bytes[sleep]: count 5, average 832, total 4160 @bytes[ls]: count 7, average 886, total 6208 @
sum variants
•   sum(int64 n)
Calculate the sum of all n passed.
zero variants
•   zero(map m)
Set all values for all keys to zero.
Functions that are marked async are asynchronous which can lead to unexpected behaviour, see the [sync and async] section for more information.
compile time functions are evaluated at compile time, a static value will be compiled into the program.
unsafe functions can have dangerous side effects and should be used with care, the --unsafe flag is required for use.
buf variants
•   buf_t buf(void * data, [int64 length])
buf reads length amount of bytes from address data. The maximum value of length is limited to the BPFTRACE_STRLEN variable. For arrays the length is optional, it is automatically inferred from the signature.
buf is address space aware and will call the correct helper based on the address space associated with data.
The buf_t object returned by buf can safely be printed as a hex encoded string with the %r format specifier.
Bytes with values >=32 and <=126 are printed using their ASCII character, other bytes are printed in hex form (e.g. \x00).
i:s:1 { printf("%r\n", buf(kaddr("avenrun"), 8)); }
\x00\x03\x00\x00\x00\x00\x00\x00 \xc2\x02\x00\x00\x00\x00\x00\x00
cat variants
•   void cat(string namefmt, [...args])
Dump the contents of the named file to stdout. cat supports the same format string and arguments that printf does. If the file cannot be opened or read an error is printed to stderr.
t:syscalls:sys_enter_execve { cat("/proc/%d/maps", pid); }
55f683ebd000-55f683ec1000 r--p 00000000 08:01 1843399                    /usr/bin/ls 55f683ec1000-55f683ed6000 r-xp 00004000 08:01 1843399                    /usr/bin/ls 55f683ed6000-55f683edf000 r--p 00019000 08:01 1843399                    /usr/bin/ls 55f683edf000-55f683ee2000 rw-p 00021000 08:01 1843399                    /usr/bin/ls 55f683ee2000-55f683ee3000 rw-p 00000000 00:00 0
cgroupid variants