neulab/tldr huggingface dataset:
-v     Causes guacd to simply print its version information and exit.
SSL/TLS OPTIONS If libssl was present at the time guacd was compiled, it will contain SSL/TLS support, and connections between the web application and guacd can be encrypted if a certificate file is given.
When using a chain of certificates, you must append the additional certificates to your server certificate. This can be done easily with the standard cat command. Beware that the certificate for guacd must be the first certificate in the file.
-C CERTIFICATE FILE Enables SSL/TLS using the given cerficiate file. Future connections to this instance of guacd will require SSL/TLS enabled in the client (the web application). If this option is not given, communication with guacd must be unencrypted.
-K KEY FILE Enables SSL/TLS using the given private key file. Future connections to this instance of guacd will require SSL/TLS enabled in the client (the web application). If this option is not given, communication with guacd must be unencrypted.
guacd.conf(5)
NOTMUCH(1)                           notmuch                          NOTMUCH(1)
notmuch [option ...] command [arg ...]
Notmuch is a command-line based program for indexing, searching, reading, and tagging large collections of email messages.
This page describes how to get started using notmuch from the command line, and gives a brief overview of the commands available. For more information on e.g. notmuch show consult the notmuch-show(1) man page, also accessible via notmuch help show
The quickest way to get started with Notmuch is to simply invoke the notmuch command with no arguments, which will interactively guide you through the process of indexing your mail.
While the command-line program notmuch provides powerful functionality, it does not provide the most convenient interface for that functionality. More sophisticated interfaces are expected to be built on top of either the command-line interface, or more likely, on top of the notmuch library interface. See https://notmuchmail.org for more about alternate interfaces to notmuch. The emacs-based interface to notmuch (available under emacs/ in the Notmuch source distribution) is probably the most widely used at this time.
Supported global options for notmuch include
--help [command-name] Print a synopsis of available commands and exit. With an optional command name, show the man page for that subcommand.
--version Print the installed version of notmuch, and exit.
--config=FILE Specify the configuration file to use. This overrides any configuration file specified by NOTMUCH_CONFIG. The empty string is a permitted and sometimes useful value of FILE, which tells notmuch to use only configuration metadata from the database.
--uuid=HEX Enforce that the database UUID (a unique identifier which persists until e.g. the database is compacted) is HEX; exit with an error if it is not. This is useful to detect rollover in modification counts on messages. You can find this UUID using e.g. notmuch count --lastmod
All global options except --config can also be specified after the command. For example, notmuch subcommand --uuid=HEX is equivalent to notmuch --uuid=HEX subcommand.
SETUP The notmuch setup command is used to configure Notmuch for first use, (or to reconfigure it later).
The setup command will prompt for your full name, your primary email address, any alternate email addresses you use, and the directory containing your email archives. Your answers will be written to a configuration file in NOTMUCH_CONFIG (if set) or ${HOME}/.notmuch-config . This configuration file will be created with descriptive comments, making it easy to edit by hand later to change the configuration. Or you can run notmuch setup again to change the configuration.
The mail directory you specify can contain any number of sub-directories and should primarily contain only files with individual email messages (eg. maildir or mh archives are perfect). If there are other, non-email files (such as indexes maintained by other email programs) then notmuch will do its best to detect those and ignore them.
Mail storage that uses mbox format, (where one mbox file contains many messages), will not work with notmuch. If that's how your mail is currently stored, it is recommended you first convert it to maildir format with a utility such as mb2md(1) before running notmuch setup.
Invoking notmuch with no command argument will run setup if the setup command has not previously been completed.
OTHER COMMANDS Several of the notmuch commands accept search terms with a common syntax. See notmuch-search-terms(7) for more details on the supported syntax.
The notmuch-search(1), notmuch-show(1), notmuch-address(1) and notmuch-count(1) commands are used to query the email database.
The notmuch-reply(1) command is useful for preparing a template for an email reply.
The notmuch-tag(1) command is the only command available for manipulating database contents.
The notmuch-dump(1) and notmuch-restore(1) commands can be used to create a textual dump of email tags for backup purposes, and to restore from that dump.
The notmuch-config(1) command can be used to get or set settings in the notmuch configuration file.
CUSTOM COMMANDS If the given command is not known to notmuch, notmuch tries to execute the external notmuch-<subcommand> in PATH instead. This allows users to have their own notmuch related tools to be run via the notmuch command. By design, this does not allow notmuch's own commands to be overridden using external commands.
OPTION SYNTAX All options accepting an argument can be used with '=' or ':' as a separator. Except for boolean options (which would be ambiguous), a space can also be used as a separator. The following are all equivalent:
notmuch --config=alt-config config get user.name notmuch --config:alt-config config get user.name notmuch --config alt-config config get user.name
The following environment variables can be used to control the behavior of notmuch.
NOTMUCH_CONFIG Specifies the location of the notmuch configuration file. See notmuch-config(1) for details.
NOTMUCH_DATABASE Specifies the location of the notmuch database. See notmuch-config(1) for details.
NOTMUCH_PROFILE Selects among notmuch configurations. See notmuch-config(1) for details.
NOTMUCH_TALLOC_REPORT Location to write a talloc memory usage report. See talloc_enable_leak_report_full in talloc(3) for more information.
NOTMUCH_DEBUG_QUERY If set to a non-empty value, the notmuch library will print (to stderr) Xapian queries it constructs.
notmuch-address(1), notmuch-compact(1), notmuch-config(1), notmuch-count(1), notmuch-dump(1), notmuch-hooks(5), notmuch-insert(1), notmuch-new(1), notmuch-properties(7), notmuch-reindex(1), notmuch-reply(1), notmuch-restore(1), notmuch-search(1), notmuch-search-terms(7), notmuch-show(1), notmuch-tag(1)
The notmuch website: https://notmuchmail.org
Feel free to send questions, comments, or kudos to the notmuch mailing list <notmuch@notmuchmail.org> . Subscription is not required before posting, but is available from the notmuchmail.org website.
Real-time interaction with the Notmuch community is available via IRC (server: irc.libera.chat, channel: #notmuch).
Carl Worth and many others
2009-2021, Carl Worth and many others
PARALLEL(1)                         parallel                         PARALLEL(1)
parallel - build and execute shell command lines from standard input in parallel
parallel [options] [command [arguments]] < list_of_arguments
parallel [options] [command [arguments]] ( ::: arguments | :::+ arguments | :::: argfile(s) | ::::+ argfile(s) ) ...
parallel --semaphore [options] command
#!/usr/bin/parallel --shebang [options] [command [arguments]]
#!/usr/bin/parallel --shebang-wrap [options] [command [arguments]]
Read the Reader's guide below if you are new to GNU parallel.
GNU parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input into blocks and pipe a block into each command in parallel.
If you use xargs and tee today you will find GNU parallel very easy to use as GNU parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel.
GNU parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU parallel as input for other programs.
For each line of input GNU parallel will execute command with the line as arguments. If no command is given, the line of input is executed. Several lines will be run in parallel. GNU parallel can often be used as a substitute for xargs or cat | bash.
Reader's guide GNU parallel includes the 4 types of documentation: Tutorial, how-to, reference and explanation.
If you prefer reading a book buy GNU Parallel 2018 at https://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html or download it at: https://doi.org/10.5281/zenodo.1146014 Read at least chapter 1+2. It should take you less than 20 minutes.
Otherwise start by watching the intro videos for a quick introduction: https://youtube.com/playlist?list=PL284C9FF2488BC6D1
If you want to dive deeper: spend a couple of hours walking through the tutorial (man parallel_tutorial). Your command line will love you for it.
You can find a lot of EXAMPLEs of use after the list of OPTIONS in man parallel (Use LESS=+/EXAMPLE: man parallel). That will give you an idea of what GNU parallel is capable of, and you may find a solution you can simply adapt to your situation.
If you need a one page printable cheat sheet you can find it on: https://www.gnu.org/software/parallel/parallel_cheat.pdf
The man page is the reference for all options.
Design discussion
If you want to know the design decisions behind GNU parallel, try: man parallel_design. This is also a good intro if you intend to change GNU parallel.
command Command to execute.  If command or the following arguments contain replacement strings (such as {}) every instance will be substituted with the input.
If command is given, GNU parallel solve the same tasks as xargs. If command is not given GNU parallel will behave similar to cat | sh.
The command must be an executable, a script, a composed command, an alias, or a function.
Bash functions: export -f the function first or use env_parallel.
Bash, Csh, or Tcsh aliases: Use env_parallel.
Zsh, Fish, Ksh, and Pdksh functions and aliases: Use env_parallel.
{}  Input line. This replacement string will be replaced by a full line read from the input source. The input source is normally stdin (standard input), but can also be given with -a, :::, or ::::.
The replacement string {} can be changed with -I.
If the command line contains no replacement strings then {} will be appended to the command line.
Replacement strings are normally quoted, so special characters are not parsed by the shell. The exception is if the command starts with a replacement string; then the string is not quoted.
{.} Input line without extension. This replacement string will be replaced by the input with the extension removed. If the input line contains . after the last /, the last . until the end of the string will be removed and {.} will be replaced with the remaining. E.g. foo.jpg becomes foo, subdir/foo.jpg becomes subdir/foo, sub.dir/foo.jpg becomes sub.dir/foo, sub.dir/bar remains sub.dir/bar. If the input line does not contain . it will remain unchanged.
The replacement string {.} can be changed with --er.
To understand replacement strings see {}.
{/} Basename of input line. This replacement string will be replaced by the input with the directory part removed.
The replacement string {/} can be changed with --basenamereplace.
{//} Dirname of input line. This replacement string will be replaced by the dir of the input line. See dirname(1).
The replacement string {//} can be changed with --dirnamereplace.
{/.} Basename of input line without extension. This replacement string will be replaced by the input with the directory and extension part removed. It is a combination of {/} and {.}.
The replacement string {/.} can be changed with --basenameextensionreplace.
{#} Sequence number of the job to run. This replacement string will be replaced by the sequence number of the job being run. It contains the same number as $PARALLEL_SEQ.
The replacement string {#} can be changed with --seqreplace.
{%} Job slot number. This replacement string will be replaced by the job's slot number between 1 and number of jobs to run in parallel. There will never be 2 jobs running at the same time with the same job slot number.
The replacement string {%} can be changed with --slotreplace.
If the job needs to be retried (e.g using --retries or --retry-failed) the job slot is not automatically updated. You should then instead use $PARALLEL_JOBSLOT:
$ do_test() { id="$3 {%}=$1 PARALLEL_JOBSLOT=$2" echo run "$id"; sleep 1 # fail if {%} is odd return `echo $1%2 | bc` } $ export -f do_test $ parallel -j3 --jl mylog do_test {%} \$PARALLEL_JOBSLOT {} ::: A B C D run A {%}=1 PARALLEL_JOBSLOT=1 run B {%}=2 PARALLEL_JOBSLOT=2 run C {%}=3 PARALLEL_JOBSLOT=3 run D {%}=1 PARALLEL_JOBSLOT=1 $ parallel --retry-failed -j3 --jl mylog do_test {%} \$PARALLEL_JOBSLOT {} ::: A B C D run A {%}=1 PARALLEL_JOBSLOT=1 run C {%}=3 PARALLEL_JOBSLOT=2 run D {%}=1 PARALLEL_JOBSLOT=3
Notice how {%} and $PARALLEL_JOBSLOT differ in the retry run of C and D.
{n} Argument from input source n or the n'th argument. This positional replacement string will be replaced by the input from input source n (when used with -a or ::::) or with the n'th argument (when used with -N). If n is negative it refers to the n'th last argument.
{n.} Argument from input source n or the n'th argument without extension. It is a combination of {n} and {.}.
This positional replacement string will be replaced by the input from input source n (when used with -a or ::::) or with the n'th argument (when used with -N). The input will have the extension removed.
To understand positional replacement strings see {n}.
{n/} Basename of argument from input source n or the n'th argument.  It is a combination of {n} and {/}.
This positional replacement string will be replaced by the input from input source n (when used with -a or ::::) or with the n'th argument (when used with -N). The input will have the directory (if any) removed.
{n//} Dirname of argument from input source n or the n'th argument.  It is a combination of {n} and {//}.
This positional replacement string will be replaced by the dir of the input from input source n (when used with -a or ::::) or with the n'th argument (when used with -N). See dirname(1).
{n/.} Basename of argument from input source n or the n'th argument without extension.  It is a combination of {n}, {/}, and {.}.
This positional replacement string will be replaced by the input from input source n (when used with -a or ::::) or with the n'th argument (when used with -N). The input will have the directory (if any) and extension removed.
{=perl expression=} Replace with calculated perl expression. $_ will contain the same as {}. After evaluating perl expression $_ will be used as the value. It is recommended to only change $_ but you have full access to all of GNU parallel's internal functions and data structures.
The expression must give the same result if evaluated twice - otherwise the behaviour is undefined. E.g. this will not work as expected:
parallel echo '{= $_= ++$wrong_counter =}' ::: a b c
A few convenience functions and data structures have been made:
Q(string)     shell quote a string
pQ(string)    perl quote a string
uq() (or uq)  do not quote current replacement string
hash(val)     compute B::hash(val)
total_jobs()  number of jobs in total
slot()        slot number of job
seq()         sequence number of job
@arg          the arguments
skip()        skip this job (see also --filter)
yyyy_mm_dd_hh_mm_ss() yyyy_mm_dd_hh_mm() yyyy_mm_dd() yyyymmddhhmmss() yyyymmddhhmm() yyyymmdd()    time functions
seq 10 | parallel echo {} + 1 is {= '$_++' =} parallel csh -c {= '$_="mkdir ".Q($_)' =} ::: '12" dir' seq 50 | parallel echo job {#} of {= '$_=total_jobs()' =}
See also: --rpl --parens
{=n perl expression=} Positional equivalent to {=perl expression=}. To understand positional replacement strings see {n}.
See also: {=perl expression=} {n}.
::: arguments Use arguments from the command line as input source instead of stdin (standard input). Unlike other options for GNU parallel ::: is placed after the command and before the arguments.
The following are equivalent:
(echo file1; echo file2) | parallel gzip parallel gzip ::: file1 file2 parallel gzip {} ::: file1 file2 parallel --arg-sep ,, gzip {} ,, file1 file2 parallel --arg-sep ,, gzip ,, file1 file2 parallel ::: "gzip file1" "gzip file2"
To avoid treating ::: as special use --arg-sep to set the argument separator to something else. See also --arg-sep.
If multiple ::: are given, each group will be treated as an input source, and all combinations of input sources will be generated. E.g. ::: 1 2 ::: a b c will result in the combinations (1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing nested for-loops.
::: and :::: can be mixed. So these are equivalent:
parallel echo {1} {2} {3} ::: 6 7 ::: 4 5 ::: 1 2 3 parallel echo {1} {2} {3} :::: <(seq 6 7) <(seq 4 5) \ :::: <(seq 1 3) parallel -a <(seq 6 7) echo {1} {2} {3} :::: <(seq 4 5) \ :::: <(seq 1 3) parallel -a <(seq 6 7) -a <(seq 4 5) echo {1} {2} {3} \ ::: 1 2 3 seq 6 7 | parallel -a - -a <(seq 4 5) echo {1} {2} {3} \ ::: 1 2 3 seq 4 5 | parallel echo {1} {2} {3} :::: <(seq 6 7) - \ ::: 1 2 3
:::+ arguments Like ::: but linked like --link to the previous input source.
Contrary to --link, values do not wrap: The shortest input source determines the length.
parallel echo ::: a b c :::+ 1 2 3 ::: X Y :::+ 11 22
:::: argfiles Another way to write -a argfile1 -a argfile2 ...
::: and :::: can be mixed.
See -a, ::: and --link.
::::+ argfiles Like :::: but linked like --link to the previous input source.
--null -0  Use NUL as delimiter.  Normally input lines will end in \n (newline). If they end in \0 (NUL), then use this option. It is useful for processing arguments that may contain \n (newline).
--arg-file input-file -a input-file Use input-file as input source. If you use this option, stdin (standard input) is given to the first process run.  Otherwise, stdin (standard input) is redirected from /dev/null.
If multiple -a are given, each input-file will be treated as an input source, and all combinations of input sources will be generated. E.g. The file foo contains 1 2, the file bar contains a b c.  -a foo -a bar will result in the combinations (1,a) (1,b) (1,c) (2,a) (2,b) (2,c). This is useful for replacing nested for-loops.
See also: --link and {n}.
--arg-file-sep sep-str Use sep-str instead of :::: as separator string between command and argument files. Useful if :::: is used for something else by the command.
See also: ::::.
--arg-sep sep-str Use sep-str instead of ::: as separator string. Useful if ::: is used for something else by the command.
Also useful if you command uses ::: but you still want to read arguments from stdin (standard input): Simply change --arg-sep to a string that is not in the command line.
See also: :::.
--bar Show progress as a progress bar. In the bar is shown: % of jobs completed, estimated seconds left, and number of jobs started.
It is compatible with zenity:
seq 1000 | parallel -j30 --bar '(echo {};sleep 0.1)' \ 2> >(perl -pe 'BEGIN{$/="\r";$|=1};s/\r/\n/g' | zenity --progress --auto-kill) | wc
--basefile file --bf file file will be transferred to each sshlogin before a job is started. It will be removed if --cleanup is active. The file may be a script to run or some common base data needed for the job.  Multiple --bf can be specified to transfer more basefiles. The file will be transferred the same way as --transferfile.
--basenamereplace replace-str --bnr replace-str Use the replacement string replace-str instead of {/} for basename of input line.
--basenameextensionreplace replace-str --bner replace-str Use the replacement string replace-str instead of {/.} for basename of input line without extension.
--bin binexpr Use binexpr as binning key and bin input to the jobs.
binexpr is [column number|column name] [perlexpression] e.g. 3, Address, 3 $_%=100, Address s/\D//g.
Each input line is split using --colsep. The value of the column is put into $_, the perl expression is executed, the resulting value is is the job slot that will be given the line. If the value is bigger than the number of jobslots the value will be modulo number of jobslots.
This is similar to --shard but the hashing algorithm is a simple modulo, which makes it predictible which jobslot will receive which value.
The performance is in the order of 100K rows per second. Faster if the bincol is small (<10), slower if it is big (>100).
--bin requires --pipe and a fixed numeric value for --jobs.
See also: --shard, --group-by, --roundrobin.
--bg Run command in background thus GNU parallel will not wait for completion of the command before exiting. This is the default if --semaphore is set.
See also: --fg, man sem.
Implies --semaphore.
--bibtex --citation Print the citation notice and BibTeX entry for GNU parallel, silence citation notice for all future runs, and exit. It will not run any commands.
If it is impossible for you to run --citation you can instead use --will-cite, which will run commands, but which will only silence the citation notice for this single run.
If you use --will-cite in scripts to be run by others you are making it harder for others to see the citation notice.  The development of GNU parallel is indirectly financed through citations, so if your users do not know they should cite then you are making it harder to finance development. However, if you pay 10000 EUR, you have done your part to finance future development and should feel free to use --will-cite in scripts.
If you do not want to help financing future development by letting other users see the citation notice or by paying, then please use another tool instead of GNU parallel. You can find some of the alternatives in man parallel_alternatives.
--block size --block-size size Size of block in bytes to read at a time. The size can be postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
GNU parallel tries to meet the block size but can be off by the length of one record. For performance reasons size should be bigger than a two records. GNU parallel will warn you and automatically increase the size if you choose a size that is too small.
If you use -N, --block-size should be bigger than N+1 records.
size defaults to 1M.
When using --pipepart a negative block size is not interpreted as a blocksize but as the number of blocks each jobslot should have. So this will run 10*5 = 50 jobs in total:
parallel --pipepart -a myfile --block -10 -j5 wc
This is an efficient alternative to --roundrobin because data is never read by GNU parallel, but you can still have very few jobslots process a large amount of data.
See --pipe and --pipepart for use of this.
--blocktimeout duration --bt duration Time out for reading block when using --pipe. If it takes longer than duration to read a full block, use the partial block read so far.
duration must be in whole seconds, but can be expressed as floats postfixed with s, m, h, or d which would multiply the float by 1, 60, 3600, or 86400. Thus these are equivalent: --blocktimeout 100000 and --blocktimeout 1d3.5h16.6m4s.
--cat Create a temporary file with content. Normally --pipe/--pipepart will give data to the program on stdin (standard input). With --cat GNU parallel will create a temporary file with the name in {}, so you can do: parallel --pipe --cat wc {}.
Implies --pipe unless --pipepart is used.
See also: --fifo.
--cleanup Remove transferred files. --cleanup will remove the transferred files on the remote computer after processing is done.
find log -name '*gz' | parallel \ --sshlogin server.example.com --transferfile {} \ --return {.}.bz2 --cleanup "zcat {} | bzip -9 >{.}.bz2"
With --transferfile {} the file transferred to the remote computer will be removed on the remote computer. Directories on the remote computer containing the file will be removed if they are empty.
With --return the file transferred from the remote computer will be removed on the remote computer. Directories on the remote computer containing the file will be removed if they are empty.
--cleanup is ignored when not used with --transferfile or --return.
--colsep regexp -C regexp Column separator. The input will be treated as a table with regexp separating the columns. The n'th column can be accessed using {n} or {n.}. E.g. {3} is the 3rd column.
If there are more input sources, each input source will be separated, but the columns from each input source will be linked (see --link).
parallel --colsep '-' echo {4} {3} {2} {1} \ ::: A-B C-D ::: e-f g-h
--colsep implies --trim rl, which can be overridden with --trim n.
regexp is a Perl Regular Expression: https://perldoc.perl.org/perlre.html
--compress Compress temporary files. If the output is big and very compressible this will take up less disk space in $TMPDIR and possibly be faster due to less disk I/O.
GNU parallel will try pzstd, lbzip2, pbzip2, zstd, pigz, lz4, lzop, plzip, lzip, lrz, gzip, pxz, lzma, bzip2, xz, clzip, in that order, and use the first available.
--compress-program prg --decompress-program prg Use prg for (de)compressing temporary files. It is assumed that prg -dc will decompress stdin (standard input) to stdout (standard output) unless --decompress-program is given.
--csv Treat input as CSV-format. --colsep sets the field delimiter. It works very much like --colsep except it deals correctly with quoting:
echo '"1 big, 2 small","2""x4"" plank",12.34' | parallel --csv echo {1} of {2} at {3}
Even quoted newlines are parsed correctly:
(echo '"Start of field 1 with newline' echo 'Line 2 in field 1";value 2') | parallel --csv --colsep ';' echo Field 1: {1} Field 2: {2}
When used with --pipe only pass full CSV-records.
--ctag str Color tag. See --tag.
--ctagstring str Color tagstring. See --tagstring.
--delay mytime Delay starting next job by mytime. GNU parallel will pause mytime after starting each job. mytime is normally in seconds, but can be floats postfixed with s, m, h, or d which would multiply the float by 1, 60, 3600, or 86400. Thus these are equivalent: --delay 100000 and --delay 1d3.5h16.6m4s.
If you append 'auto' to mytime (e.g. 13m3sauto) GNU parallel will automatically try to find the optimal value: If a job fails, mytime is doubled. If a job succeeds, mytime is decreased by 10%.
--delimiter delim -d delim Input items are terminated by delim.  Quotes and backslash are not special; every character in the input is taken literally.  Disables the end-of-file string, which is treated like any other argument. The specified delimiter may be characters, C-style character escapes such as \n, or octal or hexadecimal escape codes.  Octal and hexadecimal escape codes are understood as for the printf command.  Multibyte characters are not supported.
--dirnamereplace replace-str --dnr replace-str Use the replacement string replace-str instead of {//} for dirname of input line.
--dry-run Print the job to run on stdout (standard output), but do not run the job. Use -v -v to include the wrapping that GNU parallel generates (for remote jobs, --tmux, --nice, --pipe, --pipepart, --fifo and --cat). Do not count on this literally, though, as the job may be scheduled on another computer or the local computer if : is in the list.
-E eof-str Set the end of file string to eof-str.  If the end of file string occurs as a line of input, the rest of the input is not read.  If neither -E nor -e is used, no end of file string is used.
--eof[=eof-str] -e[eof-str] This option is a synonym for the -E option.  Use -E instead, because it is POSIX compliant for xargs while this option is not.  If eof-str is omitted, there is no end of file string.  If neither -E nor -e is used, no end of file string is used.
--embed Embed GNU parallel in a shell script. If you need to distribute your script to someone who does not want to install GNU parallel you can embed GNU parallel in your own shell script:
parallel --embed > new_script
After which you add your code at the end of new_script. This is tested on ash, bash, dash, ksh, sh, and zsh.
--env var Copy environment variable var. This will copy var to the environment that the command is run in. This is especially useful for remote execution.
In Bash var can also be a Bash function - just remember to export -f the function, see command.
The variable '_' is special. It will copy all exported environment variables except for the ones mentioned in ~/.parallel/ignored_vars.
To copy the full environment (both exported and not exported variables, arrays, and functions) use env_parallel.
See also: --record-env, --session.
--eta Show the estimated number of seconds before finishing. This forces GNU parallel to read all jobs before starting to find the number of jobs. GNU parallel normally only reads the next job to run.
The estimate is based on the runtime of finished jobs, so the first estimate will only be shown when the first job has finished.
Implies --progress.
See also: --bar, --progress.
--fg Run command in foreground.
With --tmux and --tmuxpane GNU parallel will start tmux in the foreground.
With --semaphore GNU parallel will run the command in the foreground (opposite --bg), and wait for completion of the command before exiting.
See also: --bg, man sem.
--fifo Create a temporary fifo with content. Normally --pipe and --pipepart will give data to the program on stdin (standard input). With --fifo GNU parallel will create a temporary fifo with the name in {}, so you can do: parallel --pipe --fifo wc {}.
Beware: If data is not read from the fifo, the job will block forever.
See also: --cat.
--filter filter Only run jobs where filter is true. filter can contain replacement strings and Perl code. Example:
parallel --filter '{1} < {2}+1' echo ::: {1..3} ::: {1..3}
Outputs: 1,1 1,2 1,3 2,2 2,3 3,3
--filter-hosts Remove down hosts. For each remote host: check that login through ssh works. If not: do not use this host.
For performance reasons, this check is performed only at the start and every time --sshloginfile is changed. If an host goes down after the first check, it will go undetected until --sshloginfile is changed; --retries can be used to mitigate this.
Currently you can not put --filter-hosts in a profile, $PARALLEL, /etc/parallel/config or similar. This is because GNU parallel uses GNU parallel to compute this, so you will get an infinite loop. This will likely be fixed in a later release.
--gnu Behave like GNU parallel. This option historically took precedence over --tollef. The --tollef option is now retired, and therefore may not be used. --gnu is kept for compatibility.
--group Group output. Output from each job is grouped together and is only printed when the command is finished. Stdout (standard output) first followed by stderr (standard error).
This takes in the order of 0.5ms per job and depends on the speed of your disk for larger output. It can be disabled with -u, but this means output from different commands can get mixed.
--group is the default. Can be reversed with -u.
See also: --line-buffer --ungroup
--group-by val Group input by value. Combined with --pipe/--pipepart --group-by groups lines with the same value into a record.
The value can be computed from the full line or from a single column.
val can be:
column number Use the value in the column numbered.
column name   Treat the first line as a header and use the value in the column named.
(Not supported with --pipepart).
perl expression Run the perl expression and use $_ as the value.
column number perl expression Put the value of the column put in $_, run the perl expression, and use $_ as the value.
column name perl expression Put the value of the column put in $_, run the perl expression, and use $_ as the value.
UserID, Consumption 123,    1 123,    2 12-3,   1 221,    3 221,    1 2/21,   5
If you want to group 123, 12-3, 221, and 2/21 into 4 records and pass one record at a time to wc:
tail -n +2 table.csv | \ parallel --pipe --colsep , --group-by 1 -kN1 wc
Make GNU parallel treat the first line as a header:
cat table.csv | \ parallel --pipe --colsep , --header : --group-by 1 -kN1 wc
Address column by column name:
cat table.csv | \ parallel --pipe --colsep , --header : --group-by UserID -kN1 wc
If 12-3 and 123 are really the same UserID, remove non-digits in UserID when grouping:
cat table.csv | parallel --pipe --colsep , --header : \ --group-by 'UserID s/\D//g' -kN1 wc
See also: --shard, --roundrobin.
--help -h  Print a summary of the options to GNU parallel and exit.
--halt-on-error val --halt val When should GNU parallel terminate? In some situations it makes no sense to run all jobs. GNU parallel should simply give up as soon as a condition is met.
val defaults to never, which runs all jobs no matter what.
val can also take on the form of when,why.
when can be 'now' which means kill all running jobs and halt immediately, or it can be 'soon' which means wait for all running jobs to complete, but start no new jobs.
why can be 'fail=X', 'fail=Y%', 'success=X', 'success=Y%', 'done=X', or 'done=Y%' where X is the number of jobs that has to fail, succeed, or be done before halting, and Y is the percentage of jobs that has to fail, succeed, or be done before halting.
--halt now,fail=1     exit when the first job fails. Kill running jobs.
--halt soon,fail=3    exit when 3 jobs fail, but wait for running jobs to complete.
--halt soon,fail=3%   exit when 3% of the jobs have failed, but wait for running jobs to complete.
--halt now,success=1  exit when a job succeeds. Kill running jobs.
--halt soon,success=3 exit when 3 jobs succeeds, but wait for running jobs to complete.
--halt now,success=3% exit when 3% of the jobs have succeeded. Kill running jobs.
--halt now,done=1     exit when one of the jobs finishes. Kill running jobs.
--halt soon,done=3    exit when 3 jobs finishes, but wait for running jobs to complete.
--halt now,done=3%    exit when 3% of the jobs have finished. Kill running jobs.
For backwards compatibility these also work:
0           never
1           soon,fail=1
2           now,fail=1
-1          soon,success=1
-2          now,success=1
1-99%       soon,fail=1-99%
--header regexp Use regexp as header. For normal usage the matched header (typically the first line: --header '.*\n') will be split using --colsep (which will default to '\t') and column names can be used as replacement variables: {column name}, {column name/}, {column name//}, {column name/.}, {column name.}, {=column name perl expression =}, ..
For --pipe the matched header will be prepended to each output.
--header : is an alias for --header '.*\n'.
If regexp is a number, it is a fixed number of lines.
--hostgroups --hgrp Enable hostgroups on arguments. If an argument contains '@' the string after '@' will be removed and treated as a list of hostgroups on which this job is allowed to run. If there is no --sshlogin with a corresponding group, the job will run on any hostgroup.
parallel --hostgroups \ --sshlogin @grp1/myserver1 -S @grp1+grp2/myserver2 \ --sshlogin @grp3/myserver3 \ echo ::: my_grp1_arg@grp1 arg_for_grp2@grp2 third@grp1+grp3
my_grp1_arg may be run on either myserver1 or myserver2, third may be run on either myserver1 or myserver3, but arg_for_grp2 will only be run on myserver2.
See also: --sshlogin, $PARALLEL_HOSTGROUPS, $PARALLEL_ARGHOSTGROUPS.
-I replace-str Use the replacement string replace-str instead of {}.
--replace[=replace-str] -i[replace-str] This option is a synonym for -Ireplace-str if replace-str is specified, and for -I {} otherwise.  This option is deprecated; use -I instead.
--joblog logfile Logfile for executed jobs. Save a list of the executed jobs to logfile in the following TAB separated format: sequence number, sshlogin, start time as seconds since epoch, run time in seconds, bytes in files transferred, bytes in files returned, exit status, signal, and command run.
For --pipe bytes transferred and bytes returned are number of input and output of bytes.
If logfile is prepended with '+' log lines will be appended to the logfile.
To convert the times into ISO-8601 strict do:
cat logfile | perl -a -F"\t" -ne \ 'chomp($F[2]=`date -d \@$F[2] +%FT%T`); print join("\t",@F)'
If the host is long, you can use column -t to pretty print it:
cat joblog | column -t
See also: --resume --resume-failed.
--jobs N -j N --max-procs N -P N Number of jobslots on each machine. Run up to N jobs in parallel.  0 means as many as possible (this can take a while to determine). Default is 100% which will run one job per CPU on each machine.
Due to a bug -j 0 will also evaluate replacement strings twice up to the number of joblots:
# This will not count from 1 but from number-of-jobslots seq 10000 | parallel -j0   echo '{= $_ = $foo++; =}' | head # This will count from 1 seq 10000 | parallel -j100 echo '{= $_ = $foo++; =}' | head
If --semaphore is set, the default is 1 thus making a mutex.
--jobs +N -j +N --max-procs +N -P +N Add N to the number of CPUs.  Run this many jobs in parallel.  See also --use-cores-instead-of-threads and --use-sockets-instead-of-threads.
--jobs -N -j -N --max-procs -N -P -N Subtract N from the number of CPUs.  Run this many jobs in parallel. If the evaluated number is less than 1 then 1 will be used.  See also --use-cores-instead-of-threads and --use-sockets-instead-of-threads.
--jobs N% -j N% --max-procs N% -P N% Multiply N% with the number of CPUs.  Run this many jobs in parallel. See also --use-cores-instead-of-threads and --use-sockets-instead-of-threads.
--jobs procfile -j procfile --max-procs procfile -P procfile Read parameter from file. Use the content of procfile as parameter for -j. E.g. procfile could contain the string 100% or +2 or 10. If procfile is changed when a job completes, procfile is read again and the new number of jobs is computed. If the number is lower than before, running jobs will be allowed to finish but new jobs will not be started until the wanted number of jobs has been reached.  This makes it possible to change the number of simultaneous running jobs while GNU parallel is running.
--keep-order -k  Keep sequence of output same as the order of input. Normally the output of a job will be printed as soon as the job completes. Try this to see the difference:
parallel -j4 sleep {}\; echo {} ::: 2 1 4 3 parallel -j4 -k sleep {}\; echo {} ::: 2 1 4 3
If used with --onall or --nonall the output will grouped by sshlogin in sorted order.
If used with --pipe --roundrobin and the same input, the jobslots will get the same blocks in the same order in every run.
-k only affects the order in which the output is printed - not the order in which jobs are run.
-L recsize When used with --pipe: Read records of recsize.
When used otherwise: Use at most recsize nonblank input lines per command line.  Trailing blanks cause an input line to be logically continued on the next input line.
-L 0 means read one line, but insert 0 arguments on the command line.
recsize can be postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
Implies -X unless -m, --xargs, or --pipe is set.
--max-lines[=recsize] -l[recsize] When used with --pipe: Read records of recsize lines.
When used otherwise: Synonym for the -L option.  Unlike -L, the recsize argument is optional.  If recsize is not specified, it defaults to one.  The -l option is deprecated since the POSIX standard specifies -L instead.
-l 0 is an alias for -l 1.
--limit "command args" Dynamic job limit. Before starting a new job run command with args. The exit value of command determines what GNU parallel will do:
0   Below limit. Start another job.
1   Over limit. Start no jobs.
2   Way over limit. Kill the youngest job.
You can use any shell command. There are 3 predefined commands:
"io n"    Limit for I/O. The amount of disk I/O will be computed as a value 0-100, where 0 is no I/O and 100 is at least one disk is 100% saturated.
"load n"  Similar to --load.
"mem n"   Similar to --memfree.
--line-buffer --lb Buffer output on line basis. --group will keep the output together for a whole job. --ungroup allows output to mixup with half a line coming from one job and half a line coming from another job. --line-buffer fits between these two: GNU parallel will print a full line, but will allow for mixing lines of different jobs.
--line-buffer takes more CPU power than both --group and --ungroup, but can be much faster than --group if the CPU is not the limiting factor.
Normally --line-buffer does not buffer on disk, and can thus process an infinite amount of data, but it will buffer on disk when combined with: --keep-order, --results, --compress, and --files. This will make it as slow as --group and will limit output to the available disk space.
With --keep-order --line-buffer will output lines from the first job continuously while it is running, then lines from the second job while that is running. It will buffer full lines, but jobs will not mix. Compare:
parallel -j0 'echo {};sleep {};echo {}' ::: 1 3 2 4 parallel -j0 --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4 parallel -j0 -k --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4
See also: --group --ungroup
--xapply --link Link input sources. Read multiple input sources like xapply. If multiple input sources are given, one argument will be read from each of the input sources. The arguments can be accessed in the command as {1} .. {n}, so {1} will be a line from the first input source, and {6} will refer to the line with the same line number from the 6th input source.
Compare these two:
parallel echo {1} {2} ::: 1 2 3 ::: a b c parallel --link echo {1} {2} ::: 1 2 3 ::: a b c
Arguments will be recycled if one input source has more arguments than the others:
parallel --link echo {1} {2} {3} \ ::: 1 2 ::: I II III ::: a b c d e f g
See also --header, :::+, ::::+.
--load max-load Do not start new jobs on a given computer unless the number of running processes on the computer is less than max-load. max-load uses the same syntax as --jobs, so 100% for one per CPU is a valid setting. Only difference is 0 which is interpreted as 0.01.
--controlmaster -M  Use ssh's ControlMaster to make ssh connections faster. Useful if jobs run remote and are very fast to run. This is disabled for sshlogins that specify their own ssh command.
-m  Multiple arguments. Insert as many arguments as the command line length permits. If multiple jobs are being run in parallel: distribute the arguments evenly among the jobs. Use -j1 or --xargs to avoid this.
If {} is not used the arguments will be appended to the line.  If {} is used multiple times each {} will be replaced with all the arguments.
Support for -m with --sshlogin is limited and may fail.
See also -X for context replace. If in doubt use -X as that will most likely do what is needed.
--memfree size Minimum memory free when starting another job. The size can be postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
If the jobs take up very different amount of RAM, GNU parallel will only start as many as there is memory for. If less than size bytes are free, no more jobs will be started. If less than 50% size bytes are free, the youngest job will be killed, and put back on the queue to be run later.
--retries must be set to determine how many times GNU parallel should retry a given job.
See also: --memsuspend
--memsuspend size Suspend jobs when there is less than 2 * size memory free. The size can be postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
If the available memory falls below 2 * size, GNU parallel will suspend some of the running jobs. If the available memory falls below size, only one job will be running.
If a single job takes up at most size RAM, all jobs will complete without running out of memory. If you have swap available, you can usually lower size to around half the size of a single job - with the slight risk of swapping a little.
Jobs will be resumed when more RAM is available - typically when the oldest job completes.
--memsuspend only works on local jobs because there is no obvious way to suspend remote jobs.
See also: --memfree
--minversion version Print the version GNU parallel and exit.  If the current version of GNU parallel is less than version the exit code is 255. Otherwise it is 0.
This is useful for scripts that depend on features only available from a certain version of GNU parallel.
--max-args=max-args -n max-args Use at most max-args arguments per command line.  Fewer than max-args arguments will be used if the size (see the -s option) is exceeded, unless the -x option is given, in which case GNU parallel will exit.
-n 0 means read one argument, but insert 0 arguments on the command line.
max-args can be postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
Implies -X unless -m is set.
--max-replace-args=max-args -N max-args Use at most max-args arguments per command line. Like -n but also makes replacement strings {1} .. {max-args} that represents argument 1 .. max-args. If too few args the {n} will be empty.
-N 0 means read one argument, but insert 0 arguments on the command line.
This will set the owner of the homedir to the user:
tr ':' '\n' < /etc/passwd | parallel -N7 chown {1} {6}
Implies -X unless -m or --pipe is set.
When used with --pipe -N is the number of records to read. This is somewhat slower than --block.
--nonall --onall with no arguments. Run the command on all computers given with --sshlogin but take no arguments. GNU parallel will log into --jobs number of computers in parallel and run the job on the computer. -j adjusts how many computers to log into in parallel.
This is useful for running the same command (e.g. uptime) on a list of servers.
--onall Run all the jobs on all computers given with --sshlogin. GNU parallel will log into --jobs number of computers in parallel and run one job at a time on the computer. The order of the jobs will not be changed, but some computers may finish before others.
When using --group the output will be grouped by each server, so all the output from one server will be grouped together.
--joblog will contain an entry for each job on each server, so there will be several job sequence 1.
--output-as-files --outputasfiles --files Instead of printing the output to stdout (standard output) the output of each job is saved in a file and the filename is then printed.
See also: --results
--pipe --spreadstdin Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one job.
The block size is determined by --block. The strings --recstart and --recend tell GNU parallel how a record starts and/or ends. The block read will have the final partial record removed before the block is passed on to the job. The partial record will be prepended to next block.
If --recstart is given this will be used to split at record start.
If --recend is given this will be used to split at record end.
If both --recstart and --recend are given both will have to match to find a split position.
If neither --recstart nor --recend are given --recend defaults to '\n'. To have no record separator use --recend "".
--files is often used with --pipe.
--pipe maxes out at around 1 GB/s input, and 100 MB/s output. If performance is important use --pipepart.
See also: --recstart, --recend, --fifo, --cat, --pipepart, --files.
--pipepart Pipe parts of a physical file. --pipepart works similar to --pipe, but is much faster.
--pipepart has a few limitations:
•  The file must be a normal file or a block device (technically it must be seekable) and must be given using -a or ::::. The file cannot be a pipe, a fifo, or a stream as they are not seekable.
If using a block device with lot of NUL bytes, remember to set --recend ''.
•  Record counting (-N) and line counting (-L/-l) do not work. Instead use --recstart and --recend to determine where records end.
--plain Ignore any --profile, $PARALLEL, and ~/.parallel/config to get full control on the command line (used by GNU parallel internally when called with --sshlogin).