neulab/tldr huggingface dataset:
--plus Activate additional replacement strings: {+/} {+.} {+..} {+...} {..} {...} {/..} {/...} {##}. The idea being that '{+foo}' matches the opposite of '{foo}' and {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..} = {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}
{##} is the total number of jobs to be run. It is incompatible with -X/-m/--xargs.
{0%} zero-padded jobslot.
{0#} zero-padded sequence number.
{choose_k} is inspired by n choose k: Given a list of n elements, choose k. k is the number of input sources and n is the number of arguments in an input source.  The content of the input sources must be the same and the arguments must be unique.
Shorthands for variables:
{slot}        $PARALLEL_JOBSLOT (see {%}) {sshlogin}    $PARALLEL_SSHLOGIN {host}        $PARALLEL_SSHHOST {agrp}        $PARALLEL_ARGHOSTGROUPS {hgrp}        $PARALLEL_HOSTGROUPS
The following dynamic replacement strings are also activated. They are inspired by bash's parameter expansion:
{:-str}       str if the value is empty {:num}        remove the first num characters {:num1:num2}  characters from num1 to num2 {#regexp}     remove prefix regexp (non-greedy) {##regexp}    remove prefix regexp (greedy) {%regexp}     remove postfix regexp (non-greedy) {%%regexp}    remove postfix regexp (greedy) {/regexp/str} replace regexp with str {^str}        uppercase str if found at the start {^^str}       uppercase str {,str}        lowercase str if found at the start {,,str}       lowercase str
--progress Show progress of computations. List the computers involved in the task with number of CPUs detected and the max number of jobs to run. After that show progress for each computer: number of running jobs, number of completed jobs, and percentage of all jobs done by this computer. The percentage will only be available after all jobs have been scheduled as GNU parallel only read the next job when ready to schedule it - this is to avoid wasting time and memory by reading everything at startup.
By sending GNU parallel SIGUSR2 you can toggle turning on/off --progress on a running GNU parallel process.
See also: --eta and --bar.
--max-line-length-allowed Print the maximal number of characters allowed on the command line and exit (used by GNU parallel itself to determine the line length on remote computers).
--number-of-cpus (obsolete) Print the number of physical CPU cores and exit.
--number-of-cores Print the number of physical CPU cores and exit (used by GNU parallel itself to determine the number of physical CPU cores on remote computers).
--number-of-sockets Print the number of filled CPU sockets and exit (used by GNU parallel itself to determine the number of filled CPU sockets on remote computers).
--number-of-threads Print the number of hyperthreaded CPU cores and exit (used by GNU parallel itself to determine the number of hyperthreaded CPU cores on remote computers).
--no-keep-order Overrides an earlier --keep-order (e.g. if set in ~/.parallel/config).
--nice niceness Run the command at this niceness.
By default GNU parallel will run jobs at the same nice level as GNU parallel is started - both on the local machine and remote servers, so you are unlikely to ever use this option.
Setting --nice will override this nice level. If the nice level is smaller than the current nice level, it will only affect remote jobs (e.g. if current level is 10 then --nice 5 will cause local jobs to be run at level 10, but remote jobs run at nice level 5).
--interactive -p  Prompt the user about whether to run each command line and read a line from the terminal.  Only run the command line if the response starts with 'y' or 'Y'.  Implies -t.
--parens parensstring Define start and end parenthesis for {= perl expression =}. The left and the right parenthesis can be multiple characters and are assumed to be the same length. The default is {==} giving {= as the start parenthesis and =} as the end parenthesis.
Another useful setting is ,,,, which would make both parenthesis ,,:
parallel --parens ,,,, echo foo is ,,s/I/O/g,, ::: FII
See also: --rpl {= perl expression =}
--profile profilename -J profilename Use profile profilename for options. This is useful if you want to have multiple profiles. You could have one profile for running jobs in parallel on the local computer and a different profile for running jobs on remote computers. See the section PROFILE FILES for examples.
profilename corresponds to the file ~/.parallel/profilename.
You can give multiple profiles by repeating --profile. If parts of the profiles conflict, the later ones will be used.
Default: config
--quote -q  Quote command. If your command contains special characters that should not be interpreted by the shell (e.g. ; \ | *), use --quote to escape these. The command must be a simple command (see man bash) without redirections and without variable assignments.
See the section QUOTING. Most people will not need this.  Quoting is disabled by default.
--no-run-if-empty -r  If the stdin (standard input) only contains whitespace, do not run the command.
If used with --pipe this is slow.
--noswap Do not start new jobs on a given computer if there is both swap-in and swap-out activity.
The swap activity is only sampled every 10 seconds as the sampling takes 1 second to do.
Swap activity is computed as (swap-in)*(swap-out) which in practice is a good value: swapping out is not a problem, swapping in is not a problem, but both swapping in and out usually indicates a problem.
--memfree and --memsuspend may give better results, so try using those first.
--record-env Record current environment variables in ~/.parallel/ignored_vars. This is useful before using --env _.
See also: --env, --session.
--recstart startstring --recend endstring If --recstart is given startstring will be used to split at record start.
If --recend is given endstring will be used to split at record end.
If both --recstart and --recend are given the combined string endstringstartstring will have to match to find a split position. This is useful if either startstring or endstring match in the middle of a record.
If neither --recstart nor --recend are given then --recend defaults to '\n'. To have no record separator use --recend "".
--recstart and --recend are used with --pipe.
Use --regexp to interpret --recstart and --recend as regular expressions. This is slow, however.
--regexp Use --regexp to interpret --recstart and --recend as regular expressions. This is slow, however.
--remove-rec-sep --removerecsep --rrs Remove the text matched by --recstart and --recend before piping it to the command.
Only used with --pipe.
--results name --res name Save the output into files.
Simple string output dir
If name does not contain replacement strings and does not end in .csv/.tsv, the output will be stored in a directory tree rooted at name.  Within this directory tree, each command will result in three files: name/<ARGS>/stdout and name/<ARGS>/stderr, name/<ARGS>/seq, where <ARGS> is a sequence of directories representing the header of the input source (if using --header :) or the number of the input source and corresponding values.
E.g:
parallel --header : --results foo echo {a} {b} \ ::: a I II ::: b III IIII
will generate the files:
foo/a/II/b/III/seq foo/a/II/b/III/stderr foo/a/II/b/III/stdout foo/a/II/b/IIII/seq foo/a/II/b/IIII/stderr foo/a/II/b/IIII/stdout foo/a/I/b/III/seq foo/a/I/b/III/stderr foo/a/I/b/III/stdout foo/a/I/b/IIII/seq foo/a/I/b/IIII/stderr foo/a/I/b/IIII/stdout
parallel --results foo echo {1} {2} ::: I II ::: III IIII
foo/1/II/2/III/seq foo/1/II/2/III/stderr foo/1/II/2/III/stdout foo/1/II/2/IIII/seq foo/1/II/2/IIII/stderr foo/1/II/2/IIII/stdout foo/1/I/2/III/seq foo/1/I/2/III/stderr foo/1/I/2/III/stdout foo/1/I/2/IIII/seq foo/1/I/2/IIII/stderr foo/1/I/2/IIII/stdout
If name ends in .csv/.tsv the output will be a CSV-file named name.
.csv gives a comma separated value file. .tsv gives a TAB separated value file.
-.csv/-.tsv are special: It will give the file on stdout (standard output).
JSON file output
If name ends in .json the output will be a JSON-file named name.
-.json is special: It will give the file on stdout (standard output).
Replacement string output file
If name contains a replacement string and the replaced result does not end in /, then the standard output will be stored in a file named by this result. Standard error will be stored in the same file name with '.err' added, and the sequence number will be stored in the same file name with '.seq' added.
parallel --results my_{} echo ::: foo bar baz
my_bar my_bar.err my_bar.seq my_baz my_baz.err my_baz.seq my_foo my_foo.err my_foo.seq
Replacement string output dir
If name contains a replacement string and the replaced result ends in /, then output files will be stored in the resulting dir.
parallel --results my_{}/ echo ::: foo bar baz
my_bar/seq my_bar/stderr my_bar/stdout my_baz/seq my_baz/stderr my_baz/stdout my_foo/seq my_foo/stderr my_foo/stdout
See also: --files, --tag, --header, --joblog.
--resume Resumes from the last unfinished job. By reading --joblog or the --results dir GNU parallel will figure out the last unfinished job and continue from there. As GNU parallel only looks at the sequence numbers in --joblog then the input, the command, and --joblog all have to remain unchanged; otherwise GNU parallel may run wrong commands.
See also: --joblog, --results, --resume-failed, --retries.
--resume-failed Retry all failed and resume from the last unfinished job. By reading --joblog GNU parallel will figure out the failed jobs and run those again. After that it will resume last unfinished job and continue from there. As GNU parallel only looks at the sequence numbers in --joblog then the input, the command, and --joblog all have to remain unchanged; otherwise GNU parallel may run wrong commands.
See also: --joblog, --resume, --retry-failed, --retries.
--retry-failed Retry all failed jobs in joblog. By reading --joblog GNU parallel will figure out the failed jobs and run those again.
--retry-failed ignores the command and arguments on the command line: It only looks at the joblog.
Differences between --resume, --resume-failed, --retry-failed
In this example exit {= $_%=2 =} will cause every other job to fail.
timeout -k 1 4 parallel --joblog log -j10 \ 'sleep {}; exit {= $_%=2 =}' ::: {10..1}
4 jobs completed. 2 failed:
Seq   [...]   Exitval Signal  Command 10    [...]   1       0       sleep 1; exit 1 9     [...]   0       0       sleep 2; exit 0 8     [...]   1       0       sleep 3; exit 1 7     [...]   0       0       sleep 4; exit 0
--resume does not care about the Exitval, but only looks at Seq. If the Seq is run, it will not be run again. So if needed, you can change the command for the seqs not run yet:
parallel --resume --joblog log -j10 \ 'sleep .{}; exit {= $_%=2 =}' ::: {10..1}
Seq   [...]   Exitval Signal  Command [... as above ...] 1     [...]   0       0       sleep .10; exit 0 6     [...]   1       0       sleep .5; exit 1 5     [...]   0       0       sleep .6; exit 0 4     [...]   1       0       sleep .7; exit 1 3     [...]   0       0       sleep .8; exit 0 2     [...]   1       0       sleep .9; exit 1
--resume-failed cares about the Exitval, but also only looks at Seq to figure out which commands to run. Again this means you can change the command, but not the arguments. It will run the failed seqs and the seqs not yet run:
parallel --resume-failed --joblog log -j10 \ 'echo {};sleep .{}; exit {= $_%=3 =}' ::: {10..1}
Seq   [...]   Exitval Signal  Command [... as above ...] 10    [...]   1       0       echo 1;sleep .1; exit 1 8     [...]   0       0       echo 3;sleep .3; exit 0 6     [...]   2       0       echo 5;sleep .5; exit 2 4     [...]   1       0       echo 7;sleep .7; exit 1 2     [...]   0       0       echo 9;sleep .9; exit 0
--retry-failed cares about the Exitval, but takes the command from the joblog. It ignores any arguments or commands given on the command line:
parallel --retry-failed --joblog log -j10 this part is ignored
Seq   [...]   Exitval Signal  Command [... as above ...] 10    [...]   1       0       echo 1;sleep .1; exit 1 6     [...]   2       0       echo 5;sleep .5; exit 2 4     [...]   1       0       echo 7;sleep .7; exit 1
See also: --joblog, --resume, --resume-failed, --retries.
--retries n If a job fails, retry it on another computer on which it has not failed. Do this n times. If there are fewer than n computers in --sshlogin GNU parallel will re-use all the computers. This is useful if some jobs fail for no apparent reason (such as network failure).
--return filename Transfer files from remote computers. --return is used with --sshlogin when the arguments are files on the remote computers. When processing is done the file filename will be transferred from the remote computer using rsync and will be put relative to the default login dir. E.g.
echo foo/bar.txt | parallel --return {.}.out \ --sshlogin server.example.com touch {.}.out
This will transfer the file $HOME/foo/bar.out from the computer server.example.com to the file foo/bar.out after running touch foo/bar.out on server.example.com.
parallel -S server --trc out/./{}.out touch {}.out ::: in/file
This will transfer the file in/file.out from the computer server.example.com to the files out/in/file.out after running touch in/file.out on server.
echo /tmp/foo/bar.txt | parallel --return {.}.out \ --sshlogin server.example.com touch {.}.out
This will transfer the file /tmp/foo/bar.out from the computer server.example.com to the file /tmp/foo/bar.out after running touch /tmp/foo/bar.out on server.example.com.
Multiple files can be transferred by repeating the option multiple times:
echo /tmp/foo/bar.txt | parallel \ --sshlogin server.example.com \ --return {.}.out --return {.}.out2 touch {.}.out {.}.out2
--return is often used with --transferfile and --cleanup.
--return is ignored when used with --sshlogin : or when not used with --sshlogin.
For details on transferring see --transferfile.
--round-robin --round Normally --pipe will give a single block to each instance of the command. With --roundrobin all blocks will at random be written to commands already running. This is useful if the command takes a long time to initialize.
--keep-order will not work with --roundrobin as it is impossible to track which input block corresponds to which output.
--roundrobin implies --pipe, except if --pipepart is given.
See also: --group-by, --shard.
--rpl 'tag perl expression' Use tag as a replacement string for perl expression. This makes it possible to define your own replacement strings. GNU parallel's 7 replacement strings are implemented as:
--rpl '{} ' --rpl '{#} 1 $_=$job->seq()' --rpl '{%} 1 $_=$job->slot()' --rpl '{/} s:.*/::' --rpl '{//} $Global::use{"File::Basename"} ||= eval "use File::Basename; 1;"; $_ = dirname($_);' --rpl '{/.} s:.*/::; s:\.[^/.]+$::;' --rpl '{.} s:\.[^/.]+$::'
The --plus replacement strings are implemented as:
--rpl '{+/} s:/[^/]*$::' --rpl '{+.} s:.*\.::' --rpl '{+..} s:.*\.([^.]*\.):$1:' --rpl '{+...} s:.*\.([^.]*\.[^.]*\.):$1:' --rpl '{..} s:\.[^/.]+$::; s:\.[^/.]+$::' --rpl '{...} s:\.[^/.]+$::; s:\.[^/.]+$::; s:\.[^/.]+$::' --rpl '{/..} s:.*/::; s:\.[^/.]+$::; s:\.[^/.]+$::' --rpl '{/...} s:.*/::;s:\.[^/.]+$::;s:\.[^/.]+$::;s:\.[^/.]+$::' --rpl '{##} $_=total_jobs()' --rpl '{:-(.+?)} $_ ||= $$1' --rpl '{:(\d+?)} substr($_,0,$$1) = ""' --rpl '{:(\d+?):(\d+?)} $_ = substr($_,$$1,$$2);' --rpl '{#([^#].*?)} s/^$$1//;' --rpl '{%(.+?)} s/$$1$//;' --rpl '{/(.+?)/(.*?)} s/$$1/$$2/;' --rpl '{^(.+?)} s/^($$1)/uc($1)/e;' --rpl '{^^(.+?)} s/($$1)/uc($1)/eg;' --rpl '{,(.+?)} s/^($$1)/lc($1)/e;' --rpl '{,,(.+?)} s/($$1)/lc($1)/eg;'
If the user defined replacement string starts with '{' it can also be used as a positional replacement string (like {2.}).
It is recommended to only change $_ but you have full access to all of GNU parallel's internal functions and data structures.
Is the job sequence even or odd? --rpl '{odd} $_ = seq() % 2 ? "odd" : "even"' Pad job sequence with leading zeros to get equal width --rpl '{0#} $f=1+int("".(log(total_jobs())/log(10))); $_=sprintf("%0${f}d",seq())' Job sequence counting from 0 --rpl '{#0} $_ = seq() - 1' Job slot counting from 2 --rpl '{%1} $_ = slot() + 1' Remove all extensions --rpl '{:} s:(\.[^/]+)*$::'
You can have dynamic replacement strings by including parenthesis in the replacement string and adding a regular expression between the parenthesis. The matching string will be inserted as $$1:
parallel --rpl '{%(.*?)} s/$$1//' echo {%.tar.gz} ::: my.tar.gz parallel --rpl '{:%(.+?)} s:$$1(\.[^/]+)*$::' \ echo {:%_file} ::: my_file.tar.gz parallel -n3 --rpl '{/:%(.*?)} s:.*/(.*)$$1(\.[^/]+)*$:$1:' \ echo job {#}: {2} {2.} {3/:%_1} ::: a/b.c c/d.e f/g_1.h.i
You can even use multiple matches:
parallel --rpl '{/(.+?)/(.*?)} s/$$1/$$2/;' echo {/replacethis/withthis} {/b/C} ::: a_replacethis_b
parallel --rpl '{(.*?)/(.*?)} $_="$$2$_$$1"' \ echo {swap/these} ::: -middle-
See also: {= perl expression =} --parens
--rsync-opts options Options to pass on to rsync. Setting --rsync-opts takes precedence over setting the environment variable $PARALLEL_RSYNC_OPTS.
--max-chars=max-chars -s max-chars Use at most max-chars characters per command line, including the command and initial-arguments and the terminating nulls at the ends of the argument strings.  The largest allowed value is system- dependent, and is calculated as the argument length limit for exec, less the size of your environment.  The default value is the maximum.
max-chars can be postfixed with K, M, G, T, P, k, m, g, t, or p (see UNIT PREFIX).
--show-limits Display the limits on the command-line length which are imposed by the operating system and the -s option.  Pipe the input from /dev/null (and perhaps specify --no-run-if-empty) if you don't want GNU parallel to do anything.
--semaphore Work as a counting semaphore. --semaphore will cause GNU parallel to start command in the background. When the number of jobs given by --jobs is reached, GNU parallel will wait for one of these to complete before starting another command.
--semaphore implies --bg unless --fg is specified.
--semaphore implies --semaphorename `tty` unless --semaphorename is specified.
Used with --fg, --wait, and --semaphorename.
The command sem is an alias for parallel --semaphore.
See also: man sem.
--semaphorename name --id name Use name as the name of the semaphore. Default is the name of the controlling tty (output from tty).
The default normally works as expected when used interactively, but when used in a script name should be set. $$ or my_task_name are often a good value.
The semaphore is stored in ~/.parallel/semaphores/
--semaphoretimeout secs --st secs If secs > 0: If the semaphore is not released within secs seconds, take it anyway.
If secs < 0: If the semaphore is not released within secs seconds, exit.
--seqreplace replace-str Use the replacement string replace-str instead of {#} for job sequence number.
--session Record names in current environment in $PARALLEL_IGNORED_NAMES and exit. Only used with env_parallel. Aliases, functions, and variables with names in $PARALLEL_IGNORED_NAMES will not be copied.
Only supported in Ash, Bash, Dash, Ksh, Sh, and Zsh.
See also: --env, --record-env.
--shard shardexpr Use shardexpr as shard key and shard input to the jobs.
shardexpr is [column number|column name] [perlexpression] e.g. 3, Address, 3 $_%=100, Address s/\d//g.
Each input line is split using --colsep. The value of the column is put into $_, the perl expression is executed, the resulting value is hashed so that all lines of a given value is given to the same job slot.
This is similar to sharding in databases.
The performance is in the order of 100K rows per second. Faster if the shardcol is small (<10), slower if it is big (>100).
--shard requires --pipe and a fixed numeric value for --jobs.
See also: --bin, --group-by, --roundrobin.
--shebang --hashbang GNU parallel can be called as a shebang (#!) command as the first line of a script. The content of the file will be treated as inputsource.
#!/usr/bin/parallel --shebang -r wget
https://ftpmirror.gnu.org/parallel/parallel-20120822.tar.bz2 https://ftpmirror.gnu.org/parallel/parallel-20130822.tar.bz2 https://ftpmirror.gnu.org/parallel/parallel-20140822.tar.bz2
--shebang must be set as the first option.
On FreeBSD env is needed:
#!/usr/bin/env -S parallel --shebang -r wget
There are many limitations of shebang (#!) depending on your operating system. See details on https://www.in-ulm.de/~mascheck/various/shebang/
--shebang-wrap GNU parallel can parallelize scripts by wrapping the shebang line. If the program can be run like this:
cat arguments | parallel the_program
then the script can be changed to:
#!/usr/bin/parallel --shebang-wrap /original/parser --options
#!/usr/bin/parallel --shebang-wrap /usr/bin/python
If the program can be run like this:
cat data | parallel --pipe the_program
#!/usr/bin/parallel --shebang-wrap --pipe /orig/parser --opts
#!/usr/bin/parallel --shebang-wrap --pipe /usr/bin/perl -w
--shebang-wrap must be set as the first option.
--shellquote Does not run the command but quotes it. Useful for making quoted composed commands for GNU parallel.
Multiple --shellquote with quote the string multiple times, so parallel --shellquote | parallel --shellquote can be written as parallel --shellquote --shellquote.
--shuf Shuffle jobs. When having multiple input sources it is hard to randomize jobs. --shuf will generate all jobs, and shuffle them before running them. This is useful to get a quick preview of the results before running the full batch.
--skip-first-line Do not use the first line of input (used by GNU parallel itself when called with --shebang).
--sql DBURL (obsolete) Use --sqlmaster instead.
--sqlmaster DBURL Submit jobs via SQL server. DBURL must point to a table, which will contain the same information as --joblog, the values from the input sources (stored in columns V1 .. Vn), and the output (stored in columns Stdout and Stderr).
If DBURL is prepended with '+' GNU parallel assumes the table is already made with the correct columns and appends the jobs to it.
If DBURL is not prepended with '+' the table will be dropped and created with the correct amount of V-columns unless
--sqlmaster does not run any jobs, but it creates the values for the jobs to be run. One or more --sqlworker must be run to actually execute the jobs.
If --wait is set, GNU parallel will wait for the jobs to complete.
The format of a DBURL is:
[sql:]vendor://[[user][:pwd]@][host][:port]/[db]/table
sql:mysql://hr:hr@localhost:3306/hrdb/jobs mysql://scott:tiger@my.example.com/pardb/paralleljobs sql:oracle://scott:tiger@ora.example.com/xe/parjob postgresql://scott:tiger@pg.example.com/pgdb/parjob pg:///parjob sqlite3:///%2Ftmp%2Fpardb.sqlite/parjob csv:///%2Ftmp%2Fpardb/parjob
Notice how / in the path of sqlite and CVS must be encoded as %2F. Except the last / in CSV which must be a /.
It can also be an alias from ~/.sql/aliases:
:myalias mysql:///mydb/paralleljobs
--sqlandworker DBURL Shorthand for: --sqlmaster DBURL --sqlworker DBURL.
--sqlworker DBURL Execute jobs via SQL server. Read the input sources variables from the table pointed to by DBURL. The command on the command line should be the same as given by --sqlmaster.
If you have more than one --sqlworker jobs may be run more than once.
If --sqlworker runs on the local machine, the hostname in the SQL table will not be ':' but instead the hostname of the machine.
--ssh sshcommand GNU parallel defaults to using ssh for remote access. This can be overridden with --ssh. It can also be set on a per server basis (see --sshlogin).
--sshdelay mytime Delay starting next ssh by mytime. GNU parallel will not start another ssh for the next mytime.
For details on mytime see --delay.
-S [@hostgroups/][ncpus/]sshlogin[,[@hostgroups/][ncpus/]sshlogin[,...]] -S @hostgroup --sshlogin [@hostgroups/][ncpus/]sshlogin[,[@hostgroups/][ncpus/]sshlogin[,...]] --sshlogin @hostgroup Distribute jobs to remote computers. The jobs will be run on a list of remote computers.
If hostgroups is given, the sshlogin will be added to that hostgroup. Multiple hostgroups are separated by '+'. The sshlogin will always be added to a hostgroup named the same as sshlogin.
If only the @hostgroup is given, only the sshlogins in that hostgroup will be used. Multiple @hostgroup can be given.
GNU parallel will determine the number of CPUs on the remote computers and run the number of jobs as specified by -j.  If the number ncpus is given GNU parallel will use this number for number of CPUs on the host. Normally ncpus will not be needed.
An sshlogin is of the form:
[sshcommand [options]] [username@]hostname
The sshlogin must not require a password (ssh-agent, ssh-copy-id, and sshpass may help with that).
The sshlogin ':' is special, it means 'no ssh' and will therefore run on the local computer.
The sshlogin '..' is special, it read sshlogins from ~/.parallel/sshloginfile or $XDG_CONFIG_HOME/parallel/sshloginfile
The sshlogin '-' is special, too, it read sshlogins from stdin (standard input).
To specify more sshlogins separate the sshlogins by comma, newline (in the same string), or repeat the options multiple times.
For examples: see --sshloginfile.
The remote host must have GNU parallel installed.
--sshlogin is known to cause problems with -m and -X.
--sshlogin is often used with --transferfile, --return, --cleanup, and --trc.
--sshloginfile filename --slf filename File with sshlogins. The file consists of sshlogins on separate lines. Empty lines and lines starting with '#' are ignored. Example:
server.example.com username@server2.example.com 8/my-8-cpu-server.example.com 2/my_other_username@my-dualcore.example.net # This server has SSH running on port 2222 ssh -p 2222 server.example.net 4/ssh -p 2222 quadserver.example.net # Use a different ssh program myssh -p 2222 -l myusername hexacpu.example.net # Use a different ssh program with default number of CPUs //usr/local/bin/myssh -p 2222 -l myusername hexacpu # Use a different ssh program with 6 CPUs 6//usr/local/bin/myssh -p 2222 -l myusername hexacpu # Assume 16 CPUs on the local computer 16/: # Put server1 in hostgroup1 @hostgroup1/server1 # Put myusername@server2 in hostgroup1+hostgroup2 @hostgroup1+hostgroup2/myusername@server2 # Force 4 CPUs and put 'ssh -p 2222 server3' in hostgroup1 @hostgroup1/4/ssh -p 2222 server3
When using a different ssh program the last argument must be the hostname.
Multiple --sshloginfile are allowed.
GNU parallel will first look for the file in current dir; if that fails it look for the file in ~/.parallel.
The sshloginfile '..' is special, it read sshlogins from ~/.parallel/sshloginfile
The sshloginfile '.' is special, it read sshlogins from /etc/parallel/sshloginfile
The sshloginfile '-' is special, too, it read sshlogins from stdin (standard input).
If the sshloginfile is changed it will be re-read when a job finishes though at most once per second. This makes it possible to add and remove hosts while running.
This can be used to have a daemon that updates the sshloginfile to only contain servers that are up:
cp original.slf tmp2.slf while [ 1 ] ; do nice parallel --nonall -j0 -k --slf original.slf \ --tag echo | perl 's/\t$//' > tmp.slf if diff tmp.slf tmp2.slf; then mv tmp.slf tmp2.slf fi sleep 10 done & parallel --slf tmp2.slf ...
--slotreplace replace-str Use the replacement string replace-str instead of {%} for job slot number.
--silent Silent.  The job to be run will not be printed. This is the default. Can be reversed with -v.
--template file=repl --tmpl file=repl Copy file to repl. All replacement strings in the contents of file will be replaced. All replacement strings in the name repl will be replaced.
With --cleanup the new file will be removed when the job is done.
If my.tmpl contains this:
Xval: {x} Yval: {y} FixedValue: 9 # x with 2 decimals DecimalX: {=x $_=sprintf("%.2f",$_) =} TenX: {=x $_=$_*10 =} RandomVal: {=1 $_=rand() =}
it can be used like this:
myprog() { echo Using "$@"; cat "$@"; } export -f myprog parallel --cleanup --header : --tmpl my.tmpl={#}.t myprog {#}.t \ ::: x 1.234 2.345 3.45678 ::: y 1 2 3
--tty Open terminal tty. If GNU parallel is used for starting a program that accesses the tty (such as an interactive program) then this option may be needed. It will default to starting only one job at a time (i.e. -j1), not buffer the output (i.e. -u), and it will open a tty for the job.
You can of course override -j1 and -u.
Using --tty unfortunately means that GNU parallel cannot kill the jobs (with --timeout, --memfree, or --halt). This is due to GNU parallel giving each child its own process group, which is then killed. Process groups are dependant on the tty.
--tag Tag lines with arguments. Each output line will be prepended with the arguments and TAB (\t). When combined with --onall or --nonall the lines will be prepended with the sshlogin instead.
--tag is ignored when using -u.
--ctag gives the tag a color.
--tagstring str Tag lines with a string. Each output line will be prepended with str and TAB (\t). str can contain replacement strings such as {}.
--tagstring is ignored when using -u, --onall, and --nonall.
--ctagstring gives the tag a color.
--tee Pipe all data to all jobs. Used with --pipe/--pipepart and :::.
seq 1000 | parallel --pipe --tee -v wc {} ::: -w -l -c
How many numbers in 1..1000 contain 0..9, and how many bytes do they fill:
seq 1000 | parallel --pipe --tee --tag \ 'grep {1} | wc {2}' ::: {0..9} ::: -l -c
How many words contain a..z and how many bytes do they fill?
parallel -a /usr/share/dict/words --pipepart --tee --tag \ 'grep {1} | wc {2}' ::: {a..z} ::: -l -c
--termseq sequence Termination sequence. When a job is killed due to --timeout, --memfree, --halt, or abnormal termination of GNU parallel, sequence determines how the job is killed. The default is:
TERM,200,TERM,100,TERM,50,KILL,25
which sends a TERM signal, waits 200 ms, sends another TERM signal, waits 100 ms, sends another TERM signal, waits 50 ms, sends a KILL signal, waits 25 ms, and exits. GNU parallel detects if a process dies before the waiting time is up.
--tmpdir dirname Directory for temporary files. GNU parallel normally buffers output into temporary files in /tmp. By setting --tmpdir you can use a different dir for the files. Setting --tmpdir is equivalent to setting $TMPDIR.
--tmux (Long beta testing) Use tmux for output. Start a tmux session and run each job in a window in that session. No other output will be produced.
--tmuxpane (Long beta testing) Use tmux for output but put output into panes in the first window. Useful if you want to monitor the progress of less than 100 concurrent jobs.
--timeout duration Time out for command. If the command runs for longer than duration seconds it will get killed as per --termseq.
If duration is followed by a % then the timeout will dynamically be computed as a percentage of the median average runtime of successful jobs. Only values > 100% will make sense.
duration is normally in seconds, but can be floats postfixed with s, m, h, or d which would multiply the float by 1, 60, 3600, or 86400. Thus these are equivalent: --timeout 100000 and --timeout 1d3.5h16.6m4s.
--verbose -t  Print the job to be run on stderr (standard error).
See also: -v, -p.
--transfer Transfer files to remote computers. Shorthand for: --transferfile {}.
--transferfile filename --tf filename --transferfile is used with --sshlogin to transfer files to the remote computers. The files will be transferred using rsync and will be put relative to the work dir (see --workdir).
The filename will normally contain a replacement string.
If the path contains /./ the remaining path will be relative to the work dir (for details: see rsync). If the work dir is /home/user, the transferring will be as follows:
/tmp/foo/bar   => /tmp/foo/bar tmp/foo/bar    => /home/user/tmp/foo/bar /tmp/./foo/bar => /home/user/foo/bar tmp/./foo/bar  => /home/user/foo/bar
This will transfer the file foo/bar.txt to the computer server.example.com to the file $HOME/foo/bar.txt before running wc foo/bar.txt on server.example.com:
echo foo/bar.txt | parallel --transferfile {} \ --sshlogin server.example.com wc
This will transfer the file /tmp/foo/bar.txt to the computer server.example.com to the file /tmp/foo/bar.txt before running wc /tmp/foo/bar.txt on server.example.com:
echo /tmp/foo/bar.txt | parallel --transferfile {} \ --sshlogin server.example.com wc
This will transfer the file /tmp/foo/bar.txt to the computer server.example.com to the file foo/bar.txt before running wc ./foo/bar.txt on server.example.com:
echo /tmp/./foo/bar.txt | parallel --transferfile {} \ --sshlogin server.example.com wc {= s:.*/\./:./: =}
--transferfile is often used with --return and --cleanup. A shorthand for --transferfile {} is --transfer.
--transferfile is ignored when used with --sshlogin : or when not used with --sshlogin.
--trc filename Transfer, Return, Cleanup. Shorthand for:
--transferfile {} --return filename --cleanup
--trim <n|l|r|lr|rl> Trim white space in input.
n   No trim. Input is not modified. This is the default.
l   Left trim. Remove white space from start of input. E.g. " a bc " -> "a bc ".
r   Right trim. Remove white space from end of input. E.g. " a bc " -> " a bc".
lr rl  Both trim. Remove white space from both start and end of input. E.g. " a bc " -> "a bc". This is the default if --colsep is used.
--ungroup -u  Ungroup output.  Output is printed as soon as possible and bypasses GNU parallel internal processing. This may cause output from different commands to be mixed thus should only be used if you do not care about the output. Compare these:
seq 4 | parallel -j0 \ 'sleep {};echo -n start{};sleep {};echo {}end' seq 4 | parallel -u -j0 \ 'sleep {};echo -n start{};sleep {};echo {}end'
It also disables --tag. GNU parallel outputs faster with -u. Compare the speeds of these:
parallel seq ::: 300000000 >/dev/null parallel -u seq ::: 300000000 >/dev/null parallel --line-buffer seq ::: 300000000 >/dev/null
Can be reversed with --group.
See also: --line-buffer --group
--extensionreplace replace-str --er replace-str Use the replacement string replace-str instead of {.} for input line without extension.
--use-sockets-instead-of-threads --use-cores-instead-of-threads --use-cpus-instead-of-cores (obsolete) Determine how GNU parallel counts the number of CPUs. GNU parallel uses this number when the number of jobslots is computed relative to the number of CPUs (e.g. 100% or +1).
CPUs can be counted in three different ways:
sockets The number of filled CPU sockets (i.e. the number of physical chips).
cores   The number of physical cores (i.e. the number of physical compute cores).
threads The number of hyperthreaded cores (i.e. the number of virtual cores - with some of them possibly being hyperthreaded)
Normally the number of CPUs is computed as the number of CPU threads. With --use-sockets-instead-of-threads or --use-cores-instead-of-threads you can force it to be computed as the number of filled sockets or number of cores instead.
Most users will not need these options.
--use-cpus-instead-of-cores is a (misleading) alias for --use-sockets-instead-of-threads and is kept for backwards compatibility.
-v  Verbose.  Print the job to be run on stdout (standard output). Can be reversed with --silent. See also -t.
Use -v -v to print the wrapping ssh command when running remotely.
--version -V  Print the version GNU parallel and exit.
--workdir mydir --wd mydir Jobs will be run in the dir mydir. The default is the current dir for the local machine, and the login dir for remote computers.
Files transferred using --transferfile and --return will be relative to mydir on remote computers.
The special mydir value ... will create working dirs under ~/.parallel/tmp/. If --cleanup is given these dirs will be removed.
The special mydir value . uses the current working dir.  If the current working dir is beneath your home dir, the value . is treated as the relative path to your home dir. This means that if your home dir is different on remote computers (e.g. if your login is different) the relative path will still be relative to your home dir.
To see the difference try:
parallel -S server pwd ::: "" parallel --wd . -S server pwd ::: "" parallel --wd ... -S server pwd ::: ""
mydir can contain GNU parallel's replacement strings.
--wait Wait for all commands to complete.
Used with --semaphore or --sqlmaster.
-X  Multiple arguments with context replace. Insert as many arguments as the command line length permits. If multiple jobs are being run in parallel: distribute the arguments evenly among the jobs. Use -j1 to avoid this.
If {} is not used the arguments will be appended to the line.  If {} is used as part of a word (like pic{}.jpg) then the whole word will be repeated. If {} is used multiple times each {} will be replaced with the arguments.
Normally -X will do the right thing, whereas -m can give unexpected results if {} is used as part of a word.
Support for -X with --sshlogin is limited and may fail.
See also: -m.
--exit -x  Exit if the size (see the -s option) is exceeded.
--xargs Multiple arguments. Insert as many arguments as the command line length permits.
Support for --xargs with --sshlogin is limited and may fail.
EXAMPLE: Working as xargs -n1. Argument appending GNU parallel can work similar to xargs -n1.
To compress all html files using gzip run:
find . -name '*.html' | parallel gzip --best
If the file names may contain a newline use -0. Substitute FOO BAR with FUBAR in all files in this dir and subdirs:
find . -type f -print0 | \ parallel -q0 perl -i -pe 's/FOO BAR/FUBAR/g'
Note -q is needed because of the space in 'FOO BAR'.
EXAMPLE: Simple network scanner prips can generate IP-addresses from CIDR notation. With GNU parallel you can build a simple network scanner to see which addresses respond to ping:
prips 130.229.16.0/20 | \ parallel --timeout 2 -j0 \ 'ping -c 1 {} >/dev/null && echo {}' 2>/dev/null
EXAMPLE: Reading arguments from command line GNU parallel can take the arguments from command line instead of stdin (standard input). To compress all html files in the current dir using gzip run:
parallel gzip --best ::: *.html
To convert *.wav to *.mp3 using LAME running one process per CPU run:
parallel lame {} -o {.}.mp3 ::: *.wav
EXAMPLE: Inserting multiple arguments When moving a lot of files like this: mv *.log destdir you will sometimes get the error:
bash: /bin/mv: Argument list too long
because there are too many files. You can instead do:
ls | grep -E '\.log$' | parallel mv {} destdir
This will run mv for each file. It can be done faster if mv gets as many arguments that will fit on the line:
ls | grep -E '\.log$' | parallel -m mv {} destdir
In many shells you can also use printf:
printf '%s\0' *.log | parallel -0 -m mv {} destdir
EXAMPLE: Context replace To remove the files pict0000.jpg .. pict9999.jpg you could do:
seq -w 0 9999 | parallel rm pict{}.jpg
You could also do:
seq -w 0 9999 | perl -pe 's/(.*)/pict$1.jpg/' | parallel -m rm
The first will run rm 10000 times, while the last will only run rm as many times needed to keep the command line length short enough to avoid Argument list too long (it typically runs 1-2 times).
You could also run:
seq -w 0 9999 | parallel -X rm pict{}.jpg
This will also only run rm as many times needed to keep the command line length short enough.
EXAMPLE: Compute intensive jobs and substitution If ImageMagick is installed this will generate a thumbnail of a jpg file:
convert -geometry 120 foo.jpg thumb_foo.jpg
This will run with number-of-cpus jobs in parallel for all jpg files in a directory:
ls *.jpg | parallel convert -geometry 120 {} thumb_{}
To do it recursively use find:
find . -name '*.jpg' | \ parallel convert -geometry 120 {} {}_thumb.jpg
Notice how the argument has to start with {} as {} will include path (e.g. running convert -geometry 120 ./foo/bar.jpg thumb_./foo/bar.jpg would clearly be wrong). The command will generate files like ./foo/bar.jpg_thumb.jpg.
Use {.} to avoid the extra .jpg in the file name. This command will make files like ./foo/bar_thumb.jpg:
find . -name '*.jpg' | \ parallel convert -geometry 120 {} {.}_thumb.jpg
EXAMPLE: Substitution and redirection This will generate an uncompressed version of .gz-files next to the .gz-file:
parallel zcat {} ">"{.} ::: *.gz
Quoting of > is necessary to postpone the redirection. Another solution is to quote the whole command:
parallel "zcat {} >{.}" ::: *.gz
Other special shell characters (such as * ; $ > < |  >> <<) also need to be put in quotes, as they may otherwise be interpreted by the shell and not given to GNU parallel.
EXAMPLE: Composed commands A job can consist of several commands. This will print the number of files in each directory:
ls | parallel 'echo -n {}" "; ls {}|wc -l'
To put the output in a file called <name>.dir:
ls | parallel '(echo -n {}" "; ls {}|wc -l) >{}.dir'
Even small shell scripts can be run by GNU parallel:
find . | parallel 'a={}; name=${a##*/};' \ 'upper=$(echo "$name" | tr "[:lower:]" "[:upper:]");'\ 'echo "$name - $upper"'
ls | parallel 'mv {} "$(echo {} | tr "[:upper:]" "[:lower:]")"'
Given a list of URLs, list all URLs that fail to download. Print the line number and the URL.
cat urlfile | parallel "wget {} 2>/dev/null || grep -n {} urlfile"
Create a mirror directory with the same filenames except all files and symlinks are empty files.
cp -rs /the/source/dir mirror_dir find mirror_dir -type l | parallel -m rm {} '&&' touch {}
Find the files in a list that do not exist
cat file_list | parallel 'if [ ! -e {} ] ; then echo {}; fi'
EXAMPLE: Composed command with perl replacement string You have a bunch of file. You want them sorted into dirs. The dir of each file should be named the first letter of the file name.
parallel 'mkdir -p {=s/(.).*/$1/=}; mv {} {=s/(.).*/$1/=}' ::: *
EXAMPLE: Composed command with multiple input sources You have a dir with files named as 24 hours in 5 minute intervals: 00:00, 00:05, 00:10 .. 23:55. You want to find the files missing:
parallel [ -f {1}:{2} ] "||" echo {1}:{2} does not exist \ ::: {00..23} ::: {00..55..5}
EXAMPLE: Calling Bash functions If the composed command is longer than a line, it becomes hard to read. In Bash you can use functions. Just remember to export -f the function.
doit() { echo Doing it for $1 sleep 2 echo Done with $1 } export -f doit parallel doit ::: 1 2 3
doubleit() { echo Doing it for $1 $2 sleep 2 echo Done with $1 $2 } export -f doubleit parallel doubleit ::: 1 2 3 ::: a b
To do this on remote servers you need to transfer the function using --env:
parallel --env doit -S server doit ::: 1 2 3 parallel --env doubleit -S server doubleit ::: 1 2 3 ::: a b
If your environment (aliases, variables, and functions) is small you can copy the full environment without having to export -f anything. See env_parallel.
EXAMPLE: Function tester To test a program with different parameters:
tester() { if (eval "$@") >&/dev/null; then perl -e 'printf "\033[30;102m[ OK ]\033[0m @ARGV\n"' "$@" else perl -e 'printf "\033[30;101m[FAIL]\033[0m @ARGV\n"' "$@" fi } export -f tester parallel tester my_program ::: arg1 arg2 parallel tester exit ::: 1 0 2 0
If my_program fails a red FAIL will be printed followed by the failing command; otherwise a green OK will be printed followed by the command.
EXAMPLE: Continously show the latest line of output It can be useful to monitor the output of running jobs.
This shows the most recent output line until a job finishes. After which the output of the job is printed in full:
parallel '{} | tee >(cat >&3)' ::: 'command 1' 'command 2' \ 3> >(perl -ne '$|=1;chomp;printf"%.'$COLUMNS's\r",$_." "x100')
EXAMPLE: Log rotate Log rotation renames a logfile to an extension with a higher number: log.1 becomes log.2, log.2 becomes log.3, and so on. The oldest log is removed. To avoid overwriting files the process starts backwards from the high number to the low number.  This will keep 10 old versions of the log:
seq 9 -1 1 | parallel -j1 mv log.{} log.'{= $_++ =}' mv log log.1
EXAMPLE: Removing file extension when processing files When processing files removing the file extension using {.} is often useful.
Create a directory for each zip-file and unzip it in that dir:
parallel 'mkdir {.}; cd {.}; unzip ../{}' ::: *.zip
Recompress all .gz files in current directory using bzip2 running 1 job per CPU in parallel:
parallel "zcat {} | bzip2 >{.}.bz2 && rm {}" ::: *.gz
Convert all WAV files to MP3 using LAME:
find sounddir -type f -name '*.wav' | parallel lame {} -o {.}.mp3
Put all converted in the same directory:
find sounddir -type f -name '*.wav' | \ parallel lame {} -o mydir/{/.}.mp3
EXAMPLE: Removing strings from the argument If you have directory with tar.gz files and want these extracted in the corresponding dir (e.g foo.tar.gz will be extracted in the dir foo) you can do:
parallel --plus 'mkdir {..}; tar -C {..} -xf {}' ::: *.tar.gz
If you want to remove a different ending, you can use {%string}:
parallel --plus echo {%_demo} ::: mycode_demo keep_demo_here
You can also remove a starting string with {#string}
parallel --plus echo {#demo_} ::: demo_mycode keep_demo_here
To remove a string anywhere you can use regular expressions with {/regexp/replacement} and leave the replacement empty:
parallel --plus echo {/demo_/} ::: demo_mycode remove_demo_here
EXAMPLE: Download 24 images for each of the past 30 days Let us assume a website stores images like:
https://www.example.com/path/to/YYYYMMDD_##.jpg
where YYYYMMDD is the date and ## is the number 01-24. This will download images for the past 30 days:
getit() { date=$(date -d "today -$1 days" +%Y%m%d) num=$2 echo wget https://www.example.com/path/to/${date}_${num}.jpg } export -f getit
parallel getit ::: $(seq 30) ::: $(seq -w 24)
$(date -d "today -$1 days" +%Y%m%d) will give the dates in YYYYMMDD with $1 days subtracted.
EXAMPLE: Download world map from NASA NASA provides tiles to download on earthdata.nasa.gov. Download tiles for Blue Marble world map and create a 10240x20480 map.
base=https://map1a.vis.earthdata.nasa.gov/wmts-geo/wmts.cgi service="SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0" layer="LAYER=BlueMarble_ShadedRelief_Bathymetry" set="STYLE=&TILEMATRIXSET=EPSG4326_500m&TILEMATRIX=5" tile="TILEROW={1}&TILECOL={2}" format="FORMAT=image%2Fjpeg" url="$base?$service&$layer&$set&$tile&$format"
parallel -j0 -q wget "$url" -O {1}_{2}.jpg ::: {0..19} ::: {0..39} parallel eval convert +append {}_{0..39}.jpg line{}.jpg ::: {0..19} convert -append line{0..19}.jpg world.jpg
EXAMPLE: Download Apollo-11 images from NASA using jq Search NASA using their API to get JSON for images related to 'apollo 11' and has 'moon landing' in the description.
The search query returns JSON containing URLs to JSON containing collections of pictures. One of the pictures in each of these collection is large.
wget is used to get the JSON for the search query. jq is then used to extract the URLs of the collections. parallel then calls wget to get each collection, which is passed to jq to extract the URLs of all images. grep filters out the large images, and parallel finally uses wget to fetch the images.
base="https://images-api.nasa.gov/search" q="q=apollo 11" description="description=moon landing" media_type="media_type=image" wget -O - "$base?$q&$description&$media_type" | jq -r .collection.items[].href | parallel wget -O - | jq -r .[] | grep large | parallel wget
EXAMPLE: Download video playlist in parallel youtube-dl is an excellent tool to download videos. It can, however, not download videos in parallel. This takes a playlist and downloads 10 videos in parallel.
url='youtu.be/watch?v=0wOf2Fgi3DE&list=UU_cznB5YZZmvAmeq7Y3EriQ' export url youtube-dl --flat-playlist "https://$url" | parallel --tagstring {#} --lb -j10 \ youtube-dl --playlist-start {#} --playlist-end {#} '"https://$url"'
EXAMPLE: Prepend last modified date (ISO8601) to file name parallel mv {} '{= $a=pQ($_); $b=$_;' \ '$_=qx{date -r "$a" +%FT%T}; chomp; $_="$_ $b" =}' ::: *
{= and =} mark a perl expression. pQ perl-quotes the string. date +%FT%T is the date in ISO8601 with time.
EXAMPLE: Save output in ISO8601 dirs Save output from ps aux every second into dirs named yyyy-mm-ddThh:mm:ss+zz:zz.
seq 1000 | parallel -N0 -j1 --delay 1 \ --results '{= $_=`date -Isec`; chomp=}/' ps aux
EXAMPLE: Digital clock with "blinking" : The : in a digital clock blinks. To make every other line have a ':' and the rest a ' ' a perl expression is used to look at the 3rd input source. If the value modulo 2 is 1: Use ":" otherwise use " ":
parallel -k echo {1}'{=3 $_=$_%2?":":" "=}'{2}{3} \ ::: {0..12} ::: {0..5} ::: {0..9}
EXAMPLE: Aggregating content of files This:
parallel --header : echo x{X}y{Y}z{Z} \> x{X}y{Y}z{Z} \ ::: X {1..5} ::: Y {01..10} ::: Z {1..5}
will generate the files x1y01z1 .. x5y10z5. If you want to aggregate the output grouping on x and z you can do this: