Question: Hello all :)

I often find myself having shell scripts - or rather, a list of commands to run via bash - which *could* be run concurrently if only there was a way to do that easily.

For example, here is a snippet one such file:

```
java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27ac" \
READ_GROUP_NAME="39V34V1.D265MACXX.3.CAGATC" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \

java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L004_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L004_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L004.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27ac" \
READ_GROUP_NAME="39V34V1.D265MACXX.4.CAGATC" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \

java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27ac" \
READ_GROUP_NAME="39V34V1.D2GD2ACXX.3.CAGATC" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \

java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27me3_F_1_CGATGT_L003_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27me3_F_1_CGATGT_L003_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27me3_F_1_CGATGT_L003.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27me3" \
READ_GROUP_NAME="39V34V1.D265MACXX.3.CGATGT" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \
..
```

Lets say there were 1000 of such commands in a file called **jobs.sh**, and I could run it with `bash jobs.sh` to have each one done one-after-the-other. However, since each job clearly does not rely on one another being completed, it might be nice to run them in parallel, say 5 at a time.

Bash natively has no way to do that without some really complicated logic wrapping each command, deciding when and when not to background a command and proceed to the next.

The Unix tool **parallel** can do it with something like:

parallel --jobs 5 sh -c {} < jobs.sh

However this will only work if the commands in jobs.sh are one-per-line. Multiline commands like the ones above break it. Furthermore, if the commands behave differently when their stdout/stderr are being redirected, they will detect that parallel is redirecting and, well, act differently - no status output, refuse to run, etc.
You can also do this with native Unix **xargs** (https://www.gnu.org/software/parallel/man.html#DIFFERENCES-BETWEEN-xargs-AND-GNU-Parallel) but its fiddlier and has all the same issues as parallel.

So my question is two-fold.

1. Does anyone know of a tool to parallelize a list of commands that would otherwise run consecutively in bash with little extra effort. Ideally each command gets run into it's own screen session so we can drop in at any time to see how its getting along (and no stdout/err redirection issues), and also spin up/down more worker processes as we go along, etc.
2. How do you manage jobs currently? Perhaps you never have to run 100 nearly-identical commands. Perhaps your commands are generated by something else (like Galaxy) which manages jobs for you also. Perhaps you, like I was doing last week, open up 32 new terminal windows to run 1/32th of the commands of jobs.sh in each :P Or perhaps something much more sophisticated.... hopefully :)

Answer: > However this will only work if the commands in jobs.sh are one-per-line. Multiline commands like the ones above break it.

That can be mitigated simply by using `-d '\n\n'`. That way you will split on two newlines instead of one.

> Furthermore, if the commands behave differently when their stdout/stderr are being redirected, they will detect that parallel is redirecting and, well, act differently - no status output, refuse to run, etc.

If you do not like redirection `-u` is for you. But your output may be mixed up.

It seems your commands are extremely similar. I would personally use a template for that - maybe a bash function:

```
toSam() {
java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="$1" \
FASTQ="$2" \
FASTQ2="$3" \
OUTPUT="$4" \
SAMPLE_NAME="$5" \
LIBRARY_NAME="$6" \
READ_GROUP_NAME="$7" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina"
}
export -f toSam

parallel --header : toSam {Description} {Fastq} {=2 s/_R1_/_R2_/=} {=2 's/(_L\d\d\d)_.*/$1/' =} {Samplename} {Library} {Readgroup} ::: samplesheet.csv
```

You need a samplesheet.csv like this:

```
Description,Fastq,Samplename,Library,Readgroup
Circadian Day,/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R1_001.fastq.gz,C57BL/6J,Mm01.H3K27ac,39V34V1.D265MACXX.3.CAGATC
```

Before going all in on GNU Parallel use `--dryrun` to see what will be run. Try a couple of these commands by hand and see that they work as expected.