Question: Hello all :)

I often find myself having shell scripts - or rather, a list of commands to run via bash - which *could* be run concurrently if only there was a way to do that easily.

For example, here is a snippet one such file:

```
java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27ac" \
READ_GROUP_NAME="39V34V1.D265MACXX.3.CAGATC" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \

java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L004_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L004_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L004.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27ac" \
READ_GROUP_NAME="39V34V1.D265MACXX.4.CAGATC" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \

java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27ac_F_1_CAGATC_L003.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27ac" \
READ_GROUP_NAME="39V34V1.D2GD2ACXX.3.CAGATC" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \

java -jar /ex/picard/picard.jar FastqToSam \
DESCRIPTION="Circadian Day" \
FASTQ="/sequence/44_Mm01_WEAd_C2_H3K27me3_F_1_CGATGT_L003_R1_001.fastq.gz" \
FASTQ2="/sequence/44_Mm01_WEAd_C2_H3K27me3_F_1_CGATGT_L003_R2_001.fastq.gz" \
OUTPUT="/sequence/44_Mm01_WEAd_C2_H3K27me3_F_1_CGATGT_L003.bam" \
SAMPLE_NAME="C57BL/6J" \
LIBRARY_NAME="Mm01.H3K27me3" \
READ_GROUP_NAME="39V34V1.D265MACXX.3.CGATGT" \
SEQUENCING_CENTER="Freiburg" \
PLATFORM="illumina" \
..
```

Lets say there were 1000 of such commands in a file called **jobs.sh**, and I could run it with `bash jobs.sh` to have each one done one-after-the-other. However, since each job clearly does not rely on one another being completed, it might be nice to run them in parallel, say 5 at a time.

Bash natively has no way to do that without some really complicated logic wrapping each command, deciding when and when not to background a command and proceed to the next.

The Unix tool **parallel** can do it with something like:

parallel --jobs 5 sh -c {} < jobs.sh

However this will only work if the commands in jobs.sh are one-per-line. Multiline commands like the ones above break it. Furthermore, if the commands behave differently when their stdout/stderr are being redirected, they will detect that parallel is redirecting and, well, act differently - no status output, refuse to run, etc.
You can also do this with native Unix **xargs** (https://www.gnu.org/software/parallel/man.html#DIFFERENCES-BETWEEN-xargs-AND-GNU-Parallel) but its fiddlier and has all the same issues as parallel.

So my question is two-fold.

1. Does anyone know of a tool to parallelize a list of commands that would otherwise run consecutively in bash with little extra effort. Ideally each command gets run into it's own screen session so we can drop in at any time to see how its getting along (and no stdout/err redirection issues), and also spin up/down more worker processes as we go along, etc.
2. How do you manage jobs currently? Perhaps you never have to run 100 nearly-identical commands. Perhaps your commands are generated by something else (like Galaxy) which manages jobs for you also. Perhaps you, like I was doing last week, open up 32 new terminal windows to run 1/32th of the commands of jobs.sh in each :P Or perhaps something much more sophisticated.... hopefully :)

Answer: You're looking for **GNU make** and the `-j` option. Some clusters use a custom version of make (e.g., 'qmake' for SGE )

Search biostars for some other solutions (snakemake, ... ): e.g: https://www.biostars.org/p/115745/,

An example of Makefile:

```
samtools.dir=../samtools/
samtools.exe=${samtools.dir}samtools
wgsim.exe=${samtools.dir}misc/wgsim
bwa.exe=../bwa/bwa
picard.jar=../picard-tools-1.138/picard.jar
bcftools.exe=../bcftools/bcftools

REF=ref.fa

mutations.vcf.gz : ${REF} \
$(addsuffix .bam.bai ,S1) \
$(addsuffix .bam.bai ,S2) \
$(addsuffix .bam.bai ,S3) \
$(addsuffix .bam.bai ,S4)
${samtools.exe} mpileup -u -f $< $(filter %.bam,$(basename $^)) |\
${bcftools.exe} call -c -v -O z -o $@ -

$(addsuffix .bam.bai ,S1): $(addsuffix .bam,S1)
${samtools.exe} index $<

$(addsuffix .bam,S1): \
$(addsuffix .bam,S1_01_R1.fq.gz) \
$(addsuffix .bam,S1_02_R1.fq.gz) \
$(addsuffix .bam,S1_03_R1.fq.gz)
java -jar ${picard.jar} MergeSamFiles AS=true O=$@ $(foreach B,$^, I=${B} )

$(addsuffix .bam,S1_01_R1.fq.gz): $(addsuffix .bwt,${REF}) S1_01_R1.fq.gz S1_01_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S1\tSM:S1' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S1_01_R2.fq.gz : S1_01_R1.fq.gz
gzip --best -f $(basename $@)

S1_01_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 1 -N 1000 $< $(basename S1_01_R1.fq.gz S1_01_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam,S1_02_R1.fq.gz): $(addsuffix .bwt,${REF}) S1_02_R1.fq.gz S1_02_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S1\tSM:S1' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S1_02_R2.fq.gz : S1_02_R1.fq.gz
gzip --best -f $(basename $@)

S1_02_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 1 -N 1000 $< $(basename S1_02_R1.fq.gz S1_02_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam,S1_03_R1.fq.gz): $(addsuffix .bwt,${REF}) S1_03_R1.fq.gz S1_03_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S1\tSM:S1' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S1_03_R2.fq.gz : S1_03_R1.fq.gz
gzip --best -f $(basename $@)

S1_03_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 1 -N 1000 $< $(basename S1_03_R1.fq.gz S1_03_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam.bai ,S2): $(addsuffix .bam,S2)
${samtools.exe} index $<

$(addsuffix .bam,S2): \
$(addsuffix .bam,S2_01_R1.fq.gz) \
$(addsuffix .bam,S2_02_R1.fq.gz) \
$(addsuffix .bam,S2_03_R1.fq.gz)
java -jar ${picard.jar} MergeSamFiles AS=true O=$@ $(foreach B,$^, I=${B} )

$(addsuffix .bam,S2_01_R1.fq.gz): $(addsuffix .bwt,${REF}) S2_01_R1.fq.gz S2_01_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S2\tSM:S2' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S2_01_R2.fq.gz : S2_01_R1.fq.gz
gzip --best -f $(basename $@)

S2_01_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 2 -N 1000 $< $(basename S2_01_R1.fq.gz S2_01_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam,S2_02_R1.fq.gz): $(addsuffix .bwt,${REF}) S2_02_R1.fq.gz S2_02_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S2\tSM:S2' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S2_02_R2.fq.gz : S2_02_R1.fq.gz
gzip --best -f $(basename $@)

S2_02_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 2 -N 1000 $< $(basename S2_02_R1.fq.gz S2_02_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam,S2_03_R1.fq.gz): $(addsuffix .bwt,${REF}) S2_03_R1.fq.gz S2_03_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S2\tSM:S2' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S2_03_R2.fq.gz : S2_03_R1.fq.gz
gzip --best -f $(basename $@)

S2_03_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 2 -N 1000 $< $(basename S2_03_R1.fq.gz S2_03_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam.bai ,S3): $(addsuffix .bam,S3)
${samtools.exe} index $<

$(addsuffix .bam,S3): \
$(addsuffix .bam,S3_01_R1.fq.gz) \
$(addsuffix .bam,S3_02_R1.fq.gz) \
$(addsuffix .bam,S3_03_R1.fq.gz) \
$(addsuffix .bam,S3_04_R1.fq.gz)
java -jar ${picard.jar} MergeSamFiles AS=true O=$@ $(foreach B,$^, I=${B} )

$(addsuffix .bam,S3_01_R1.fq.gz): $(addsuffix .bwt,${REF}) S3_01_R1.fq.gz S3_01_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S3\tSM:S3' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S3_01_R2.fq.gz : S3_01_R1.fq.gz
gzip --best -f $(basename $@)

S3_01_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 3 -N 1000 $< $(basename S3_01_R1.fq.gz S3_01_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam,S3_02_R1.fq.gz): $(addsuffix .bwt,${REF}) S3_02_R1.fq.gz S3_02_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S3\tSM:S3' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S3_02_R2.fq.gz : S3_02_R1.fq.gz
gzip --best -f $(basename $@)

S3_02_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 3 -N 1000 $< $(basename S3_02_R1.fq.gz S3_02_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam,S3_03_R1.fq.gz): $(addsuffix .bwt,${REF}) S3_03_R1.fq.gz S3_03_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S3\tSM:S3' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S3_03_R2.fq.gz : S3_03_R1.fq.gz
gzip --best -f $(basename $@)

S3_03_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 3 -N 1000 $< $(basename S3_03_R1.fq.gz S3_03_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam,S3_04_R1.fq.gz): $(addsuffix .bwt,${REF}) S3_04_R1.fq.gz S3_04_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S3\tSM:S3' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S3_04_R2.fq.gz : S3_04_R1.fq.gz
gzip --best -f $(basename $@)

S3_04_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 3 -N 1000 $< $(basename S3_04_R1.fq.gz S3_04_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bam.bai ,S4): $(addsuffix .bam,S4)
${samtools.exe} index $<

$(addsuffix .bam,S4): \
$(addsuffix .bam,S4_01_R1.fq.gz)
java -jar ${picard.jar} MergeSamFiles AS=true O=$@ $(foreach B,$^, I=${B} )

$(addsuffix .bam,S4_01_R1.fq.gz): $(addsuffix .bwt,${REF}) S4_01_R1.fq.gz S4_01_R2.fq.gz
${bwa.exe} mem -R '@RG\tID:S4\tSM:S4' ${REF} $(filter %.gz,$^) |\
${samtools.exe} view -Sbu - |\
${samtools.exe} sort -O sam -o $@ -T $(basename $@) -

S4_01_R2.fq.gz : S4_01_R1.fq.gz
gzip --best -f $(basename $@)

S4_01_R1.fq.gz : ${REF}
@sleep 1
${wgsim.exe} -r 0.01 -S 4 -N 1000 $< $(basename S4_01_R1.fq.gz S4_01_R2.fq.gz)
gzip --best -f $(basename $@)

$(addsuffix .bwt,${REF}) : ${REF}
${bwa.exe} index $<
${REF}:
echo ">rotavirus" > $@
echo "GGCTTTTAATGCTTTTCAGTGGTTGCTGCTCAAGATGGAGTCTACTCAGCAGATGGTAAGCTCTATTATT" >> $@
echo "AATACTTCTTTTGAAGCTGCAGTTGTTGCTGCTACTTCAACATTAGAATTAATGGGTATTCAATATGATT" >> $@
echo "ACAATGAAGTATTTACCAGAGTTAAAAGTAAATTTGATTATGTGATGGATGACTCTGGTGTTAAAAACAA" >> $@
echo "TCTTTTGGGTAAAGCTATAACTATTGATCAGGCGTTAAATGGAAAGTTTAGCTCAGCTATTAGAAATAGA" >> $@
echo "AATTGGATGACTGATTCTAAAACGGTTGCTAAATTAGATGAAGACGTGAATAAACTTAGAATGACTTTAT" >> $@
echo "CTTCTAAAGGGATCGACCAAAAGATGAGAGTACTTAATGCTTGTTTTAGTGTAAAAAGAATACCAGGAAA" >> $@
echo "ATCATCATCAATAATTAAATGCACTAGACTTATGAAGGATAAAATAGAACGTGGAGAAGTTGAGGTTGAT" >> $@
echo "GATTCATATGTTGATGAGAAAATGGAAATTGATACTATTGATTGGAAATCTCGTTATGATCAGTTAGAAA" >> $@
echo "AAAGATTTGAATCACTAAAACAGAGGGTTAATGAGAAATACAATACTTGGGTACAAAAAGCGAAGAAAGT" >> $@
echo "AAATGAAAATATGTACTCTCTTCAGAATGTTATCTCACAACAGCAAAACCAAATAGCAGATCTTCAACAA" >> $@
echo "TATTGTAGTAAATTGGAAGCTGATTTGCAAGGTAAATTTAGTTCATTAGTGTCATCAGTTGAGTGGTATC" >> $@
echo "TAAGGTCTATGGAATTACCAGATGATGTAAAGAATGACATTGAACAGCAGTTAAATTCAATTGATTTAAT" >> $@
echo "TAATCCCATTAATGCTATAGATGATATCGAATCGCTGATTAGAAATTTAATTCAAGATTATGACAGAACA" >> $@
echo "TTTTTAATGTTAAAAGGACTGTTGAAGCAATGCAACTATGAATATGCATATGAGTAGTCATATAATTAAA" >> $@
echo "AATATTAACCATCTACACATGACCCTCTATGAGCACAATAGTTAAAAGCTAACACTGTCAAAAACCTAAA" >> $@
echo "TGGCTATAGGGGCGGTTTGTGACC" >> $@
echo "" >> $@

graph.png:
make -ndrB -f Makefile | ../makefile2graph/make2graph | dot -Tpng -o$@
```